Improve/merge scripts, avoid RegEx, format when crawling #945
Conversation
✅ Deploy Preview for place-atlas ready!
To edit notification comments on pull requests, go to your Netlify site settings. |
|
I actually just saw that #908 also gone JSON-based (I think)? Oh, well. |
|
Crawler is untested, but in theory it should work. If someone can test this then I would appreciate it. |
Hans5958
force-pushed
the
regex-is-old-school-af
branch
2 times, most recently
from
82e87d8
to
e72dd87
Compare
|
Hmm, what should be done with the user links? |
Hans5958
force-pushed
the
regex-is-old-school-af
branch
3 times, most recently
from
2f4bc97
to
143d2d5
Compare
|
Is there ambiguity about user links? The main two cases are handled, being Reddit's Markdown format and URLs without protocol. |
I mean those that they put on the subreddit fields |
|
If the Reddit link is just a full URL to the subreddit, it should be simple to detect where the |
Hans5958
force-pushed
the
regex-is-old-school-af
branch
from
9f8227a
to
009d6ab
Compare
I see. I have implement that on CSTW. I want to clarify again that what I mean is this. I thought it can be supported on the Atlas, but who knows? [
{"id": "txs8wu", "submitted_by": "_Neroxis", "name": "Neroxis", "description": "The profile picture of the redditor _Neroxis.", "website": "http://neroxis.net", "subreddit": "/u/_neroxis", "center": [1530.5, 163.5], "path": [[1522.5, 159.5], [1527.5, 154.5], [1534.5, 154.5], [1539.5, 160.5], [1539.5, 166.5], [1534.5, 171.5], [1527.5, 171.5], [1522.5, 166.5]]},
] |
|
For some reason either my connection is drunk or what, but I have pushed 3 commit yet it didn't appear. Really classic. |
|
Oh interesting, a user link. I don't think that's supported right now; we could probably handle that manually by moving it to the description. |
Hans5958
force-pushed
the
regex-is-old-school-af
branch
from
dbbeb43
to
1e5c4f8
Compare
So, that would be like this? I mean, I think it would be better (or worse) to support it directly on the JS, but IDK. |
|
In this case, would work. I don't think you need to script that, since it's a case-by-case basis what looks the best. Maybe at most print out that the issue is there so someone can manually fix it. |
|
Ah, case-by-case basis. Gotcha. One question, does description parsed as Markdown, or how newlines work? Two newlines is a paragraph split? or single? |
|
The description is not parsed as Markdown, so Markdown syntax like links and bold/italics will show up raw and incorrect. I think two newlines is a paragraph split. |
Ah, okay, so I can just leave it like that. By the way...
I just did that. It's also called "validation," pretty weird. Maybe sometime we can merge it with the REAL validation script. Anyways, there has been a lot of errors with it. You can't wait to see those. |
|
I think we can keep it separate. The current validation script serves its purpose of making sure the JSON isn't invalid, which is a showstopper for Atlas running. These other validations are things for us to manually fix, good to have but not blocking the operation of the app. |
Logic and port made/adapted from Nick's Co-authored-by: Nicolas Abram <abramlujan@gmail.com>
We're trying to unescape them, it seems to work for me? But you're right about reddit escaping the backslash. Your current regex should work on submissions not using a code block or the markdown editor. |
Also adapted from Nick Co-authored-by: Nicolas Abram <abramlujan@gmail.com>
Hans5958
force-pushed
the
regex-is-old-school-af
branch
from
bab540d
to
121d865
Compare
|
*unescape, apologize. Oh, I misread, apologize again for that. |
|
Could you keep this as either 0, 1 or 2 for now? We wanna avoid re-accepting entries that were explicitly removed via edits |
|
Oh shoot forgot to port that one. I saw you got 0 so I'll do 0. |
Hans5958
force-pushed
the
regex-is-old-school-af
branch
from
df658e8
to
c835060
Compare
Hans5958
force-pushed
the
regex-is-old-school-af
branch
from
c835060
to
cc497d5
Compare
|
Looks good to me, one final question: Does this properly handle entries with multiple subreddits? It looks like the js code splits them by commas: |
|
Yes, and actually also trims the spaces, so both |
Fun fact: The JS script also includes those that have no r/, but I made the validator warns it so future contributors can confirm and change it into a proper format, or delete it.
Hans5958
force-pushed
the
regex-is-old-school-af
branch
2 times, most recently
from
43897eb
to
5b4ca08
Compare
as in, new entries will be formatted automatically, but full format may be needed when there is a change on the formatter itself
Hans5958
force-pushed
the
regex-is-old-school-af
branch
from
5b4ca08
to
ef0f1fd
Compare
|
That was a panic in #1072. But, alas! 🎉 I'd like to thank everyone who have helped me on this pull request. May this be a useful addition for everyone who uses the site. :) Here are the errors of today's full format. If anyone wants to fix it then go ahead. |
Inspired by #908*, I made this. I suppose wanted to merge this for a while, but I just separated so you guys can check the changed files better because it seperated, but now here we are.
Also I migrated it to be JSON-based instead of RegEx-based, and would you look at that speedup! Crawler would be also simplified to just use JSON.
And also more additions such as converting
andon subreddits, and resolving #707, so that's something.* This is a different implementation of it.
Fix #707