feat: unify parseWithCheerio, add parseWithLinkeDOM [WIP] #1890
Conversation
For cheerio it's fine to add it to the http crawler package dependencies, it is technically already there as a transitive one. Not so much for linkedom 👇
Hmm, that's actually a tricky one to do in the base layer, as it requires the parsed response for finding the links. So maybe the idea with the helper won't fly - but I don't mind having the linkedom crawler as a class with its enqueue implementation, like the other crawlers do. So basically what you were trying in the first place (but with a lot of copypasta I guess, haven't really checked the code tbh). In the end, the helper for parsing with linkedom would be quite a weird thing outside of the http crawler. Let's make this PR just about adding the cheerio helper to http crawler, that's what we can merge first easily. |
TBH, I did not want to do that initially (as I mentioned, cheerio was "fine" for me), but after using it I came about to liking it :) Agree! 🙏 |
) As discussed in #1890 --------- Co-authored-by: Martin Adámek <banan23@gmail.com>
The clean approach of #1879 as discussed #1890 (comment) 2023-05-09 16:05 Rebased --------- Co-authored-by: Martin Adámek <banan23@gmail.com>
|
Closing #1906 |
Just draft 👷♂️
Some issues with this approach:
parseWith*in HttpCrawler without importing cheerio & linkedom