-
Notifications
You must be signed in to change notification settings - Fork 41
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: add playwright scraper #26
Conversation
cc @barjin, maybe you can reuse something from this |
… scraper-tools package
@B4nan could not add you as a reviewer 🤷♂️ Updated the readme (as well as quite a few other things), tried to build on the platform - seems like all is good. There are some rough things to be polished in scraper tools, but seem to be working fine. Using big playwright image and runs both chromium/firefox. Webkit has some issues with fingerprints for some reason. I guess if it's gonna be slow - in the future we could create Playwright scraper+chromium and separate with firefox one. |
btw - I put some links to Apify SDK with |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
thank you, great job :)
@B4nan so, apparently I fucked up package-lock on merging changes from master here: bff9f9e I compared the changes from this branch with master and tried to install everything/run the tests locally - it was fine when I noticed that the only difference is lock file and there's something off with it. So yeah, I thought I am going crazy 🤯🤯🤯 But finally all green. |
The thing with reinstalling I noticed some time ago - if you want to wipe node modules, lock file and reinstall, you also need to wipe the node modules folder inside |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
gave it just a very bried look, i'd say lets merge and push to the platform as beta to test it properly there
As for the previous comment (node modules wiping etc) - that would actually explain it - I was kinda thinking even how I ended up with that new lock file - as when I tried to wipe it and npm install fresh - it was still different - so yeah - must have left some node_modules in packages |
No description provided.