Skip to content

Conversation

@robotboy655
Copy link
Contributor

This is different from #31, this PR aims to speed up the "initial" wiki scrape. It is way too slow right now.

Also wipe the output folder by default when running npm run scrape-wiki, so that files do not have duplicated data.

This significantly speeds up wiki scraping
Is there a reason not to do this?
@luttje
Copy link
Owner

luttje commented Feb 15, 2024

Hi there, thanks a bunch for contributing to this little project. I really appreciate you taking time form working on Garry's Mod to help out here.

IIRC me not wiping the output folder by default was so I could separately test the packing functionality on preexisting output (without accidentally wiping my "test data"). However since everything seem to work alright, there's no reason not to wipe now.

The scraping really is a lot faster thanks to you. I remember thinking taking the slow way would be better not to trigger any throttling on the wiki server. You coming in with this PR really helps, since I wouldn't have dared to make such a change myself.

Thanks again!

@luttje luttje merged commit 5213c35 into luttje:main Feb 15, 2024
@luttje
Copy link
Owner

luttje commented Feb 15, 2024

@all-contributors add @robotboy655 for code

@allcontributors
Copy link
Contributor

@luttje

I've put up a pull request to add @robotboy655! 🎉

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants