Finished working on a basic parallel crawler for Structured Data in Rust #1
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Implemented a better command line argument read, now takes an input file with pages (does not yet perform crawling) on which to perform structured data scraping. Now it also writes data naked and simple into another file.
There is a lot of new stuff inside, it doesn't work yet though, a couple of issues to resolve with parsing out relative urls. A lot of todos inside + have to write tests
The architecture still sucks, going to rework it today-tomorrow hopefully. I need to have a smarter thread pool, probably don't have to actually create so many threads each time, can create them once and just update their urls or something like that.
Added nice TUI, divided everything up into several structures and their associated functions. Still a lot of TODOs and cleanup to do, but it should work roughly as expected.
Update Dockerfile, since we now need an input file copied to work, and also have to provide command line parameters to the executable. Image is still super small, only 14megs. Improved error handling in link normalization, now threads don't panick there if something goes wrong. They still can panic in other places, will have to improve error handling there too once we start working with more data from different websites. Improved progress bar display, add progress and finish message. Scratch Docker image is not capable of displaying the progress bar, probably will have to fix that somehow or just implement something workable for stupid terminals.
10 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I've implemented a simple crawler that uses thread pool and message passing to crawl webpages in parallel. While it uses most of CPU's resources and does everything we agreed upon for this week, there is still a lot of room for improvement, more on this below.
Completed:
Things left to do, roughly in the descending order of importance: