Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Estimate Size and Download time #2

Closed
blackforestboi opened this issue Oct 22, 2016 · 1 comment
Closed

Estimate Size and Download time #2

blackforestboi opened this issue Oct 22, 2016 · 1 comment
Assignees
Labels

Comments

@blackforestboi
Copy link
Member

blackforestboi commented Oct 22, 2016

Part of #6

Summary

Based on the amount of urls to download, we need to estimate the size and download time.
This is important, since we develop the tool for heavy web researchers, that could have collected many links, therefore the size on the hard drive could be big or it would take substantial amount of time to crawl all the pages.

Technical Implementation:

I would say we estimate a size of 50kb per page that needs to be downloaded and 2 seconds for the parsing. Download will run multithreaded. 3 downloads simultaneously.

@blackforestboi
Copy link
Member Author

done with #33

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

4 participants