You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Based on the amount of urls to download, we need to estimate the size and download time.
This is important, since we develop the tool for heavy web researchers, that could have collected many links, therefore the size on the hard drive could be big or it would take substantial amount of time to crawl all the pages.
Technical Implementation:
I would say we estimate a size of 50kb per page that needs to be downloaded and 2 seconds for the parsing. Download will run multithreaded. 3 downloads simultaneously.
The text was updated successfully, but these errors were encountered:
Part of #6
Summary
Based on the amount of urls to download, we need to estimate the size and download time.
This is important, since we develop the tool for heavy web researchers, that could have collected many links, therefore the size on the hard drive could be big or it would take substantial amount of time to crawl all the pages.
Technical Implementation:
I would say we estimate a size of 50kb per page that needs to be downloaded and 2 seconds for the parsing. Download will run multithreaded. 3 downloads simultaneously.
The text was updated successfully, but these errors were encountered: