-
Notifications
You must be signed in to change notification settings - Fork 255
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
reduce "redundancy", so the check only happens once and that value is stored, allowing that value to be compared. #1201
Conversation
It seems there's some memory leak, it currently committed 1G+ memory in a long-run test. But maybe the reason is not this change |
Even if all images are skipped downloading (i.e. without writing any file), the committed memory still grows. |
can you compare before this PR and after? how do you test it? Last time when this happen, usually due to the BeautifulSoup page is not also maybe because the downloaded page is being cached for 1hour (useful if it need to get the same artist info multiple time and reduce access to pixiv server) |
Without this change, it seems 23M workset memory for a middle-run (maybe fewer hours and cool delay), don't remember what memory is in a long run.
It also costs ~300MB committed memory size if 10000 skipped via "Already downloaded in DB". Therefore may have nothing to do with file writing or beautifulsoup. |
In the future, these checks are not "useless", each check itself is important to the code. What you've done here, however, is reduce "redundancy", so the check only happens once and that value is stored, allowing that value to be compared. "Remove redundant file checks" would have been a more apt title for this pr. |
well, grammar is not my strong poing, but at least I can understand it 😄 |
Just firing up PixivUtil2 again and the "Skipped getting remote file size because local file not exists" is really getting to me... It should be "Remote file size check skipped because local file does not exist" or something like that. Ironically printing this message creates greater latency than the "redundant" checks it intended to remove, so they have actually managed to slow down the program where they intended to speed it up... If no message is printed, it should be implied that the file size check is skipped, else it prints the check when it the conditions to check the file size is met. This notification for every new image is superfluous... |
|
I had to see the numbers for myself, and you are right. The line printing adds only an additional If you are able to perceive a difference of even |
|
Issue: #1200