Join GitHub today
GitHub is home to over 36 million developers working together to host and review code, manage projects, and build software together.Sign up
fix: prevent multiple downloads of same file #2477
This PR is proof-of-concept to prevent multiple downloads of same file as described in #2442.
For example given in #2442, here's downloads before this patch:
And after patch:
It builds on idea of
@ry could you take a look at this PR?
Tests are failing but I believe they are not replicating "real life" scenario - I don't think it's possible to change
Also I had other idea: update
@ry I think I managed to find solution using initial approach that involves marking which URLs had already been downloaded. It's much simpler compared to saving whole response.
I changed two tests to behave like a real world scenario and added a test for multiple downloads (verifying that headers file modified timestamp is unchanged). Let me know what you think.
But as you said earlier this is definitely to be rewritten from ground up.