New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[model weights caching] model upload doesn't check model weights hash #6916
Comments
I can confirm it was previously checking the model weights and re-downloading if the weights had been changed. Investigating. |
This is due to the CDN caching files, with a 24 hour delay. After 24 hours it should download your file, but if you want it now you can use the |
Thank you for the hint, @LysandreJik. So But that might be tricky for end users who won't know that the code base has changed yet the model weights they get are out sync. Is there a way to signal CDN to invalidate the cache for some files? It could then be done from the upload util. |
FWIW, I wrote a one liner to force cache update for the 4 models I'm working at the moment.
I now have that in my script, so I don't need to think about it. |
@LysandreJik, unfortunately this doesn't solve the issue
Indeed forces a download of the recently updated model - but then if this flag is no longer used in the application - it still downloads the CDN cached version and ends up using the wrong version. So, basically, this results in 2 copies (different hashes) sitting in the cache dir. And normal usage w/o using Thanks. |
can you run |
I can do that, but I already checked that it downloads the updated model w/ |
Oh yeah ok, I see. Can you |
Yes. Except others can't as they don't have my local copy. e.g. @sshleifer wants to eval my PR #6940, but now has to wait till tomorrow for CDN to expire (or hack around it). Last night I uploaded an experimental model, which proved to be invalid, thought I re-downloaded it OK as it was working OK and made a PR, except I was testing against the non-current cached version, which was a good one. |
Can we please re-open this ticket? It hasn't been resolved |
Can we add a In our dev workflow we mostly don't use the cdn while the files are still in-flux. Cloudfront invalidation comes with its own set of issues so it's better to view cdn as a means to distribute permanent files. (for this reason we don't serve config.json files from Cloudfront) |
It could be done. I have a feeling then there will be others. Perhaps an alternative solution would be to introduce an env var, that would transparently override cdn cache in any situation w/o needing to change every script?
Understood! How do you let others onto testing the model files? Putting them on dropbox or something and sharing the link? |
No, just S3 links! |
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. |
#8324 should resolve this. |
I have re-uploaded model weights via
transformers-cli upload
and noticed that when I tried to use it - it didn't get re-downloaded, and instead continued to use the cached version.The problem seems to come from the fact that the other uploaded files haven't changed, only the model weights.
I double checked that the md5sum of the old weights file is different from the new one.
I re-uploaded the whole folder using:
If I hunt down the cached files (not an easy task), and delete those, it does re-download the new version.
If I diff the cached weights file and the updated cache file, which gets re-downloaded if I move away the original cached file, they aren't the same.:
Could we please include the model weights file in the hash calculation?
Thank you.
The text was updated successfully, but these errors were encountered: