Fetch test data from s3 instead of using git-lfs #925
Conversation
Codecov Report
@@ Coverage Diff @@
## master #925 +/- ##
==========================================
+ Coverage 81.30% 81.75% +0.44%
==========================================
Files 54 54
Lines 8201 8220 +19
==========================================
+ Hits 6668 6720 +52
+ Misses 1533 1500 -33
Continue to review full report at Codecov.
|
| - v1.2.5-deps-{{ checksum "get_deps.sh" }}-gpu | ||
| - restore_cache: | ||
| keys: | ||
| - v1.2-tests_data-gpu |
There was a problem hiding this comment.
what is the reason we are having an additional cache key? Should be the same for both modes, no?
There was a problem hiding this comment.
You're right, that's how it was before, but I got access denied when restoring cache, so I tried this option instead....
|
|
||
| #---------------------------------------------------------------------------------------------- | ||
|
|
||
| get_tests_data() { |
There was a problem hiding this comment.
I don't mind having this right now, but it forces developers to have aws account to run this command.
Please see https://stackoverflow.com/questions/273743/using-wget-to-recursively-fetch-a-directory-with-arbitrary-files-in-it if we can use wget with aws HTTP.
There was a problem hiding this comment.
Done. Using recursive wget was not possible, so we got the desired behavior by using a file that specify all the relevant resources to download.
…without aws configure + update onnx benchmark
…will get the data if it's not cached in CI)
Remove git-lfs from this project, and use S3 to store tests data (which includes some heavy models). We use
s3://dev.cto.redisbucket, and restore data from it in CI using sync command. The data is also stored in circle-CI local cache