Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use git-lfs for large testing dependencies #5967

Closed
jonringer opened this issue Nov 28, 2020 · 1 comment
Closed

Use git-lfs for large testing dependencies #5967

jonringer opened this issue Nov 28, 2020 · 1 comment
Labels
build build issues; typically submitted using template

Comments

@jonringer
Copy link

Is your feature request related to a problem? Please describe.
The current unpacked directory size (without git history) for onnxruntime is 450MB. This is what package repositories usually consume for packaging software, and it's massive. Git submodules are another 550MB+, which means a full checkout will be around 1GB+.

System information

  • ONNX Runtime version (you are using):

Describe the solution you'd like
Use git-lfs

Describe alternatives you've considered
Use an alternate repo to hold test artifacts

Additional context

$ find . -type f -exec du -hd0 {} + | sort -rh | head -30
98M	./onnxruntime/python/tools/quantization/E2E_example_model/resnet50_v1.onnx
40M	./onnxruntime/test/testdata/ort_ckpt/bert_toy_lamb.ZeRO.3.3.ort.pt
35M	./onnxruntime/test/testdata/ort_ckpt/bert_toy_lamb.ZeRO.2.3.ort.pt
35M	./onnxruntime/test/testdata/ort_ckpt/bert_toy_lamb.ZeRO.1.3.ort.pt
35M	./onnxruntime/test/testdata/ort_ckpt/bert_toy_lamb.ZeRO.0.3.ort.pt
32M	./winml/test/scenario/models/coreml_Resnet50_ImageNet-dq.onnx
20M	./onnxruntime/test/testdata/bert_toy_postprocessed.onnx
10M	./winml/test/scenario/models/onnxzoo_lotus_inception_v1-dq.onnx
4.8M	./winml/test/common/testdata/squeezenet/model.onnx
4.8M	./winml/test/collateral/models/squeezenet_modifiedforruntimestests.onnx
4.8M	./winml/test/collateral/models/modelWithMetaData.onnx
4.8M	./winml/test/collateral/models/modelWith2MetaData.onnx
4.8M	./winml/test/collateral/models/ModelSubdirectory/ModelInSubdirectory.onnx
4.8M	./onnxruntime/test/testdata/squeezenet/model.onnx
4.8M	./nodejs/test/testdata/squeezenet.onnx
4.8M	./csharp/testdata/squeezenet.onnx
3.3M	./winml/test/collateral/models/starry-night-fp16.onnx
3.1M	./onnxruntime/test/testdata/transform/fusion/test_data_set_0/output_0.pb
2.8M	./winml/test/collateral/models/squeezenet_tensor_input.onnx
2.7M	./onnxruntime/python/tools/tensorrt/perf/models/yolov4/custom_test_data/test_data_set/output0.pb
2.0M	./onnxruntime/python/tools/tensorrt/perf/models/yolov4/custom_test_data/test_data_set/input0.pb
2.0M	./nodejs/test/testdata/squeezenet.input0.json
2.0M	./csharp/testdata/bench.in
1.7M	./winml/test/image/models/fns-candy_tensor.onnx
1.7M	./winml/test/image/models/fns-candy_Rgb8.onnx
1.7M	./winml/test/image/models/fns-candy_Bgr8_tensor_Batch3.onnx
1.7M	./winml/test/image/models/fns-candy_Bgr8.onnx
1.7M	./winml/test/image/models/fns-candy_Bgr8_freeDimInput.onnx
1.7M	./winml/test/image/models/fns-candy_Bgr8_freeDimInput_Batch10.onnx
1.7M	./winml/test/image/models/fns-candy_Bgr8_Batch3.onnx

related packaging issue: #5966

@thiagocrepaldi thiagocrepaldi added the build build issues; typically submitted using template label Nov 30, 2020
jonringer added a commit to jonringer/nixpkgs that referenced this issue Dec 5, 2020
Poor cmake and git hygiene has made maintaining this difficult, see:
microsoft/onnxruntime#5966
microsoft/onnxruntime#5967
@jonringer
Copy link
Author

closing due to staleness

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
build build issues; typically submitted using template
Projects
None yet
Development

No branches or pull requests

3 participants
@thiagocrepaldi @jonringer and others