Some concerns before the implementation of the Model Zoo CI

[lyd911]

About the ONNX Model Zoo CI Proposal

Sorry for the delay, and we will kick off our jobs based on the proposal submitted on April. Though getting positive feedbacks from the ONNX community, some questions still remains regarding this proposal before actual implementing.
I list some questions that the OpenLab team comes up with and want to hear the community’s idea:

1. The current ONNX model zoo provides a .md file which contains the url to download the model. Most models provide the url in the Download section while some models didn't follow the template. For the convenience of the CI system to run the script and extract this information, it is still necessary for the model developers to upload their works strictly under the formal ONNX Model Zoo template in order to successfully trigger the CI system.
2. The runtime used in this project shall obey the basic principle of interoperability thus the runtime should be changeable. We choose the ONNX Runtime as the default runtime and can be expanded to launch CI jobs with other runtimes in the future.
3. Neither Model Zoo nor CI system can auto sync with the model update, so it is depend on the owner of the model to update its ONNX implementation. The owner of one specific ONNX model has the responsibility to sync with the original model and adapt to the current ONNX version.
4. In order to run the model on the OpenLab platform, some executable source code may be modified, and the modified source code will be merged under the name of the original model (e.g. alexnet). Is the distribution of these models with modified source code causing any copyright issues?
5. The performance test is a long term goal which the metrics are defined by each individual scenario, currently only image processing models are collected in the model zoo while other models (e.g. for nlp problems) and its test cases are still rely on the contribution of the community members.

We are looking forward to hear your idea, thanks.

[prasanthpul]

We've been looking at enabling GIT LFS. That way the model binaries can be stored in Github as well instead of on external location and providing a URL. By being in Github, you should be able to trigger CI when files change. Would this simplify the CI system development?

[wangxiyuan]

@prasanthpul that's the best way!

[lyd911]

@prasanthpul Sounds great, is there a timeline when this feature can be online?