Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Our own PyPI repository to host Backend.AI-optimized Python packages #160

Open
achimnol opened this issue Jan 20, 2019 · 1 comment
Open
Assignees
Labels
area:infrastructure Infrastructure-related issues

Comments

@achimnol
Copy link
Member

We are now managing our own builds of TensorFlow and Alpine Linux version of precompiled Python pacakges.

Let's host them on a private PyPI repository so that we could simplify docker-build steps here.
For instance, we could run a separate script to build those packages using containers and automatically upload them to the private PyPI repository. Then downloading/installing those packages inside the Dockerfiles will become much simpler than using multi-staged builds. (The big shortcoming of multi-staged builds is that it is difficult to control caches and intermediate image name/tags...)

Previously it was cumbersome and resource-consuming to host a private PyPI repository because we had to run a dedicated server. But now, there is a tool called s3pypi which allows us to build and host a PyPI repository with only S3 static website hosting + CloudFront.

Suggested repository URLs:

  • https://pypi.backend.ai/kernels/alpine/3.8/
  • https://pypi.backend.ai/kernels/ubuntu/16.04/
  • https://pypi.backend.ai/kernels/ubuntu/18.04/

This could eliminate use of our "build.py" script and complicated build chains.

Also, @tlqaksqhr's https://github.com/lablup/backend.ai-packages could be hosted in this way as well, probably via:

  • https://pypi.backend.ai/dist/ubuntu/16.04/ or something similar.
    In this case the repository is for distributing Backend.AI itself.
    Specifically, we could also make a secret repository for enterprise customers.
@achimnol achimnol transferred this issue from lablup/backend.ai-kernels Feb 6, 2020
@achimnol achimnol added the area:infrastructure Infrastructure-related issues label Feb 6, 2020
@achimnol
Copy link
Member Author

achimnol commented Jun 15, 2020

This would be useful to host the artifacts built by numpy-builder and possibly other customized wheel builds.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:infrastructure Infrastructure-related issues
Projects
None yet
Development

No branches or pull requests

3 participants