Workflow for pushing conda nightly binaries and merging all nightly runs to a single workflow

[agunapal]

Description

Merged torchserve-nightly, torch-model-archiver-nightly, torch-workflow-archiver and torchserve conda nightly
into 1 single flow.

Current Behavior:

• We have a separate workflow for each of PyPI builds of torchserve-nightly, torch-workflow-archiver-nightly and torch-model-archiever-nightly
• No workflow for conda nightly binaries

Problem:

• Unnecessary use of extra compute

Solution:

The workflow does the following:

• Launches binaries creation workflow for the 3 OS: ubtunu, macOS and windows
• Creates a conda environment
• Builds and uploads the PyPI binaries (serve, model-archiver and workflow-archiver) for ubuntu, but skips it for windows and macOS because the binaries have the same name
• Builds and uploads conda binaries for all the 3 OS

Miscellaneous:

• Introduced a new secret key called CONDA_NIGHTLY_TOKEN to push the conda nightly binaries to https://anaconda.org/pytorch-nightly/
• Wrote a new function called build_dist_whl to build wheel files. Build nightly wheel files by passing "nightly" arg
• Updated upload_pypi_packages to upload to test.pypi.org or pypi.org based on "test-pypi" arg
• Update upload_conda_packages to upload based on setting ANACONDA_API_TOKEN with CONDA_NIGHTLY_TOKEN

Fixes #(issue)

Reduces developer time to build and push binaries

Type of change

• New feature (non-breaking change which adds functionality)

Feature/Issue validation/testing

• Test A

• Test B
  You can view all the uploaded files here
  https://anaconda.org/agunapal/repo

• Test C
  Binary being updated with the latest date

• Test D
  Tested the binary on GPU as well

• Test E
  PyPI binaries test

• Test F
  PyPI binaries test

Checklist:

• [ Yes] Did you have fun?
• [ Yes] Have you added tests that prove your fix is effective or that this feature works?
• [ Yes] Has code been commented, particularly in hard-to-understand areas?
• [ Yes] Have you made corresponding changes to the documentation?

[msaroufim]

Thanks @ankithagunapal! I'm super glad to see this change. There's a few more things we need to do to merge this

1. The package should be called *-nightly e.g torchserve-nightly to clearly differentiate it from the official torchserve releases and should be part of the pytorch-nightly org https://anaconda.org/pytorch-nightly
2. The name of the package needs to include the date for example see what torchvision does https://anaconda.org/pytorch-nightly/torchvision/files. In our case let's use the same naming convention we use in our pypi nightly scripts to stay consistent https://github.com/pytorch/serve/blob/master/setup.py#L56 which will show up as in https://pypi.org/project/torchserve-nightly/#description
3. There's already a secret called CONDA_PASSWORD which you can leverage (I can rename it too if you think I didn't give it the correct name), the permissions there are fairly broad so feel to ping Eli Uriegas internally or just ping me again when I'm back from PTO in case you're not sure if we're doing something destructive. The last thing we want to do is override official releases or delete stuff

[agunapal]

@msaroufim Have updated the binaries to have a format (including date) which is consistent with other packages in pytorch.

Updated yml file to use CONDA_NIGHTLY_TOKEN which will push the binaries to https://anaconda.org/pytorch-nightly/

[msaroufim]

LGTM, some minor nits are left that that need to be fixed before merge

Also I'd like us to add instructions on the main README for installation for both regular and nightly conda https://github.com/pytorch/serve#-quick-start-with-torchserve

[lxning]

@agunapal conda binary is based on pypi. We already have pypi nightly build. To make pypi and conda binary consistent, I think they should be sit together; otherwise, it would potentially cause binary discrepancy and also waste resource. In other words, the existing pypi nightly build workflow is extended to add one more steps to build/release conda.

[msaroufim]

@agunapal conda binary is based on pypi. We already have pypi nightly build. To make pypi and conda binary consistent, I think they should be sit together; otherwise, it would potentially cause binary discrepancy and also waste resource. In other words, the existing pypi nightly build workflow is extended to add one more steps to build/release conda.

That's a good point, I think we can actually unify the 3 actions we have for nightly torchserve, workflow-archiver and model-archiver into a single action and then add a step there to also do conda https://github.com/pytorch/serve/blob/master/.github/workflows/torchserve-nightly-build.yml

[agunapal]

@agunapal conda binary is based on pypi. We already have pypi nightly build. To make pypi and conda binary consistent, I think they should be sit together; otherwise, it would potentially cause binary discrepancy and also waste resource. In other words, the existing pypi nightly build workflow is extended to add one more steps to build/release conda.

Thanks @lxning Thats a good suggestion

[agunapal]

4 workflows have been merged to a single workflow

[msaroufim]

Approved, let's just add some pypi binaries uploaded