-
Notifications
You must be signed in to change notification settings - Fork 107
Migrate anaconda prune from pytorch to test-infra #2691
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
weiwangmeta
merged 14 commits into
main
from
weiwangmeta/migrate_anaconda_prune_frome_core
Feb 10, 2023
Merged
Changes from all commits
Commits
Show all changes
14 commits
Select commit
Hold shift + click to select a range
39027bb
Migrate workflows from PyTorch to test-infra
weiwangmeta ecf0071
Migrate scripts from pytorch to test-infra
weiwangmeta 2707555
checkout test-infra
weiwangmeta 105b215
fix syntax
weiwangmeta fc4f619
see where we are
weiwangmeta 0b7f0be
space tab?
weiwangmeta 1e0fa3d
see what's around
weiwangmeta 2eb826f
fix bug in .github vs github :(
weiwangmeta 426491f
further debug
weiwangmeta 1948246
Debug path issues
weiwangmeta 7036a3d
Checkout github ref
weiwangmeta 6b4c37f
Debug
weiwangmeta 1359b25
Add dir containing scripts
weiwangmeta c8a2c09
Get rid of ls -al
weiwangmeta File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,52 @@ | ||
| #!/usr/bin/env bash | ||
|
|
||
| grab_prune_version() { | ||
| conda search -c "${CHANNEL}" --platform "${platform}" "${PKG}" 2>/dev/null | \ | ||
| grep "${CHANNEL}" | \ | ||
| awk -F ' *' '{print $2}' | \ | ||
| uniq | \ | ||
| head -n -1 | \ | ||
| xargs | ||
| } | ||
|
|
||
| grab_latest_version() { | ||
| conda search -c "${CHANNEL}" --platform "${platform}" "${PKG}" 2>/dev/null | \ | ||
| grep "${CHANNEL}" | \ | ||
| awk -F ' *' '{print $2}' | \ | ||
| uniq | \ | ||
| tail -n 1 | \ | ||
| xargs | ||
| } | ||
|
|
||
| grab_specs_for_version() { | ||
| conda search -c "${CHANNEL}" --platform "${platform}" "${PKG}" 2>/dev/null | \ | ||
| grep "${CHANNEL}" | \ | ||
| grep "$1" | \ | ||
| awk -F ' *' '{print $3}' | \ | ||
| uniq | \ | ||
| xargs | ||
| } | ||
|
|
||
| set -eou pipefail | ||
|
|
||
| CHANNEL=${CHANNEL:-pytorch-nightly} | ||
| PKG=${PKG:-pytorch} | ||
| PLATFORMS=${PLATFORMS:-noarch osx-64 osx-arm64 linux-64 win-64} | ||
|
|
||
| for platform in ${PLATFORMS}; do | ||
| latest_version="$(grab_latest_version || true)" | ||
| specs_in_latest_version="$(grab_specs_for_version "${latest_version}" || true)" | ||
| versions_to_prune="$(grab_prune_version || true)" | ||
| for version in ${versions_to_prune}; do | ||
| specs_in_prune_version="$(grab_specs_for_version "${version}" || true)" | ||
| for spec in ${specs_in_prune_version}; do | ||
| # If this spec is included in specs_in_latest_version, then remove it. | ||
| if [[ "${specs_in_latest_version}" =~ ${spec} ]];then | ||
| ( | ||
| set -x | ||
| anaconda remove --force "${CHANNEL}/${PKG}/${version}/${platform}/${PKG}-${version}-${spec}.tar.bz2" | ||
| ) | ||
| fi | ||
| done | ||
| done | ||
| done | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,14 @@ | ||
| #!/usr/bin/env bash | ||
|
|
||
| set -euo pipefail | ||
|
|
||
| DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )" | ||
|
|
||
| CHANNEL=${CHANNEL:-pytorch-nightly} | ||
| PACKAGES=${PACKAGES:-pytorch} | ||
|
|
||
| for pkg in ${PACKAGES}; do | ||
| echo "+ Attempting to prune: ${CHANNEL}/${pkg}" | ||
| CHANNEL="${CHANNEL}" PKG="${pkg}" "${DIR}/prune.sh" | ||
| echo | ||
| done |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,37 @@ | ||
| name: Prune Anaconda Binaries | ||
|
|
||
| on: | ||
| workflow_call: | ||
| inputs: | ||
| packages: | ||
| required: true | ||
| type: string | ||
| description: The packages to prune | ||
| channel: | ||
| required: true | ||
| type: string | ||
| description: The channel to prune packages | ||
| secrets: | ||
| conda-pytorchbot-token: | ||
| required: true | ||
| description: Conda PyTorchBot token | ||
| jobs: | ||
| build: | ||
| runs-on: ubuntu-22.04 | ||
| container: | ||
| image: continuumio/miniconda3:4.12.0 | ||
| steps: | ||
| - name: Checkout repository test-infra | ||
| uses: actions/checkout@v3 | ||
| with: | ||
| repository: pytorch/test-infra | ||
| ref: ${{ github.ref }} | ||
| - name: Prune binaries | ||
| env: | ||
| CHANNEl: ${{ inputs.channel }} | ||
| PACKAGES: ${{ inputs.packages }} | ||
| ANACONDA_API_TOKEN: ${{ secrets.conda-pytorchbot-token }} | ||
| run: | | ||
| set -ex | ||
| conda install -yq anaconda-client | ||
| bash ./.github/scripts/anaconda-prune/run.sh |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,33 @@ | ||
| name: anaconda-prune | ||
|
|
||
| on: | ||
| schedule: | ||
| - cron: 45 1,7,13,19 * * * | ||
| pull_request: | ||
| paths: | ||
| - .github/workflows/anaconda-prune.yml | ||
| - .github/workflows/_prune-anaconda-packages.yml | ||
| workflow_dispatch: | ||
|
|
||
| concurrency: | ||
| group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.sha }}-${{ github.event_name == 'workflow_dispatch' }} | ||
| cancel-in-progress: true | ||
|
|
||
| jobs: | ||
| anaconda-prune-pytorch-nightly: | ||
| name: anaconda-prune-pytorch-nightly | ||
| uses: ./.github/workflows/_prune-anaconda-packages.yml | ||
| with: | ||
| packages: "pytorch torchvision torchaudio torchtext torchdata ignite torchcsprng torchserve torcharrow" | ||
| channel: pytorch-nightly | ||
| secrets: | ||
| conda-pytorchbot-token: ${{ secrets.CONDA_PYTORCHBOT_TOKEN }} | ||
|
|
||
| anaconda-prune-pytorch-test: | ||
| name: anaconda-prune-pytorch-test | ||
| uses: ./.github/workflows/_prune-anaconda-packages.yml | ||
| with: | ||
| packages: "pytorch torchvision torchaudio torchtext torchdata ignite torchcsprng" | ||
| channel: pytorch-test | ||
| secrets: | ||
| conda-pytorchbot-token: ${{ secrets.CONDA_PYTORCHBOT_TOKEN }} |
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.