Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ci: move most of the prepare config into scripts #65202

Merged
merged 25 commits into from
Oct 28, 2019

Conversation

pietroalbini
Copy link
Member

This PR moves most of the configuration from the CI yamls into bash scripts, driven by a small Python script (which understands and emulates the two ##vso[ commands we use).

There are two reasons why we'd want to do this:

  • Being able to prepare the build environment locally by just running src/ci/prepare.py simplifies a lot setting up a local VM similar to CI (software pre-installed in the CI images won't be prepared, but it's a start anyway).
  • When we'll switch to GitHub Actions we'll need to either duplicate code in multiple workflows or write a preprocessor. Having all the prepare steps in a single one is going to simplify the implementation of both options.

Along with the move I did a few changes to the actual scripts:

  • Mirrored all the remaining external URLs we download (except chocolatey) to the rust-lang-ci-mirrors bucket, to increase reliability and reduce the chance of supply chain attacks. I didn't audit and mirror the CI scripts outside this PR though.
  • Extracted CI-specific behavior (like issuing ##vso[ commands and detecting the host platform) into shared.sh and included it in most of the scripts. This way a switch to another CI provider will be less painful.

It's possible (and easier) to review this commit-by-commit.
r? @alexcrichton
cc @rust-lang/infra

@rust-highfive rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Oct 8, 2019
@rust-highfive
Copy link
Collaborator

The job x86_64-gnu-llvm-6.0 of your PR failed (pretty log, raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.
2019-10-08T11:17:05.0285704Z ##[command]git remote add origin https://github.com/rust-lang/rust
2019-10-08T11:17:05.0495783Z ##[command]git config gc.auto 0
2019-10-08T11:17:05.0588106Z ##[command]git config --get-all http.https://github.com/rust-lang/rust.extraheader
2019-10-08T11:17:05.0659617Z ##[command]git config --get-all http.proxy
2019-10-08T11:17:05.0798391Z ##[command]git -c http.extraheader="AUTHORIZATION: basic ***" fetch --force --tags --prune --progress --no-recurse-submodules --depth=2 origin +refs/heads/*:refs/remotes/origin/* +refs/pull/65202/merge:refs/remotes/pull/65202/merge
---
2019-10-08T11:23:47.1619789Z    Compiling serde_json v1.0.40
2019-10-08T11:23:48.9290740Z    Compiling tidy v0.1.0 (/checkout/src/tools/tidy)
2019-10-08T11:23:59.9722143Z     Finished release [optimized] target(s) in 1m 28s
2019-10-08T11:23:59.9800235Z tidy check
2019-10-08T11:24:00.4097909Z tidy error: /checkout/src/ci/scripts/install-clang.sh:41: line longer than 100 chars
2019-10-08T11:24:00.4098515Z tidy error: /checkout/src/ci/scripts/install-innosetup.sh:14: line longer than 100 chars
2019-10-08T11:24:00.4099018Z tidy error: /checkout/src/ci/scripts/enable-docker-ipv6.sh:12: line longer than 100 chars
2019-10-08T11:24:00.4099250Z tidy error: /checkout/src/ci/scripts/install-ninja.sh:11: line longer than 100 chars
2019-10-08T11:24:00.4099474Z tidy error: /checkout/src/ci/scripts/install-mingw.sh:41: line longer than 100 chars
2019-10-08T11:24:02.1016778Z some tidy checks failed
2019-10-08T11:24:02.1022892Z 
2019-10-08T11:24:02.1022892Z 
2019-10-08T11:24:02.1023925Z command did not execute successfully: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-tools-bin/tidy" "/checkout/src" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/cargo" "--no-vendor"
2019-10-08T11:24:02.1024041Z 
2019-10-08T11:24:02.1024095Z 
2019-10-08T11:24:02.1029354Z failed to run: /checkout/obj/build/bootstrap/debug/bootstrap test src/tools/tidy
2019-10-08T11:24:02.1029663Z Build completed unsuccessfully in 0:01:31
2019-10-08T11:24:02.1029663Z Build completed unsuccessfully in 0:01:31
2019-10-08T11:24:02.1086864Z == clock drift check ==
2019-10-08T11:24:02.1110882Z   local time: Tue Oct  8 11:24:02 UTC 2019
2019-10-08T11:24:02.2600013Z   network time: Tue, 08 Oct 2019 11:24:02 GMT
2019-10-08T11:24:02.2603874Z == end clock drift check ==
2019-10-08T11:24:03.6450517Z ##[error]Bash exited with code '1'.
2019-10-08T11:24:03.6500949Z ##[section]Starting: Checkout
2019-10-08T11:24:03.6503139Z ==============================================================================
2019-10-08T11:24:03.6503201Z Task         : Get sources
2019-10-08T11:24:03.6503273Z Description  : Get sources from a repository. Supports Git, TfsVC, and SVN repositories.

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

src/ci/scripts/install-wix.sh Outdated Show resolved Hide resolved
@alexcrichton
Copy link
Member

@bors: r+

Thanks for this @pietroalbini!

@bors
Copy link
Contributor

bors commented Oct 10, 2019

📌 Commit 1591a8e has been approved by alexcrichton

@bors bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Oct 10, 2019
@pietroalbini
Copy link
Member Author

@bors rollup=never

At least for linux we're sure it works (the PR builders were fine), but there is a chance I forgot to copy something/typoed stuff for the other platforms, and I don't want this to affect a rollup.

@bors
Copy link
Contributor

bors commented Oct 13, 2019

⌛ Testing commit 1591a8e with merge f973dbb260d9d091ec71666190125d1df9d1abe6...

@bors
Copy link
Contributor

bors commented Oct 13, 2019

💔 Test failed - checks-azure

@rust-highfive
Copy link
Collaborator

Your PR failed (pretty log, raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.
2019-10-13T11:34:55.9566367Z do so (now or later) by using -b with the checkout command again. Example:
2019-10-13T11:34:55.9566506Z 
2019-10-13T11:34:55.9566636Z   git checkout -b <new-branch-name>
2019-10-13T11:34:55.9567164Z 
2019-10-13T11:34:55.9568177Z HEAD is now at f973dbb26 Auto merge of #65202 - pietroalbini:scriptify-ci-config, r=alexcrichton
2019-10-13T11:34:55.9941989Z ##[section]Starting: Collect CPU-usage statistics in the background
2019-10-13T11:34:56.0065799Z ==============================================================================
2019-10-13T11:34:56.0065909Z Task         : Bash
2019-10-13T11:34:56.0065980Z Description  : Run a Bash script on macOS, Linux, or Windows
---
2019-10-13T11:34:56.7293520Z [command]"C:\Program Files\Git\bin\bash.exe" --noprofile --norc -c pwd
2019-10-13T11:34:56.7694809Z /d/a/_temp
2019-10-13T11:34:56.7783167Z ========================== Starting Command Output ===========================
2019-10-13T11:34:56.7802821Z [command]"C:\Program Files\Git\bin\bash.exe" --noprofile --norc /d/a/_temp/5cfb4553-5c08-4566-bb58-77fe814c5c10.sh
2019-10-13T11:34:57.0392110Z /usr/bin/env: 'python3': No such file or directory
2019-10-13T11:34:57.0535809Z ##[error]Bash exited with code '127'.
2019-10-13T11:34:57.0550119Z ##[section]Finishing: Prepare the CI environment
2019-10-13T11:34:57.0728297Z ==============================================================================
2019-10-13T11:34:57.0728414Z Task         : Bash
2019-10-13T11:34:57.0728525Z Description  : Run a Bash script on macOS, Linux, or Windows
2019-10-13T11:34:57.0728611Z Version      : 3.151.3
---
2019-10-13T11:34:57.3550103Z [command]"C:\Program Files\Git\bin\bash.exe" --noprofile --norc -c pwd
2019-10-13T11:34:57.3904319Z /d/a/_temp
2019-10-13T11:34:57.3983344Z ========================== Starting Command Output ===========================
2019-10-13T11:34:57.4000146Z [command]"C:\Program Files\Git\bin\bash.exe" --noprofile --norc /d/a/_temp/aa2921f7-209f-41d0-ba9b-34e6e15b619f.sh
2019-10-13T11:34:57.4446083Z /d/a/_temp/aa2921f7-209f-41d0-ba9b-34e6e15b619f.sh: line 1: aws: command not found
2019-10-13T11:34:57.4502010Z ##[error]Bash exited with code '127'.
2019-10-13T11:34:57.4579257Z ##[section]Starting: Checkout
2019-10-13T11:34:57.4690366Z ==============================================================================
2019-10-13T11:34:57.4690477Z Task         : Get sources
2019-10-13T11:34:57.4690576Z Description  : Get sources from a repository. Supports Git, TfsVC, and SVN repositories.

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

@bors bors added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. labels Oct 13, 2019
@kennytm
Copy link
Member

kennytm commented Oct 13, 2019

aws: command not found on Windows.

@Centril
Copy link
Contributor

Centril commented Oct 13, 2019

@bors r- (bors sync goes badly...)

@bors bors added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Oct 13, 2019
@pietroalbini
Copy link
Member Author

This wasn't actually awscli missing, but the whole prepare step failed to run as it used #!/usr/bin/env python3, while on the Windows agents only python.exe is present. Added a new step to copy python.exe to python3.exe before executing the prepare script.

@bors r=alexcrichton

@bors
Copy link
Contributor

bors commented Oct 14, 2019

📌 Commit 8d1c3ff has been approved by alexcrichton

@bors bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Oct 14, 2019
@bors
Copy link
Contributor

bors commented Oct 14, 2019

⌛ Testing commit 8d1c3ff with merge a103c61b9ccbce6c6f65b03d349efbad6c906a35...

@rust-highfive
Copy link
Collaborator

Your PR failed (pretty log, raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.
2019-10-14T15:42:39.1411382Z do so (now or later) by using -b with the checkout command again. Example:
2019-10-14T15:42:39.1411821Z 
2019-10-14T15:42:39.1412362Z   git checkout -b <new-branch-name>
2019-10-14T15:42:39.1413401Z 
2019-10-14T15:42:39.1414956Z HEAD is now at a103c61b9 Auto merge of #65202 - pietroalbini:scriptify-ci-config, r=alexcrichton
2019-10-14T15:42:39.1774570Z ##[section]Finishing: Checkout
2019-10-14T15:42:39.1906405Z ##[section]Starting: Copy python.exe to python3.exe
2019-10-14T15:42:39.2054625Z Task         : Bash
2019-10-14T15:42:39.2054698Z Description  : Run a Bash script on macOS, Linux, or Windows
2019-10-14T15:42:39.2054786Z Version      : 3.151.3
2019-10-14T15:42:39.2054847Z Author       : Microsoft Corporation
2019-10-14T15:42:39.2054847Z Author       : Microsoft Corporation
2019-10-14T15:42:39.2054944Z Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/utility/bash
2019-10-14T15:42:39.2055051Z ==============================================================================
2019-10-14T15:42:39.4543072Z Generating script.
2019-10-14T15:42:39.4753647Z Script contents:
2019-10-14T15:42:39.4753754Z cp $(which python) $(dirname $(which python))/python3.exe
2019-10-14T15:42:39.5185261Z /d/a/_temp
2019-10-14T15:42:39.5257973Z ========================== Starting Command Output ===========================
2019-10-14T15:42:39.5266878Z [command]"C:\Program Files\Git\bin\bash.exe" --noprofile --norc /d/a/_temp/ff005c95-8054-45b4-8d20-9a4c14409d20.sh
2019-10-14T15:42:39.5266878Z [command]"C:\Program Files\Git\bin\bash.exe" --noprofile --norc /d/a/_temp/ff005c95-8054-45b4-8d20-9a4c14409d20.sh
2019-10-14T15:42:39.9225103Z ##[section]Finishing: Copy python.exe to python3.exe
2019-10-14T15:42:39.9354030Z ==============================================================================
2019-10-14T15:42:39.9354118Z Task         : Bash
2019-10-14T15:42:39.9354209Z Description  : Run a Bash script on macOS, Linux, or Windows
2019-10-14T15:42:39.9354277Z Version      : 3.151.3
---
2019-10-14T15:42:40.6946372Z ========================== Starting Command Output ===========================
2019-10-14T15:42:40.6974190Z [command]"C:\Program Files\Git\bin\bash.exe" --noprofile --norc /d/a/_temp/538bcdc1-6201-47e5-a9b8-9bf6374e2be5.sh
2019-10-14T15:42:42.5321686Z ==== Show the current environment ====
2019-10-14T15:42:42.6037975Z Traceback (most recent call last):
2019-10-14T15:42:42.6044451Z   File "src/ci/prepare.py", line 93, in <module>
2019-10-14T15:42:42.6065675Z     run(Build())
2019-10-14T15:42:42.6065818Z   File "src/ci/prepare.py", line 9, in run
2019-10-14T15:42:42.6065915Z     b.step("dump-environment.sh", "Show the current environment")
2019-10-14T15:42:42.6066030Z   File "src/ci/prepare.py", line 63, in step
2019-10-14T15:42:42.6066120Z     stdout=subprocess.PIPE, stderr=subprocess.STDOUT,
2019-10-14T15:42:42.6066247Z   File "C:\hostedtoolcache\windows\Python\3.6.8\x64\lib\subprocess.py", line 729, in __init__
2019-10-14T15:42:42.6066358Z     restore_signals, start_new_session)
2019-10-14T15:42:42.6067130Z   File "C:\hostedtoolcache\windows\Python\3.6.8\x64\lib\subprocess.py", line 1017, in _execute_child
2019-10-14T15:42:42.6067297Z     startupinfo)
2019-10-14T15:42:42.6067385Z OSError: [WinError 193] %1 is not a valid Win32 application
2019-10-14T15:42:42.6363394Z ##[error]Bash exited with code '1'.
2019-10-14T15:42:42.6572272Z ##[section]Finishing: Prepare the CI environment
2019-10-14T15:42:42.7073475Z ==============================================================================
2019-10-14T15:42:42.7073584Z Task         : Bash
2019-10-14T15:42:42.7073655Z Description  : Run a Bash script on macOS, Linux, or Windows
2019-10-14T15:42:42.7073736Z Version      : 3.151.3
---
2019-10-14T15:42:43.1372960Z [command]"C:\Program Files\Git\bin\bash.exe" --noprofile --norc -c pwd
2019-10-14T15:42:43.2024724Z /d/a/_temp
2019-10-14T15:42:43.2554441Z ========================== Starting Command Output ===========================
2019-10-14T15:42:43.2555216Z [command]"C:\Program Files\Git\bin\bash.exe" --noprofile --norc /d/a/_temp/bb4006b7-6135-4204-89cc-c438644dca5b.sh
2019-10-14T15:42:43.3650097Z /d/a/_temp/bb4006b7-6135-4204-89cc-c438644dca5b.sh: line 1: aws: command not found
2019-10-14T15:42:43.3796243Z ##[error]Bash exited with code '127'.
2019-10-14T15:42:43.3860654Z ##[section]Starting: Checkout
2019-10-14T15:42:43.3993917Z ==============================================================================
2019-10-14T15:42:43.3994044Z Task         : Get sources
2019-10-14T15:42:43.3994302Z Description  : Get sources from a repository. Supports Git, TfsVC, and SVN repositories.

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

@bors
Copy link
Contributor

bors commented Oct 14, 2019

💔 Test failed - checks-azure

@bors bors added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. labels Oct 14, 2019
@pietroalbini
Copy link
Member Author

@bors try

@bors
Copy link
Contributor

bors commented Oct 28, 2019

⌛ Trying commit af5cc3d with merge 0a65fe3...

bors added a commit that referenced this pull request Oct 28, 2019
ci: move most of the prepare config into scripts

This PR moves most of the configuration from the CI yamls into bash scripts, driven by a small Python script (which understands and emulates the two `##vso[` commands we use).

There are two reasons why we'd want to do this:

* Being able to prepare the build environment locally by just running `src/ci/prepare.py` simplifies a lot setting up a local VM similar to CI (software pre-installed in the CI images won't be prepared, but it's a start anyway).
* When we'll switch to GitHub Actions we'll need to either duplicate code in multiple workflows or write a preprocessor. Having all the prepare steps in a single one is going to simplify the implementation of both options.

Along with the move I did a few changes to the actual scripts:

* Mirrored all the remaining external URLs we download (except chocolatey) to the `rust-lang-ci-mirrors` bucket, to increase reliability and reduce the chance of supply chain attacks. I didn't audit and mirror the CI scripts outside this PR though.
* Extracted CI-specific behavior (like issuing `##vso[` commands and detecting the host platform) into `shared.sh` and included it in most of the scripts. This way a switch to another CI provider will be less painful.

It's possible (and easier) to review this commit-by-commit.
r? @alexcrichton
cc @rust-lang/infra
@bors
Copy link
Contributor

bors commented Oct 28, 2019

☀️ Try build successful - checks-azure
Build commit: 0a65fe3 (0a65fe3fd1bb2b9c4491cb5daecdd9cccca9a8a2)

@pietroalbini
Copy link
Member Author

@bors r=alexcrichton p=10 rollup=never

@bors
Copy link
Contributor

bors commented Oct 28, 2019

📌 Commit 7e05123 has been approved by alexcrichton

@bors bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Oct 28, 2019
@bors
Copy link
Contributor

bors commented Oct 28, 2019

⌛ Testing commit 7e05123 with merge b497e18...

bors added a commit that referenced this pull request Oct 28, 2019
ci: move most of the prepare config into scripts

This PR moves most of the configuration from the CI yamls into bash scripts, driven by a small Python script (which understands and emulates the two `##vso[` commands we use).

There are two reasons why we'd want to do this:

* Being able to prepare the build environment locally by just running `src/ci/prepare.py` simplifies a lot setting up a local VM similar to CI (software pre-installed in the CI images won't be prepared, but it's a start anyway).
* When we'll switch to GitHub Actions we'll need to either duplicate code in multiple workflows or write a preprocessor. Having all the prepare steps in a single one is going to simplify the implementation of both options.

Along with the move I did a few changes to the actual scripts:

* Mirrored all the remaining external URLs we download (except chocolatey) to the `rust-lang-ci-mirrors` bucket, to increase reliability and reduce the chance of supply chain attacks. I didn't audit and mirror the CI scripts outside this PR though.
* Extracted CI-specific behavior (like issuing `##vso[` commands and detecting the host platform) into `shared.sh` and included it in most of the scripts. This way a switch to another CI provider will be less painful.

It's possible (and easier) to review this commit-by-commit.
r? @alexcrichton
cc @rust-lang/infra
@bors
Copy link
Contributor

bors commented Oct 28, 2019

☀️ Test successful - checks-azure
Approved by: alexcrichton
Pushing b497e18 to master...

@bors bors added the merged-by-bors This PR was explicitly merged by bors. label Oct 28, 2019
@bors bors merged commit 7e05123 into rust-lang:master Oct 28, 2019
@pietroalbini pietroalbini deleted the scriptify-ci-config branch October 28, 2019 21:09
@pietroalbini
Copy link
Member Author

Finally 🎉

tmandry added a commit to tmandry/rust that referenced this pull request Oct 30, 2019
…=alexcrichton

Upload toolstates.json to rust-lang-ci2

This PR does two things:

* Following up with rust-lang#65202, it migrates deploying artifacts to CI in a script. Both uploading release artifacts and CPU stats were merged into the same script, designing it to be easily extended.
* Uploads the toolstate JSON to `rust-lang-ci2` along with the release artifacts, both for Linux and Windows. This is needed because @RalfJung wants to stop shipping MIRI when its tests are failing, and the toolstate repo doesn't have entries for each commit. Having the toolstate data (just for that specific commit) on `rust-lang-ci2` will simplify the code a lot.

r? @alexcrichton
cc @RalfJung
bors added a commit that referenced this pull request Oct 30, 2019
Upload toolstates.json to rust-lang-ci2

This PR does two things:

* Following up with #65202, it migrates deploying artifacts to CI in a script. Both uploading release artifacts and CPU stats were merged into the same script, designing it to be easily extended.
* Uploads the toolstate JSON to `rust-lang-ci2` along with the release artifacts, both for Linux and Windows. This is needed because @RalfJung wants to stop shipping MIRI when its tests are failing, and the toolstate repo doesn't have entries for each commit. Having the toolstate data (just for that specific commit) on `rust-lang-ci2` will simplify the code a lot.

r? @alexcrichton
cc @RalfJung
bors added a commit that referenced this pull request Oct 30, 2019
Upload toolstates.json to rust-lang-ci2

This PR does two things:

* Following up with #65202, it migrates deploying artifacts to CI in a script. Both uploading release artifacts and CPU stats were merged into the same script, designing it to be easily extended.
* Uploads the toolstate JSON to `rust-lang-ci2` along with the release artifacts, both for Linux and Windows. This is needed because @RalfJung wants to stop shipping MIRI when its tests are failing, and the toolstate repo doesn't have entries for each commit. Having the toolstate data (just for that specific commit) on `rust-lang-ci2` will simplify the code a lot.

r? @alexcrichton
cc @RalfJung
bors added a commit that referenced this pull request Oct 30, 2019
Upload toolstates.json to rust-lang-ci2

This PR does two things:

* Following up with #65202, it migrates deploying artifacts to CI in a script. Both uploading release artifacts and CPU stats were merged into the same script, designing it to be easily extended.
* Uploads the toolstate JSON to `rust-lang-ci2` along with the release artifacts, both for Linux and Windows. This is needed because @RalfJung wants to stop shipping MIRI when its tests are failing, and the toolstate repo doesn't have entries for each commit. Having the toolstate data (just for that specific commit) on `rust-lang-ci2` will simplify the code a lot.

r? @alexcrichton
cc @RalfJung
Centril added a commit to Centril/rust that referenced this pull request Oct 31, 2019
…=alexcrichton

Upload toolstates.json to rust-lang-ci2

This PR does two things:

* Following up with rust-lang#65202, it migrates deploying artifacts to CI in a script. Both uploading release artifacts and CPU stats were merged into the same script, designing it to be easily extended.
* Uploads the toolstate JSON to `rust-lang-ci2` along with the release artifacts, both for Linux and Windows. This is needed because @RalfJung wants to stop shipping MIRI when its tests are failing, and the toolstate repo doesn't have entries for each commit. Having the toolstate data (just for that specific commit) on `rust-lang-ci2` will simplify the code a lot.

r? @alexcrichton
cc @RalfJung
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
merged-by-bors This PR was explicitly merged by bors. S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

8 participants