Join GitHub today
GitHub is home to over 50 million developers working together to host and review code, manage projects, and build software together.
Sign upConvert nightly upload script to Python #16565
Merged
Conversation
aneeshusa
force-pushed the
aneeshusa:move-upload-nightly-script-to-python
branch
from
c775e3d
to
c102ba6
This was referenced Apr 21, 2017
etc/ci/upload_nightly
Outdated
| package_upload_path, | ||
| ]) | ||
| subprocess.check_call([ | ||
| 'echo', |
This comment has been minimized.
This comment has been minimized.
etc/ci/upload_nightly
Outdated
| with tempfile.TemporaryDirectory(prefix='homebrew-servo') as tmp_dir: | ||
| def call_git(cmd, **kwargs): | ||
| subprocess.check_call( | ||
| ['echo', 'git', '-C', tmp_dir] + cmd, |
This comment has been minimized.
This comment has been minimized.
|
s3cmd apparently is not (well) supported on Windows, so I'll rework this to use a different tool. |
aneeshusa
changed the title
|
Can we just use the aws cli tool directly? |
|
@aneeshusa I'm happy to drive this across the line if you are busy, btw. |
|
@metajack I would appreciate that! Here's where I was thinking of taking this:
This should hopefully sidestep any Windows compatibility issues. |
aneeshusa
force-pushed the
aneeshusa:move-upload-nightly-script-to-python
branch
from
c102ba6
to
1b21126
|
I've done the steps from my last comment and this is ready for review. I did some light smoke testing, but haven't tested actually uploading to S3 or the |
|
Is this still a WIP or can that tag be removed? |
aneeshusa
changed the title
|
It looks to me like this is good to go. Should we r+ and then check to see what happens on the nightly builds? |
Now that MinGW and MSYS have been removed from the Windows builders, bash is not available to run the previous upload_nightlies.sh script. Convert the script to Python 2 for cross-platform support. Additionally, switch to the `boto3` library for easy uploading without needing to install `s3cmd`, and move the code into mach for easy `boto3` installation as the new `./mach upload-nightly` command. Also, hard-code the paths to the packages instead of using globs to look for them, as the paths are static. (The paths used to contain timestamps, but we now insert timestamps when uploading to S3 to improve reproducibility.)
aneeshusa
force-pushed the
aneeshusa:move-upload-nightly-script-to-python
branch
from
1b21126
to
043f78d
|
@bors-servo r+ |
|
|
|
Great work @aneeshusa. This looks like a huge improvement. |
bors-servo
added a commit
that referenced
this pull request
May 22, 2017
…, r=metajack Convert nightly upload script to Python Now that MinGW and MSYS have been removed from the Windows builders, bash is not available to run the previous upload_nightlies.sh script. Convert the script to Python 2 for cross-platform support. Additionally, switch to the `boto3` library for easy uploading without needing to install `s3cmd`, and move the code into mach for easy `boto3` installation as the new `./mach upload-nightly` command. Also, hard-code the paths to the packages instead of using globs to look for them, as the paths are static. (The paths used to contain timestamps, but we now insert timestamps when uploading to S3 to improve reproducibility.) <!-- Please describe your changes on the following line: --> --- <!-- Thank you for contributing to Servo! Please replace each `[ ]` by `[X]` when the step is complete, and replace `__` with appropriate data: --> - [ ] `./mach build -d` does not report any errors - [x] `./mach test-tidy` does not report any errors - [x] These changes fix #16560 (github issue number if applicable). <!-- Either: --> - [ ] There are tests for these changes OR - [x] These changes do not require tests because Buildbot will test them. <!-- Also, please make sure that "Allow edits from maintainers" checkbox is checked, so that we can help you if you get stuck somewhere along the way.--> <!-- Pull requests that do not address these steps are welcome, but they will require additional verification as part of the review process. --> <!-- Reviewable:start --> --- This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/servo/servo/16565) <!-- Reviewable:end -->
|
|
bors-servo
added a commit
to servo/saltfs
that referenced
this pull request
May 24, 2017
…y, r=jdm Provide secrets for `./mach upload-nightly` Also remove s3cmd related code, as we now use boto3 for uploads. Goes along with servo/servo#16565. <!-- Reviewable:start --> --- This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/servo/saltfs/675) <!-- Reviewable:end -->
bors-servo
added a commit
that referenced
this pull request
May 30, 2017
… r=larsbergstrom Fix `./mach upload-nightly macbrew` <!-- Please describe your changes on the following line: --> Fix a few issues with the Homebrew repository updating. Follow-up to #16565; should finish off #17045. Lightly smoke-tested locally this time. --- <!-- Thank you for contributing to Servo! Please replace each `[ ]` by `[X]` when the step is complete, and replace `__` with appropriate data: --> - [ ] `./mach build -d` does not report any errors - [x] `./mach test-tidy` does not report any errors - [ ] These changes fix #__ (github issue number if applicable). <!-- Either: --> - [ ] There are tests for these changes OR - [x] These changes do not require tests because they will be tested on the builders <!-- Also, please make sure that "Allow edits from maintainers" checkbox is checked, so that we can help you if you get stuck somewhere along the way.--> <!-- Pull requests that do not address these steps are welcome, but they will require additional verification as part of the review process. --> <!-- Reviewable:start --> --- This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/servo/servo/17085) <!-- Reviewable:end -->
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
aneeshusa commentedApr 21, 2017
•
edited
Now that MinGW and MSYS have been removed from the Windows builders,
bash is not available to run the previous upload_nightlies.sh script.
Convert the script to Python 2 for cross-platform support.
Additionally, switch to the
boto3library for easy uploadingwithout needing to install
s3cmd,and move the code into mach for easy
boto3installationas the new
./mach upload-nightlycommand.Also, hard-code the paths to the packages instead of using
globs to look for them, as the paths are static.
(The paths used to contain timestamps,
but we now insert timestamps when uploading to S3
to improve reproducibility.)
./mach build -ddoes not report any errors./mach test-tidydoes not report any errorsThis change is