Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Task] Publish Prism binary (os + architecture combos) artifacts on Beam Release. #29697

Closed
Tracked by #28187
lostluck opened this issue Dec 9, 2023 · 2 comments
Closed
Tracked by #28187

Comments

@lostluck
Copy link
Contributor

lostluck commented Dec 9, 2023

Eventually we should publish OS + Architecture specific binaries for Prism in order to make it easier to use with released versions, and avoid requiring users compile it locally with Go (if possible). Though, the "build locally" solution can always remain a fallback, in particular for dev use, or novel OS + Arch combinations we don't release.

Maintained Actions exist: https://github.com/marketplace/actions/go-release-binaries (would need to be INFRA approved, if not already) (More investigation into alternatives is recommended).

Otherwise, programmatic access exists for uploading to a release, and then downloading release assets: https://docs.github.com/en/rest/releases/assets?apiVersion=2022-11-28#upload-a-release-asset

https://docs.github.com/en/rest/releases/assets?apiVersion=2022-11-28#get-a-release-asset

There's no limit/quotas associate with github assets, outside of the 2GB per file: https://docs.github.com/en/repositories/releasing-projects-on-github/about-releases#storage-and-bandwidth-quotas

Then ultimately incorporating it into the release workflow for use with RCs and then the final releases.

@lostluck lostluck assigned lostluck and unassigned lostluck Dec 9, 2023
@lostluck lostluck changed the title Publish Prism binary (os + architecture combos) artifacts on Beam Release. [task] Publish Prism binary (os + architecture combos) artifacts on Beam Release. Dec 9, 2023
@lostluck lostluck changed the title [task] Publish Prism binary (os + architecture combos) artifacts on Beam Release. [Task] Publish Prism binary (os + architecture combos) artifacts on Beam Release. Dec 9, 2023
@lostluck lostluck added go infra P2 release github_actions Pull requests that update GitHub Actions code and removed github_actions Pull requests that update GitHub Actions code labels Dec 9, 2023
@lostluck
Copy link
Contributor Author

Investigated goreleaser https://goreleaser.com/

And it would be perfect if Beam had a pure go project repo, as our needs aren't too complicated. But it can do complicated, and handle basically everything to do with a release, from change log handling, to release notes, uploading a release to GitHub, building and publishing docker images, signing, etc.

The part that fails to work for the current beam repo is that the Go SDK uses a sub version tag to work with Go Modules properly (that is it's released under the tags "sdks/v1.2.3", to be published properly. That requires the Pro version of go-releaser which requires a subscription fee.

That would enable even more sophisticated workflows, (like re-using the RC candidate binaries for the final tag release), but that would be overly optimizing. However, we would be better off leveraging existing Github mechanisms instead (eg. Create a release on the RC, but mark it as a pre-release).

We can revisit this if we move the Go SDK to it's own repo in some future or alternative version of beam.

@lostluck
Copy link
Contributor Author

Per that linked PR, we ended up just building an action for it ourselves.

Ultimately, we have the zipped files uploaded, along with signatures and hashes, so it fully matches what we upload to the Apache SVN repository.

The binary zips we're putting up in GitHub release artifacts will be at URLs with the following pattern.
http://github.com/apache/beam/releases/download/RELEASE/apache_beam-RELEASE-prism-OS-ARCH.zip

Note: Beam versions have a v prefix, eg v2.56.0 which needs to match in the URL.

To actually finish #28187 we need to have Java (#31402) and Python (#31403) download the binaries from GitHub. For that just using whatever native library that can make the http request and handle the various redirects GitHub puts us through.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant