Adds workflows to build release binaries and push to S3 by roshkhatri · Pull Request #315 · valkey-io/valkey

roshkhatri · 2024-04-12T19:03:32Z

Adds workflows to build Valkey binaries and push to S3 to make it available to download from the website

I have commented the macOS build code as we need code-sign the binaries for Mac and need more info on that.

The Workflows can be triggered by pushing a release to the repo and the other option is manually by one of the Maintainers. For example:

Once the workflow triggers, it will generate a matrix of Jobs for the platforms we need to build from utils/releasetools/build-config.json and then the respective Jobs are triggered. These jobs make Valkey with respect to the platform binaries we want to release and would push to a private S3 bucket.

Signed-off-by: Roshan Khatri <rvkhatri@amazon.com>

hpatro

LGTM. How did we test this?

roshkhatri · 2024-04-13T03:13:52Z

How did we test this?

I have tested the binaries on my ec2 and the uploads are tested by pushing the binaries to private s3 bucket.

If we want to change any configurations of how we build we can make changes to the workflows to adapt that.

Signed-off-by: Roshan Khatri <rvkhatri@amazon.com>

roshkhatri · 2024-04-15T23:34:02Z

Now this Workflow will be triggered when a release has been published

Also updated the top comment.

Signed-off-by: Roshan Khatri <rvkhatri@amazon.com>

madolson · 2024-04-16T03:00:43Z

+        with:
+          arch: aarch64
+          distro: ${{matrix.distro.target}}
+          install: apt-get update && apt-get install -y build-essential libssl-dev libjemalloc-dev


Why are you including libejemalloc-dev? We aren't linking against it.

madolson

Took a bit of a deeper look, The code looks mostly fine, but had some minor comments.

madolson · 2024-04-16T04:27:55Z

+
+      - name: Configure AWS credentials
+        run: |
+          aws configure set region us-east-1


I think the bucket was in us-west-2? Does this even matter?

But I will change to the correct location

madolson · 2024-04-16T04:28:09Z

+jobs:
+  build-valkey:
+    # Capture source tarball and generate checksum for it
+    name: Build  package ${{ matrix.distro.target }} ${{ matrix.distro.arch }}


Suggested change

name: Build package ${{ matrix.distro.target }} ${{ matrix.distro.arch }}

name: Build package ${{ matrix.distro.target }} ${{ matrix.distro.arch }}

madolson · 2024-04-16T04:28:33Z

@@ -0,0 +1,79 @@
+name: Reusable workflow to build binary packages into S3 bucket


This isn't really that re-usable if it's only used for linux.

This can also be used to build Nightly/Weekly build. I will remove it from the description for now, might add it later.

madolson · 2024-04-16T04:32:02Z

+          ref: ${{ inputs.version }}
+
+      - name: Install dependencies
+        run: sudo apt-get update && sudo apt-get install -y build-essential libssl-dev jq wget


Seems weird we wouldn't install aws cli here as well.

madolson · 2024-04-16T04:33:09Z

+        run: sudo apt-get update && sudo apt-get install -y build-essential libssl-dev jq wget
+
+      - name: Set JEMALLOC Flags
+        run: JEMALLOC_CONFIGURE_OPTS="--with-lg-page=16"


Why are we configuring the lg-page to be 16? This will use more memory.

Yes just realized it may also increase chances of fragmentation I guess. Removing this.

Signed-off-by: Roshan Khatri <rvkhatri@amazon.com>

madolson

I think it's okay to start with, then we can test it with 7.2.5 release.

[related to](valkey-io#230) Adds workflows to build Valkey binaries and push to S3 to make it available to download from the website The Workflows can be triggered by pushing a release to the repo and the other option is manually by one of the Maintainers. Once the workflow triggers, it will generate a matrix of Jobs for the platforms we need to build from `utils/releasetools/build-config.json` and then the respective Jobs are triggered. These jobs make Valkey with respect to the platform binaries we want to release and would push to a private S3 bucket. --------- Signed-off-by: Roshan Khatri <rvkhatri@amazon.com>

…315) (#857) [related to](#230) Adds workflows to build Valkey binaries and push to S3 to make it available to download from the website The Workflows can be triggered by pushing a release to the repo and the other option is manually by one of the Maintainers. Once the workflow triggers, it will generate a matrix of Jobs for the platforms we need to build from `utils/releasetools/build-config.json` and then the respective Jobs are triggered. These jobs make Valkey with respect to the platform binaries we want to release and would push to a private S3 bucket. --------- Signed-off-by: Roshan Khatri <rvkhatri@amazon.com>

roshkhatri added 2 commits April 12, 2024 19:07

Adds workflows to build binaries and push to S3

e2fb8f5

Signed-off-by: Roshan Khatri <rvkhatri@amazon.com>

Merge branch 'unstable' into revamp-workflows

c3e9a10

Signed-off-by: Roshan Khatri <rvkhatri@amazon.com>

roshkhatri force-pushed the revamp-workflows branch from 4a41c8f to c3e9a10 Compare April 12, 2024 19:11

zuiderkwast reviewed Apr 12, 2024

View reviewed changes

Comment thread .github/workflows/build-release-packages.yml Outdated

hpatro reviewed Apr 12, 2024

View reviewed changes

Comment thread .github/workflows/call-build-linux-arm-packages.yml Outdated

Comment thread .github/workflows/call-build-linux-arm-packages.yml Outdated

hpatro reviewed Apr 12, 2024

View reviewed changes

Comment thread utils/releasetools/build-config.json

Comment thread utils/releasetools/build-config.json Outdated

roshkhatri added 2 commits April 13, 2024 03:24

addressing feedbacks

092ae89

Signed-off-by: Roshan Khatri <rvkhatri@amazon.com>

Change trigger type to run workflow on release published

deeb9a8

Signed-off-by: Roshan Khatri <rvkhatri@amazon.com>

roshkhatri force-pushed the revamp-workflows branch from d7406e8 to deeb9a8 Compare April 15, 2024 23:26

hpatro approved these changes Apr 16, 2024

View reviewed changes

madolson reviewed Apr 16, 2024

View reviewed changes

Comment thread .github/workflows/build-release-packages.yml Outdated

roshkhatri added 2 commits April 16, 2024 00:34

Removes Mac related code

d45e88f

Signed-off-by: Roshan Khatri <rvkhatri@amazon.com>

Uses jemalloc for arm build

8f95cff

Signed-off-by: Roshan Khatri <rvkhatri@amazon.com>

madolson reviewed Apr 16, 2024

View reviewed changes

Addressing Feedback.

b0f23aa

Signed-off-by: Roshan Khatri <rvkhatri@amazon.com>

bjosv reviewed Apr 16, 2024

View reviewed changes

Comment thread .github/workflows/call-build-linux-x86-packages.yml Outdated

Comment thread .github/actions/generate-package-build-matrix/action.yml Outdated

Comment thread .github/workflows/build-release-packages.yml

bjosv reviewed Apr 16, 2024

View reviewed changes

Comment thread .github/workflows/call-build-linux-arm-packages.yml Outdated

addressing feedback

e646d29

Signed-off-by: Roshan Khatri <rvkhatri@amazon.com>

roshkhatri requested review from bjosv and madolson April 16, 2024 13:31

Restricting permissions to read on top level

2631109

Signed-off-by: Roshan Khatri <rvkhatri@amazon.com>

madolson approved these changes Apr 16, 2024

View reviewed changes

madolson merged commit b16e647 into valkey-io:unstable Apr 16, 2024

zuiderkwast mentioned this pull request Jun 26, 2024

[NEW] Please publish Debian artifacts #680

Open

flyinprogrammer mentioned this pull request Aug 30, 2024

[feat] add Ubuntu 24.04 Noble package support #971

Merged

roshkhatri deleted the revamp-workflows branch September 3, 2024 16:18

	name: Build package ${{ matrix.distro.target }} ${{ matrix.distro.arch }}
	name: Build package ${{ matrix.distro.target }} ${{ matrix.distro.arch }}

		@@ -0,0 +1,79 @@
		name: Reusable workflow to build binary packages into S3 bucket

Conversation

roshkhatri commented Apr 12, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

hpatro left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

roshkhatri commented Apr 13, 2024

Uh oh!

roshkhatri commented Apr 15, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

madolson left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

madolson left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

roshkhatri commented Apr 12, 2024 •

edited

Loading

roshkhatri commented Apr 15, 2024 •

edited

Loading