build: update s5cmd to version v2.2.1 #12898

obnoxxx · 2023-09-13T10:07:30Z

This is the latest available version.
The previously used version v2.0.0 was very outdated.

Fixes: #12883

Description of your changes:

This updates the version of s5cmd built into the container(s) to the latest available version v2.2.1

Which issue is resolved by this Pull Request:
Resolves #12883

Checklist:

Commit Message Formatting: Commit titles and messages follow guidelines in the developer guide.
Reviewed the developer guide on Submitting a Pull Request
Pending release notes updated with breaking and/or notable changes for the next minor release.
Documentation has been updated, if necessary.
Unit tests have been added, if necessary.
Integration tests have been added, if necessary.

obnoxxx · 2023-09-13T10:11:12Z

This PR provides a change that should update the s5cmd version as requested in #12883 . I wanted to add a test to verify the effect but could not yet come up with a simple way to test locally with podman run instead of requiring a full k8s deployment

travisn

LGTM, will merge after you've been able to test it locally.

obnoxxx · 2023-09-14T08:55:52Z

@travisn wrote:

LGTM, will merge after you've been able to test it locally.

Thanks! thanks to all your help yesterday, I just managed to verify locally in minikube:

rook-ceph exec  rook-ceph-operator-b89ccd545-rhh8s -- s5cmd version
v2.2.1-be63977

obnoxxx · 2023-09-14T09:55:48Z

@travisn : I also added a check to the canary integration test workflow to ensure this doesn't regress

.github/workflows/canary-integration-test.yml

subhamkrai

commit title, huild h->b

obnoxxx · 2023-09-14T12:46:10Z

@subhamkrai wrote:

commit title, huild h->b

done, thanks!

.github/workflows/canary-integration-test.yml

travisn · 2023-09-14T18:28:58Z

.github/workflows/canary-integration-test.yml

          timeout 60 sh -c "until kubectl -n rook-ceph exec $toolbox -- curl --silent --show-error ${mgr_raw%%:*}:9283; do echo 'waiting for mgr prometheus exporter to be ready' && sleep 1; done"

+      - name: check s5cmd version in the toolbox image


Need to remove this section of the test since we moved it down to line 319. I see the new test in that section is passing, but this section is failing since we know the toolbox-operator-image is not running.

@travisn wrote:

Need to remove this section of the test since we moved it down to line 319. I see the new test in that section is passing, but this section is failing since we know the toolbox-operator-image is not running.

what is it that you are suggesting yo remove? lines 77 -- 79 ?

I tried something different now, since I saw the attempt at a diagnostic output of the s5cmd version failing because it could not find the binary ...

We need to remove this whole section, lines 79-88. This CI test runs the different toolbox. The other CI test does run the toolbox operator image so we only can test it on lines 319-328.

@travisn wrote:

We need to remove this whole section, lines 79-88. This CI test runs the different toolbox. The other CI test does run the toolbox operator image so we only can test it on lines 319-328.

got it, thanks. removed.

travisn · 2023-09-15T13:57:29Z

.github/workflows/canary-integration-test.yml

@@ -73,6 +73,7 @@ jobs:
          toolbox=$(kubectl get pod -l app=rook-ceph-tools -n rook-ceph -o jsonpath='{.items[*].metadata.name}')
          timeout 15 sh -c "until kubectl -n rook-ceph exec $toolbox -- ceph mgr dump -f json|jq --raw-output .active_addr|grep -Eosq \"(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\" ; do sleep 1 && echo 'waiting for the manager IP to be available'; done"
          mgr_raw=$(kubectl -n rook-ceph exec $toolbox -- ceph mgr dump -f json|jq --raw-output .active_addr)
+


revert this blank line?

done, thanks!

.github/workflows/canary-integration-test.yml

travisn · 2023-09-15T13:58:14Z

.github/workflows/canary-integration-test.yml

+          s5cmd_version="$(kubectl -n rook-ceph exec ${toolbox} -- /ust/local/bin/s5cmd version)"
+          echo ${s5cmd_version} | grep -q "^v2.2.1"  || {
+            echo " Error: the version of s5cmd version in the  toolbox is not the expected v2.2.1 but ${s5cmd_version}"
+


nit: remove blank line

done, thanks

This is the latest available version. The previously used version v2.0.0 was very outdated. Fixes: rook#12883 Co-authored-by: subham rai subhamkrai <srai@redhat.com> Signed-off-by: Michael Adam <obnox@samba.org> Signed-off-by: Subham Rai subhamkrai <srai@redhat.com> signed-off-by: Travis Nielsen <tnielsen@redhat.com> Co-authored-by: Travis Nielsen <tnielsen@redhat.com>

obnoxxx · 2023-09-15T14:20:39Z

@travisn I think I have addressed all your requests now. Let's see how the check goes this time ...

obnoxxx · 2023-09-15T14:45:02Z

@subhamkrai I think all your requests are addressed and the new test is finally passing! :-)

Please re-review

build: update s5cmd to version v2.2.1 (backport #12898)

obnoxxx mentioned this pull request Sep 13, 2023

s5cmd in the toolbox is downrev #12883

Closed

parth-gr approved these changes Sep 13, 2023

View reviewed changes

obnoxxx requested a review from travisn September 13, 2023 12:50

travisn added the backport-release-1.12 label Sep 13, 2023

travisn reviewed Sep 13, 2023

View reviewed changes

obnoxxx force-pushed the update-s5cmd branch from 8225687 to befe2b1 Compare September 14, 2023 09:54

obnoxxx requested a review from subhamkrai September 14, 2023 10:01

subhamkrai reviewed Sep 14, 2023

View reviewed changes

.github/workflows/canary-integration-test.yml Outdated Show resolved Hide resolved

obnoxxx force-pushed the update-s5cmd branch 4 times, most recently from 23af411 to 0ac473d Compare September 14, 2023 11:05

subhamkrai requested changes Sep 14, 2023

View reviewed changes

obnoxxx force-pushed the update-s5cmd branch from 0ac473d to c99263f Compare September 14, 2023 12:45

subhamkrai reviewed Sep 14, 2023

View reviewed changes

.github/workflows/canary-integration-test.yml Outdated Show resolved Hide resolved

subhamkrai reviewed Sep 14, 2023

View reviewed changes

.github/workflows/canary-integration-test.yml Outdated Show resolved Hide resolved

obnoxxx force-pushed the update-s5cmd branch 3 times, most recently from d5e6af2 to f59e919 Compare September 14, 2023 16:27

travisn reviewed Sep 14, 2023

View reviewed changes

obnoxxx force-pushed the update-s5cmd branch 3 times, most recently from 0b7eccf to 2914395 Compare September 15, 2023 13:21

travisn requested changes Sep 15, 2023

View reviewed changes

obnoxxx force-pushed the update-s5cmd branch from 2914395 to b7b8978 Compare September 15, 2023 14:10

obnoxxx force-pushed the update-s5cmd branch from 8f37949 to cef43e1 Compare September 15, 2023 14:17

travisn approved these changes Sep 15, 2023

View reviewed changes

obnoxxx requested a review from subhamkrai September 15, 2023 14:43

subhamkrai approved these changes Sep 15, 2023

View reviewed changes

travisn merged commit 0bca2ce into rook:master Sep 15, 2023
50 checks passed

mergify bot mentioned this pull request Sep 15, 2023

build: update s5cmd to version v2.2.1 (backport #12898) #12910

Merged

travisn added a commit that referenced this pull request Sep 15, 2023

Merge pull request #12910 from rook/mergify/bp/release-1.12/pr-12898

d8b736e

build: update s5cmd to version v2.2.1 (backport #12898)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

build: update s5cmd to version v2.2.1 #12898

build: update s5cmd to version v2.2.1 #12898

obnoxxx commented Sep 13, 2023 •

edited

obnoxxx commented Sep 13, 2023

travisn left a comment

obnoxxx commented Sep 14, 2023

obnoxxx commented Sep 14, 2023 •

edited

subhamkrai left a comment

obnoxxx commented Sep 14, 2023

travisn Sep 14, 2023

obnoxxx Sep 15, 2023

obnoxxx Sep 15, 2023

travisn Sep 15, 2023

obnoxxx Sep 15, 2023

travisn Sep 15, 2023

obnoxxx Sep 15, 2023

travisn Sep 15, 2023

obnoxxx Sep 15, 2023

obnoxxx commented Sep 15, 2023

obnoxxx commented Sep 15, 2023

		timeout 60 sh -c "until kubectl -n rook-ceph exec $toolbox -- curl --silent --show-error ${mgr_raw%%:*}:9283; do echo 'waiting for mgr prometheus exporter to be ready' && sleep 1; done"

		- name: check s5cmd version in the toolbox image

build: update s5cmd to version v2.2.1 #12898

build: update s5cmd to version v2.2.1 #12898

Conversation

obnoxxx commented Sep 13, 2023 • edited

obnoxxx commented Sep 13, 2023

travisn left a comment

Choose a reason for hiding this comment

obnoxxx commented Sep 14, 2023

obnoxxx commented Sep 14, 2023 • edited

subhamkrai left a comment

Choose a reason for hiding this comment

obnoxxx commented Sep 14, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

obnoxxx commented Sep 15, 2023

obnoxxx commented Sep 15, 2023

obnoxxx commented Sep 13, 2023 •

edited

obnoxxx commented Sep 14, 2023 •

edited