Track problems in Envoy 1.22.0 #34

codefromthecrypt · 2022-04-22T23:20:52Z

I clicked Run Workflow for v1.22.0 and v1.22.0_debug on
https://github.com/tetratelabs/archive-envoy/actions/workflows/release.yaml

Only windows worked.

The linux image changed, so scripts failed to find the binaries
Envoy 1.22.0 hasn't made it to OS/x due to drift envoy 1.22.0 Homebrew/homebrew-core#99389

Next steps:

figure out drift in linux image and make a change to the shell script in a way that doesn't break versions before 1.22. re-run v1.22.0 and v1.22.0_debug workflows after that workaround is merged.
re-run workflow v1.22.0 once the 1.22.0 formula is fixed in homebrew

Lord-Y · 2022-04-26T13:52:30Z

Waiting for the linux version :)

codefromthecrypt · 2022-04-27T00:02:00Z

Update:

no update on homebrew as no-one has attempted to add 1.22
below are some breadcrumbs for linux image drift

Looks like the image layer changed in 1.22.0 (for linux). It is still the same in windows and <1.22 linux

$ sh -x ./bin/archive_release_version.sh envoyproxy/envoy v1.22.0 check
--snip--
using dist: envoy-v1.22.0-linux-amd64
+ set +e
+ ./bin/car_envoy.sh v1.22.0 linux amd64 list v1.22.0/envoy-v1.22.0-linux-amd64
error: usr/local/bin/envoy not found in layer
--snip--
$ sh -x ./bin/car_envoy.sh v1.22.0 linux amd64 list v1.22.0/envoy-v1.22.0-linux-amd64
--snip--
+ reference=envoyproxy/envoy:v1.22.0
+ car -t --platform linux/amd64 --created-by-pattern 'ADD linux' --strip-components 2 -qf envoyproxy/envoy:v1.22.0 usr/local/bin/envoy
error: usr/local/bin/envoy not found in layer
--snip--
$ car -t --platform linux/amd64 --strip-components 2 -vvf envoyproxy/envoy:v1.22.0 usr/local/bin/envoy
--snip--
https://index.docker.io/v2/envoyproxy/envoy/blobs/sha256:4086e842c0d1e552c62d8b94d92cb0c5a91fbccf62284f1e1c60d60a182270fa size=21351994
CreatedBy: COPY /usr/local/bin/envoy* /usr/local/bin/ # buildkit
-rwxr-xr-x	59397736	Apr 16 01:42:42	usr/local/bin/envoy
--snip--
$ car -t --platform linux/amd64 --created-by-pattern '/usr/local/bin/envoy' --strip-components 2 -qf envoyproxy/envoy:v1.22.0 
usr/local/bin/envoy

So what happened was the image layers changed in 1.22. To fix this requires a patch. I've put this here as for the sake of project continuity someone besides me needs to become familiar with the project, even if a volunteer. PR welcome!

diff --git a/bin/car_envoy.sh b/bin/car_envoy.sh
index ea1a34f..05d9353 100755
--- a/bin/car_envoy.sh
+++ b/bin/car_envoy.sh
@@ -110,7 +110,15 @@ linux)
     reference=envoyproxy/envoy:${version}
   fi
 
-  ${car} --created-by-pattern 'ADD linux' --strip-components 2 -qf "${reference}" ${files}
+  # As the Envoy image is large, select a layer that includes the binary instead of extracting all layers.
+  case ${version} in
+    1.19.*|1.20.*|1.21.*) # Envoy doesn't release old versions
+      ${car} --created-by-pattern 'ADD linux' --strip-components 2 -qf "${reference}" ${files}
+      ;;
+    *) # 1.22.0 changed the image layout.
+      ${car} --created-by-pattern '/usr/local/bin/envoy' --strip-components 2 -qf "${reference}" ${files}
+      ;;
+  esac
   ;;
 windows)
   reference=envoyproxy/envoy-windows:${version}

With the patch applied, the following should work for linux even if darwin is still waiting for someone to get homebrew working:

$ ./bin/archive_release_version.sh envoyproxy/envoy v1.21.0 check
$ ./bin/archive_release_version.sh envoyproxy/envoy v1.22.0 check

codefromthecrypt · 2022-04-27T00:14:09Z

Next action:

someone raise a PR with the patch I mentioned and test it locally as mentioned
once the above is merged, click "run workflow" here and enter the pattern as prompted. once for v1.22.0 and again for v1.22.0_debug

After that:

someone try to get the homebrew 1.22.0 working
once that is available on homebrew-core run the v1.22.0 release again.

Note: release steps are in the README, I'm just being specific.

codefromthecrypt · 2022-04-27T22:55:36Z

Tips on OS/x.

Try local first as you can do this without effecting your laptop setup: brew install --build-from-source envoy --HEAD
The actual syntax is something like brew bump-formula-pr --no-browse --no-audit --version v1.22.0
Most Envoy minor releases break and you can enlist help once you have a PR up by pinging people who usually help. Ex envoy 1.21.0 Homebrew/homebrew-core#90921

codefromthecrypt · 2022-05-08T05:11:29Z

here's a smaller diff that @cpakulski can use to work around this. I tested it.

diff --git a/bin/car_envoy.sh b/bin/car_envoy.sh
index ea1a34f..88a3176 100755
--- a/bin/car_envoy.sh
+++ b/bin/car_envoy.sh
@@ -110,7 +110,9 @@ linux)
     reference=envoyproxy/envoy:${version}
   fi
 
-  ${car} --created-by-pattern 'ADD linux' --strip-components 2 -qf "${reference}" ${files}
+  # Don't use --created-by-pattern because Envoy 1.22.0+ changed the image layer containing the binary.
+  # Note: Unlike windows, layers preceding the Envoy binary are small, so scanning through is OK.
+  ${car} --strip-components 2 -qf "${reference}" ${files}
   ;;
 windows)
   reference=envoyproxy/envoy-windows:${version}

$ ./bin/archive_release_version.sh envoyproxy/envoy v1.22.0 
archiving envoyproxy/envoy v1.22.0 released on 2022-04-15
using dist: envoy-v1.22.0-darwin-amd64
./bin/car_envoy.sh: line 98: tag: unbound variable
using dist: envoy-v1.22.0-darwin-arm64
./bin/car_envoy.sh: line 98: tag: unbound variable
using dist: envoy-v1.22.0-linux-amd64
creating envoy-v1.22.0-linux-amd64.tar.xz
using dist: envoy-v1.22.0-linux-arm64
creating envoy-v1.22.0-linux-arm64.tar.xz
using dist: envoy-v1.22.0-windows-amd64
creating envoy-v1.22.0-windows-amd64.tar.xz
using dist: envoy-v1.22.0-windows-arm64
error: windows/arm64 is not a supported platform: windows/amd64
$ tar -tvf v1.22.0//envoy-v1.22.0-linux-arm64.tar.xz
drwxr-xr-x  0 65534  65534       0 Apr 15 08:00 envoy-v1.22.0-linux-arm64/
drwxr-xr-x  0 65534  65534       0 Apr 15 08:00 envoy-v1.22.0-linux-arm64/bin/
-rwxr-xr-x  0 65534  65534 55335328 Apr 15 08:00 envoy-v1.22.0-linux-arm64/bin/envoy

Note: this only fixes the linux thing and only becomes usable when someone clicks release

codefromthecrypt · 2022-05-09T23:32:43Z

nit: let's avoid using the keyword "fixes" in PRs unless they do that as it closes the corresponding issue

NEXT STEPS:

click through release workflow for v1.22.0 and v1.22.0_debug. The former should work for linux and windows and the latter linux only (as there's no debug release for windows or os/x)
once homebrew has envoy 1.22.0 merged, re-click release v1.22.0 to add os/x to it. Also v1.21.2 once that's fixed which is probably the same issue.

cc also @jcchavezs as due to travel I may be sporadic and I think you can help guide @cpakulski as needed

cpakulski · 2022-05-12T18:43:23Z

Current status:

linux and windows images are available for v1.22.0.
linux image is available for v.22.0_debug

Next steps:

fix homebrew for v1.22.0

cpakulski · 2022-05-13T20:44:48Z

Opened PR for homebrew-core to build version 1.21.2: Homebrew/homebrew-core#101428

cpakulski · 2022-05-24T20:37:54Z

Current status:

macOS image obtained by car works only for macOS version 12. Envoy image fails when run on macOS 10.15 and 11.
Next steps:
Do not verify macOS image for macOS 10.15 and macOS 11.

jcchavezs · 2022-05-24T22:03:04Z

Are we stop supporting this versions or just disable verification. Also can you point out the failure or describe the error (for context purposes)

cpakulski · 2022-05-25T18:40:53Z

We stop supporting those. Verification fails because binary cannot be executed on versions 10.15 and 11. We would have to introduce another dimension to indicate OS version. However, users can install Envoy on 10.15 and 11 via brew, because it takes it from different bottles.

jcchavezs · 2022-05-26T08:56:12Z

Thanks for the clarification. ons. 25. mai 2022, 20:41 skrev Christoph Pakulski ***@***.***

…

: We stop supporting those. Verification fails because binary cannot be executed on versions 10.15 and 11. We would have to introduce another dimension to indicate OS version. However, users can install Envoy on 10.15 and 11 via brew, because it takes it from different bottles. — Reply to this email directly, view it on GitHub <#34 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAXOYATD2UU3SQ7L2LBFDXTVLZX4BANCNFSM5UDW7TCA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

cpakulski · 2022-07-26T23:05:10Z

Closing this. Nothing is left to be done for rel. 1.22.0.

codefromthecrypt mentioned this issue Apr 22, 2022

Build Envoy v1.21.1 #33

Closed

cpakulski mentioned this issue May 6, 2022

Fix for extracting linux binaries from container image. #35

Merged

codefromthecrypt closed this as completed in #35 May 9, 2022

codefromthecrypt reopened this May 9, 2022

cpakulski closed this as completed Jul 26, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Track problems in Envoy 1.22.0 #34

Track problems in Envoy 1.22.0 #34

codefromthecrypt commented Apr 22, 2022

Lord-Y commented Apr 26, 2022

codefromthecrypt commented Apr 27, 2022

codefromthecrypt commented Apr 27, 2022

codefromthecrypt commented Apr 27, 2022 •

edited

Loading

codefromthecrypt commented May 8, 2022

codefromthecrypt commented May 9, 2022 •

edited

Loading

cpakulski commented May 12, 2022

cpakulski commented May 13, 2022 •

edited

Loading

cpakulski commented May 24, 2022

jcchavezs commented May 24, 2022

cpakulski commented May 25, 2022

jcchavezs commented May 26, 2022 via email

cpakulski commented Jul 26, 2022

Track problems in Envoy 1.22.0 #34

Track problems in Envoy 1.22.0 #34

Comments

codefromthecrypt commented Apr 22, 2022

Lord-Y commented Apr 26, 2022

codefromthecrypt commented Apr 27, 2022

codefromthecrypt commented Apr 27, 2022

codefromthecrypt commented Apr 27, 2022 • edited Loading

codefromthecrypt commented May 8, 2022

codefromthecrypt commented May 9, 2022 • edited Loading

cpakulski commented May 12, 2022

cpakulski commented May 13, 2022 • edited Loading

cpakulski commented May 24, 2022

jcchavezs commented May 24, 2022

cpakulski commented May 25, 2022

jcchavezs commented May 26, 2022 via email

cpakulski commented Jul 26, 2022

codefromthecrypt commented Apr 27, 2022 •

edited

Loading

codefromthecrypt commented May 9, 2022 •

edited

Loading

cpakulski commented May 13, 2022 •

edited

Loading