Keep envoy blob up-to-date

[aminjam]

Enter an issue title

Keep envoy blob up-to-date

Summary

Recently envoy has updated their process whereby they've stopped publishing latest tag for https://hub.docker.com/r/envoyproxy/envoy docker images. We need a different way to pull the latest stable version of envoy in order to keep diego-release blob up-to-date.

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this. Unfortunately, the Pivotal Tracker project is private so you may be unable to view the contents of the story.

The labels on this github issue will be updated when the story is started.

[jvshahid]

Why can't we use https://hub.docker.com/r/envoyproxy/envoy-dev/tags/? The dev docker images seem similar to what we used before. They are built after each commit to master.

[sunjayBhatia]

There was a decision made in this commit/story: cloudfoundry/wg-app-platform-runtime-ci@25c4457#diff-993d49d192428a994f10185f8f68548c

from experience, master isn't the most stable tbh, if we're aiming for no-surprises stability using tagged releases might be safer, also given the Envoy community's security-release strategy, we could even stick on a minor release and consume patch releases for a while before bumping

[sunjayBhatia]

while its unlikely our use case will cause us problems, there have been commits that get to master that cause crashes etc. or other regressions that aren't caught in Envoy CI

[jvshahid]

According to https://github.com/envoyproxy/envoy/blob/master/RELEASES.md. 1.13 is supported until early next year. We might be able to stick to 1.13 and upgrade towards the end of 2020.

I am curious to know if the @cloudfoundry/cf-networking team has any opinions.

[sunjayBhatia]

Yeah, I think we've bumped even past that to the 1.14.x range already so we can stick on that line for ~ a year from now

[jvshahid]

I see the issue now. The Envoy team doesn't publish a 1.14 tag that tracks the latest 1.14.x version and Concourse doesn't allow the regex matching of the tags. An alternative is to run docker inside Concourse instead of using the Concourse resource. Or, we could submit a PR to the resource maintainers to allow the usage of a regexp to match the tag.

[christianang]

I don't think we (cf-networking) have any strong opinions on bumping or not.

[chinigo]

Hi there! Our team needs to use a feature that envoy plans on adding in 1.15: the SNI dynamic forward proxy filter.

We don't have strong opinions about bumping or not—we could deploy our own envoy binary at whatever version we want—but it would be great if we could just invoke the one that's shipped by default on CF instances.

Thanks for considering our use case!

[sunjayBhatia]

@chinigo while it is unlikely to change, I would caution against depending on components from the platform (like the location/version of the envoy binary mounted in a container etc.) as they really are just implementation details and not defined contracts we have made with the app environment

[sunjayBhatia]

@mdelillo and I did some experimenting and have some findings for some of the above and offline discussions about how to auto-bump envoy:

• the "docker in docker" approach:
  • follow tags/releases of envoy source
  • when the job is triggered, the latest tag of envoy source will produce a version we can use in a task
  • the subsequent task will run docker to extract the envoy binary from the container image
  • cons:
    • requires running docker within a concourse task container which is fraught with troubles (ask the buildpacks team for their experiences, apparently it has been complex)
    • when envoy source is tagged, there is a v small chance the docker image hasn't been pushed yet
• download envoy image without docker approach
  • follow tags/releases of envoy source
  • when the job is triggered, the latest tag of envoy source will produce a version we can use in a task
  • the subsequent task will use this script (or some similar thing) to download the container image without docker
  • run something like ls ./*/layer.tar | xargs -n1 -I{} sh -c 'tar tf {} | grep "usr/local/bin/envoy" && tar xvf {} usr/local/bin/envoy' to extract the envoy binary
  • cons:
    • when envoy source is tagged, there is a v small chance the docker image hasn't been pushed yet
    • relying on a potentially unstable/unsupported contrib script from moby
    • we have know the internals of the container image format
• some flavor of solution where we bump the envoy tag in our concourse task file
  • follow tags/releases of envoy source
  • when a new tag is released, bump the envoy container image tag in https://github.com/cloudfoundry/diego-ci/blob/02521180b16652790a2ce12d75a3086652b252c8/tasks/create_envoy_blob.build.yml
  • commit change, push it, and then whenever the envoy bump is triggered it should use the latest container image
  • cons:
    • feels "weird and gross"? seems like there should be a better way to do this
• ask concourse resource maintainers to support a new feature so that the resource can follow semver in tags
• ask envoy to tag releases in a minor version with a latest tag for that line (e.g. 1.14 or 1.14.x-latest or something)

[chinigo]

Sunjay, Thank you for your reply. We have *no requirement to bump* envoy to the latest version. We're happy to use any version >= v1.13.0, but it doesn't need to track new releases. I agree that we might not want to use the binary that ships with CF because that imposes a dependency on the platform that might change without notice. What would you say is the easiest way to ship an arbitrary but specific version of Envoy—say v1.14.0—to run as a CF app, on an environment managed by BOSH? (This feature is targeted at customers still on BOSH, who have not yet and may not ever transition to Kubernetes.) Can we build & push a Docker image to run? Should we create a binary buildpack with a precompiled, statically-linked executable of Envoy? Should we compile from source in place on the VM?

[sunjayBhatia]

@chinigo I think there are probably a good few options which vary in maintainability etc. in that space but that is maybe a conversation for a more general forum, not this specific issue about version management of envoy in diego-release

[chinigo]

I agree. Which forum do you suggest? This is my second week here, and each forum I ask this question in I get a different perspective. Where is the most authoritative place to ask this? -- ~ Chinigo (Written on a tiny iPhone keyboard; please forgive any typos.)

[aminjam]

We are going to be consuming per-stable-version "latest" tag going forward.