che-plugin-registry cannot replace image tag according to CHE_SIDECAR_CONTAINERS_REGISTRY_URL

[wcho]

Describe the bug

che-plugin-registry does not replace image tag according to CHE_SIDECAR_CONTAINERS_REGISTRY_URL.
It seems that the replace process in che-plugin-registry entrypoint.sh does not work properly. Maybe due to invalid regex.

Che version

• latest
• nightly
• other: please specify

Steps to reproduce

I deployed CheCluster using airGapContainerRegistryHostname and airGapContainerRegistryOrganization with custom value.
It sets CHE_SIDECAR_CONTAINERS_REGISTRY_URL and CHE_SIDECAR_CONTAINERS_REGISTRY_ORGANIZATION for plugin-registry pod. But che-plugin-registry does not replace image tags in meta.yaml files.

Expected behavior

Image tags in meta.yaml files should be replaced with provided registry and organization.

Runtime

• kubernetes (include output of kubectl version)
• Openshift (include output of oc version)
• minikube (include output of minikube version and kubectl version)
• minishift (include output of minishift version and oc version)
• docker-desktop + K8S (include output of docker version and kubectl version)
• other: (please specify)

Screenshots

Installation method

• chectl
  • provide a full command that was used to deploy Eclipse Che (including the output)
  • provide an output of chectl version command
• OperatorHub
• I don't know

Eclipse Che Logs

Additional context

[cccs-eric]

Yes, I am running into the same issue as well. I did some digging and this was introduced in version 7.26.0. A single quote surrounding the image name got added by the build process. See below:
In image version 7.25.2

/usr/local/apache2 # cat /var/www/html/v3/plugins/rust-lang/rust/latest/meta.yaml
apiVersion: v2
publisher: rust-lang
name: rust
version: latest
type: VS Code extension
displayName: Rust
title: Rust
description: Rust for Visual Studio Code (powered by Rust Language Server/Rust Analyzer).
  Provides lints, code completion and navigation, formatting and more.
icon: /v3/images/rust-lang-rust-icon.png
category: Programming Languages
repository: https://github.com/rust-lang/rls-vscode
firstPublicationDate: '2020-05-13'
spec:
  containers:
    - image: quay.io/eclipse/che-plugin-sidecar:rust-52f9e9a
      name: vscode-rust
      memoryLimit: 1024Mi

See no quote around - image: quay.io/eclipse/che-plugin-sidecar:rust-52f9e9a

In image version 7.26.0 up to now

/usr/local/apache2 # cat /var/www/html/v3/plugins/rust-lang/rust/latest/meta.yaml
apiVersion: v2
publisher: rust-lang
name: rust
version: latest
type: VS Code extension
displayName: Rust
title: Rust
description: >-
  Rust for Visual Studio Code (powered by Rust Language Server/Rust Analyzer). Provides lints, code completion and
  navigation, formatting and more.
icon: /v3/images/rust-lang-rust-icon.png
category: Programming Languages
repository: 'https://github.com/rust-lang/rls-vscode'
firstPublicationDate: '2020-05-13'
latestUpdateDate: '2021-02-10'
spec:
  containers:
    - image: 'quay.io/eclipse/che-plugin-sidecar@sha256:e374ec367447afea9b12f5a8f6690a6f993526408a3731f522a53e5d0bfc287f'
      name: vscode-rust
      memoryLimit: 1Gi

We now have quotes surrounding - image: 'quay.io/eclipse/che-plugin-sidecar@sha256:e374ec367447afea9b12f5a8f6690a6f993526408a3731f522a53e5d0bfc287f'

The problem is with the IMAGE_REGEX. It is expecting double-quotes instead of single-quotes. I am not sure if the build process should be writing out double-quotes or if the IMAGE_REGEX should be updated to support single-quotes?

[cccs-eric]

There is also another problem with this IMAGE_REGEX. Once the entrypoint.sh script has run to replace the tags with sha256, some images are written as a multi-line entry. So the resulting YAML looks like this (from che-plugin-registry:/usr/local/apache2/htdocs/v3/plugins/eclipse/che-theia/latest/meta.yaml):

  initContainers:
    - name: remote-runtime-injector
      image: >-
        quay.io/eclipse/che-theia-endpoint-runtime-binary@sha256:b98f3cd34dbc8785df4ea7dc155339a66dcc72a3989d804e1990d6ad9f6d0168
      env:
        - name: PLUGIN_REMOTE_ENDPOINT_EXECUTABLE
          value: /remote-endpoint/plugin-remote-endpoint
        - name: REMOTE_ENDPOINT_VOLUME_NAME
          value: remote-endpoint

Notice image: >- which will not match the code in entrypoint.sh.

Because of this, we end up with an init-container referencing a public image, which crashes the workspace creation process in an air-gapped setup.

[flacatus]

Hello! I have the same problem in airgap mode installation with devfile and plugin registry...
Seems like che-plugin-registry and che-devfile registry build script is not working properly:

./build.sh --organization "${ORGANIZATION}" \
           --registry "${INTERNAL_REGISTRY_URL}" \
           --tag "${TAG_NIGHTLY}" \
           --offline

If I run podman command against new image builded I get:
Command:

podman run --authfile=${REG_CREDS} -it --rm \
  --entrypoint cat "${INTERNAL_REGISTRY_URL}":5000/${ORGANIZATION}/che-devfile-registry:${TAG_NIGHTLY} /var/www/html/devfiles/external_images.txt

quay.io/eclipse/che--centos--mongodb-36-centos7:latest-a915db7beca87198fcd7860086989fe8a327a1a4f6508025b64ab28fcc7423b2
quay.io/eclipse/che--centos--mysql-57-centos7:latest-e08ee4d43b7356607685b69bde6335e27cf20c020f345b6c6c59400183882764
quay.io/eclipse/che-cpp-rhel7:nightly
quay.io/eclipse/che-dotnet-2.2:nightly
quay.io/eclipse/che-dotnet-3.1:nightly
quay.io/eclipse/che-golang-1.14:nightly
quay.io/eclipse/che-java11-gradle:nightly
quay.io/eclipse/che-java11-maven:nightly
quay.io/eclipse/che-java8-maven:nightly
quay.io/eclipse/che-nodejs10-community:nightly
quay.io/eclipse/che-nodejs10-ubi:nightly
quay.io/eclipse/che-nodejs12-community:nightly
quay.io/eclipse/che-nodejs8-centos:nightly
quay.io/eclipse/che-php-7:nightly
quay.io/eclipse/che-python-3.8:nightly
quay.io/eclipse/che-quarkus:nightly
quay.io/eclipse/che-rust-1.39:nightly
registry.access.redhat.com/ubi8-minimal:8.3
registry.access.redhat.com/ubi8/ubi-minimal

Should not sustitute build.sh the quay.io organization?

[cccs-eric]

I was able to fix both the single quote issue as well as the multi-line issue. Those can easily be fixed by changing the options to js-yaml safeDump(). But it requires a migration to version 4 of js-yaml and this will require more testing than I was prepared to do. In summary, once js-yaml has been migrated to version 4, the required changes for meta-yaml-writer.ts are:

            // add spec object
            metaYaml.spec = spec;
            const yamlString = jsyaml.safeDump(metaYaml, { quotingType: '"', lineWidth: -1 });
            const generated = { ...metaYaml };
            generated.id = `${computedId}/${version}`;
            metaYamlPluginGenerated.push(generated);

and

            const devfileYaml = this.metaYamlToDevfileYaml.convert(metaYaml);
            if (devfileYaml) {
              const devfilePath = path.resolve(pluginsFolder, computedId, version, 'devfile.yaml');
              const devfileYamlString = jsyaml.safeDump(devfileYaml, { quotingType: '"', noRefs: true, lineWidth: -1 });
              promises.push(fs.writeFile(devfilePath, devfileYamlString));
            }

Changing the lineWidth option to -1 disables the multi-line output
Changing the quotingType option to use double-quotes will make IMAGE_REGEX properly match image names. But this option is only available in version 4 and some migration steps are required.

Hopefully this is all you need to create a fix @sunix @ericwill. I could have done a PR, but I was not comfortable to test everything.

[sunix]

Hello @cccs-eric, @wcho and @flacatus ,

Looks like this is duplicating #19492

I have a PR open, would be nice if you could have a look and test if it works for you: eclipse-che/che-plugin-registry#937. It should cover the multi-line and quotes issues.

[cccs-eric]

@sunix I commented on your PR.

[sunix]

Hello eclipse-che/che-plugin-registry#937 has been merged. I am closing this issue. Thanks!