Implement Airgapped Configuration Mechanism

[tsmaeder]

This is about designing and implementing one or more mechanisms for configuring build systems int he "air-gapped" case. We need to solve two distinct problems:

• Override the settings from the source code (e.g. pom.xml) and default settings (e.g. maven central)
• Make tools accept self-signed certificates for secure connections (e.g. https).

There are various desirable properties for any solution:

• The solution should work for commands define in the devfile as well as using regular tools in the workspace (e.g. "mvn clean install" from the command line should work)
• Ideally, an administrator could override settings in existing devfiles per che installation via some mechanism (I believe there is a cr.yaml file in CRW)
• If there are relevant settings in the devfile, they should take precedence over those set per installation
• If the user makes changes to any of the settings (for example, by editing a settings.xml in ~/.m2), they take precedence over system/devfile settings

[tsmaeder]

We discussed using init containers for transforming configuration values into files on in user.home. How exactly this would work needs to be designed.

[slemeur]

@tsmaeder Before going into the implementation of new mechanisms, we recently merged #15435 which should allow to override settings.xml file more easily.
Documentation on how to achieve that and leverage the existing work should be step 0

[tsmaeder]

@slemeur not really...it's a prerequisite, but we still need mechanisms for creating those files. Environment variables cannot be used in settings.xml

[tsmaeder]

The idea is to collect the various ways we can configure language support for airgapped for different tools. Once we have that, we can design a proper config mechanism.

[l0rd]

@tsmaeder what about using an init container as described in #14857 (comment). It would be one distinct init container for every dependency manager we support but the mechanism would be the same (env var + init container).

[tsmaeder]

One mechanism that seems to tick all the boxes I've come up with:

1. Add a "files" property to che components (any) in the devfile like so:

- id: redhat/java11/latest
  type: chePlugin
  files:
    -  path: ~/.m2/settings.xml
       referenceContent: "some xml here $INTERNAL_REPO_URL more xml"

2. In the container init script we read those "files" and replace the env variable references (allow for escaping if we want put env variables in the container).

3. Have a mechanism to set/override environment variables from the "resource.yml" that's use for system-wide config overrides in CRW.

[tsmaeder]

@l0rd @slemeur @skabashnyuk wdyt?

[nickboldt]

can we also have either:

       referenceContent: "some URL to the file we want injected here"

or

       referenceContentURL: "some URL to the file we want injected here"

[tsmaeder]

We can add the same configuration mechanism to the meta.yml in the plugin. Also, we could make the creation of each file depend on the presence of a particular env variable.

[tsmaeder]

We might not cover all cases and combinations, but at least we have 90% covered.

[sunix]

would part of this be covered with enabling kubernetes secrets ?
#14680

[tsmaeder]

Well, the plan includes a provision for evaluating env vars in the file content, so I guess you can use stuff from there.

[nickboldt]

FYI downstream request for this feature in https://issues.redhat.com/browse/CRW-610

Something we can do in 7.9?

[tsmaeder]

To be done here:

• Write out a description of a design and discuss on che-dev list
• Implement (or document) mechanism to override repositories (for example) in global config (resource.yaml).
• Implement Syntax in devfile
• Implement Syntax in plugin meta.yaml
• Implement mechanism to write files via container startup script or init container.

[tsmaeder]

@nickboldt can you tell how the mechanism to override stuff in crw ("resource.yaml") functions or do you know who to ask?

[tsmaeder]

I've started a write up of the problem in a google doc: https://docs.google.com/document/d/1rE7MFUPnP3pq57vGM_DqyuFvlCbjYuqe-xFXgDiXSEM/edit#heading=h.3lz6pxl55uha

[slemeur]

Would be better in a document that anybody could be able to open.

…

On Wed, Feb 12, 2020 at 4:27 PM Thomas Mäder ***@***.***> wrote: I've started a write up of the problem in a google doc: https://docs.google.com/document/d/1rE7MFUPnP3pq57vGM_DqyuFvlCbjYuqe-xFXgDiXSEM/edit#heading=h.3lz6pxl55uha — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#15518>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAMPTIOX2ZHG3LYSYANTBQTRCQIO5ANCNFSM4J4HDXEQ> .

-- Stévan LeMeur // Product Manager // Developer Tools // +336-87-11-27-55

[tsmaeder]

I'll make it public when it's ready for general review.

[sunix]

I couldn't access to the doc,
as mentionned in #14680 (comment)

We should try to do something similar to https://docs.openshift.com/container-platform/4.3/builds/creating-build-inputs.html#builds-adding-input-secrets-configmaps_creating-build-inputs

The way I see it, is a section in the Dashboard where you could define these configmaps or secrets. We could also have configmaps or secret per organisation. and we could have a way to set these configmap or secrets through chectl.

I definitely do not see these values/files defined in the devfile. But references to secrets and configmap.
These secrets and config maps could be used by any workspaces by just referencing them.

We could have default entries for configmaps and secrets that we use in our default devfiles. These ones could be overriden by the user.

[tsmaeder]

Here's my proposal for implementing the air-gapped configuration mechanism: it consists of a general way of generating preferences, env vars and config files based on configuration values:

[tsmaeder]

@l0rd

automatically by whoever creates the users namespaces (second iteration)

The question is when this can happen: since configuration can change, we would at least have to create/delete the secrets when we create a workspace. But better still to pick up change at each workspace start. Do you think that "lifecycle" is possible with secrets?

[sunix]

let's say that a user have the previous devfile and has even created and started a workspace out of it. If he figures out it is not working (cannot access to maven central for instance), he could

• back to the dashboard (or anyother UI), create the secret "m2settings"
• upload his settings.xml with the right configuration to that secret "m2settings"
• restart his workspace and ... it will work :)

And let's say he is starting another workspace from another devfile that is using maven, it would work, taking the right settings.xml

[l0rd]

@tsmaeder I would like Che to not be responsible for creating and updating those secrets. To allow git with SSH flow we currently persist users SSH key pairs in postgres, expose a Che API to manage them and mount them in workspace Pods in the form of Kubernetes secrets. That's something I would like us to stop doing. Instead, when provisioning a new Che user, an administrator should setup the user namespace, including the configmaps and secrets. When needed, the admin should update namespace secrets and configmaps (and any namespace object). Developers can update configmaps and secrets as well but they should not need to deal with infrastructure configs if they do not want to. cc @skabashnyuk

@sunix what I am suggesting is a kind of dependency injection. It will simplify the devfile and keep it separate from infrastructure configurations. But you are right, this mechanism requires the agreement on some conventions (i.e. predictable settings.xml path in ${maven.home}/conf/ or ${user.home}/.m2/).

[sunix]

@l0rd Yes, what I am suggesting is a dependency injection by name: in this container, inject m2settings in /home/user/.m2/settings.xml . This would also be what user familiar to kubernetes would expect no?

How would you see the usage of secret's annotations and labels ?

[tsmaeder]

@l0rd If I understand your suggestion correctly, I see some drawbacks to your suggestion:

1. The setup task has to be repeated by the admin for every user
2. If a setting changes (let's say a new repo server is set up), the admin needs to update the secrets for every user.
3. If I take vscode.tasks.executeTask(task) does not run dependencies eclipse-theia/theia#7516 as a guide, we need to mount the secrets per container, so Che will be involved, even if it doesn't provide the values of the secrets. If we have more than one secret we need in a workspace, we'll need to provide a volume mount per secret (becuase it will need to be mounted in a different place.
4. In order to mount secrets, we need a mount point, which as I understand it a) needs to be present in the image and b) will be replaced with a directory containing only the secrets "files" specified in the volume. This won't really work for ~/.m2/settings.xml, since maven will want to write it's local artifact cache to ~/.m2`.

[tsmaeder]

@nickboldt since air-gapped requirements are largely driven by CRW customers, I'd be interested in your input.

[l0rd]

@l0rd Yes, what I am suggesting is a dependency injection by name: in this container, inject m2settings in /home/user/.m2/settings.xml . This would also be what user familiar to kubernetes would expect no?

@sunix I think we are converging :-) We can use (optional) devfile annotations to specify injection properties like the path. Something like:

devfile.yaml
---
...
components:
 - type: dockerimage
   annotations:
      mavenSettingsPath: /my/m2/folder/  # <--- optional, overrides secret's defaultPath
   alias: maven
   image: maven:3.11
...

maven-settings-secret.yaml
---
apiVersion: v1
kind: Secret
metadata:
  name: mvn-settings-secret
  labels:
    app: che                # <--- mandatory: all secrets labeled `app: che` will be mounted
    targetContainer: maven  # <--- optional, if omitted will be mounted on all containers
  annotations:
    mavenSettingsPath: /home/user/.m2/
data:
  settings.xml:  ...

@tsmaeder for 1) and 2) provision/updating users configs is about applying a template. An admin will have a template for che namespaces and will apply that every time a new Che user needs to be provisioned. That's how it works on che.openshift.io for instance. For 3) and 4) the sample above should clarify how a container and a path would be specified.

[tsmaeder]

@l0rd what is "maven-settings-secret.yml"? And I don't understand how the above addresses my point 4) from above: ~/.m2 will be broken for use with maven if you mount the secrets there.

[tsmaeder]

If I am a user I will want to specify some env variables for one or multiple devfile components as above but in this case in my user preferences:

@l0rd I'm not objecting to having per user settings, as I stated in my original proposal, I just did not consider it something that was a stated requirement.

[tsmaeder]

@tsmaeder I would like Che to not be responsible for creating and updating those secrets. To allow git with SSH flow we currently persist users SSH key pairs in postgres, expose a Che API to manage them and mount them in workspace Pods in the form of Kubernetes secrets.

Che is responsible for translating a devfile into a set of Kubernetes configurations (a pod, basically). That translation will use some central configuration values. There is no need for an API to manage anything, as far as I can see.

[sunix]

@l0rd ok as long as it works. It also has to work with env variables. Some secrets are mounted as files, others would be be exposed as env variables.

[skabashnyuk]

@l0rd about #15518 (comment) . What are the rules to match :

 annotations:
      mvnSettingsSecretPath

and

annotations:
    defaultPath: /home/user/.m2/

[tsmaeder]

@l0rd @sunix @skabashnyuk I fear if my concerns from #15518 (comment) are not considered, we're building something that will be quite useless.

[l0rd]

@tsmaeder let me try to answer but I am not sure that I got your questions so let's do a call if those answers don't help.

what is "maven-settings-secret.yml"

It's the specification of a secret that contains maven settings. Secret can be mounted as files in containers as specified here. This secret should be created in the developer namespace. For example I should have the secret created on my mloriedo-che namespace on che.osio. The secret can be created by the administrator that creates the namespaces or by the developers.

~/.m2 will be broken for use with maven if you mount the secrets there

True. Some possible workaround: mounting settings.xml in ${maven.home}/config, specifying <localRepository/> in settings.xml, using volumes subpath to mount the local repo or using an init container that copies settings.xml to ~.m2?

[l0rd]

@skabashnyuk you are right. I think that the name of the annotation should match.

[sunix]

how could .m2 be broken? @l0rd @tsmaeder? maven writes local artifacts in .m2/repository that could be a volume and not interfering with .m2/settings.xml

[l0rd]

@skabashnyuk I have update the name of the annotation in the secret and the devfile to be identical ( mavenSettingsPath)

[l0rd]

how could .m2 be broken? @l0rd @tsmaeder? maven writes local artifacts in .m2/repository that could be a volume and not interfering with .m2/settings.xml

@sunix I haven't tested but if settings.xml is mounted in .m2 it should override the folder completely. From the docs "Specify (...) mountPath to an unused directory name where you would like the secrets to appear".

[che-bot]

Issues go stale after 180 days of inactivity. lifecycle/stale issues rot after an additional 7 days of inactivity and eventually close.

Mark the issue as fresh with /remove-lifecycle stale in a new comment.

If this issue is safe to close now please do so.

Moderators: Add lifecycle/frozen label to avoid stale mode.

[l0rd]

@ericwill @tsmaeder what's missing to close this issue?

[ericwill]

@ericwill @tsmaeder what's missing to close this issue?

Yes, I believe the requirements have been fulfilled. Further iteration on improving configuration methods can be handled separately.