extend allowed keys in config.yaml

[mvAtWork]

Is your feature request related to a problem? Please describe.
This request falls in the same context as the previous discussion #629. We have now fully switched to managed layers and it works very well both locally and in the CI. 🥳

When preparing the required (layers) scm overrides, I came across two things that could be improved with the config.yaml.

1. If I want to avoid specifying an additional file with -lc for almost every bob command (which would be an issue for the local users) I must add the layer overrides directly to the config.yaml. This is then detected as an uncommitted change by git, even though it is only necessary for the local project configuration and should not be committed.

2. The CI setup requires GitLab job tokens in the overrides. In the regular scm overrides, I can access the environment variable where the job token is stored, but this doesn't work for the layer overrides. So currently the job token gets inserted into the layer overrides in plaintext. Not ideal, but acceptable in this specific circumstance since job tokens are short-lived and I clean up the generated files afterwards. But maybe there's a better way.

Describe the solution you'd like

1. Allow includes in the config.yaml so that it doesn't need to be touched when adding layer overrides. Missing include files should be ignored silently, just as with the default.yaml. I could then simply generate the file that is already specified as an include. The generated file would be ignored by git.

2. Allow access to the system environment variables, possibly limited using whitelist? Not sure if this is a viable solution.

[jkloetzke]

Thanks for the suggestions. That's very much appreciated.

I'll look into it...

[rhubert]

I like the idea of a include in the config.yaml. 👍

Since we must also use Gitlab I've a similar setup. This is how it works ATM:

On the CI (and only on the CI) I have a additional layers_overrides.yaml with layersScmOverrides. The local build can be used without this. Of course you need to use -lc layers_overrides for every bob command on the CI than - but I don't see the problem with this? I'd suggest to use a run_bob wrapper an include this where needed.

The layers_overrides.yaml is generated using something like

cat > layers_overrides.yaml << EOF
layersScmOverrides:
 - match:
     url: git@server:*
   replace:
     url:
       pattern: git@server:
       replacement: https://gitlab-ci-token:$CI_JOB_TOKEN@server/
EOF

which avoids the need of having access to the environment. (BTW: I also generate the scmOverrides in the same way every time. Does it work with to use CI_JOB_TOKEN in scmOverrides if CI_JOB_TOKEN is in the whitelist? 🤔
Anyway - I do not see an issue with having the token in this file? It's put into the audit files as well...

For some other reasons I've to modify the config.yaml as well. To avoid the unclean state I do a git update-index --assume-unchanged config.yaml afterwards...

[mvAtWork]

Sorry, closed in error due to fat fingers 🙈

[mvAtWork]

On the CI (and only on the CI) I have a additional layers_overrides.yaml with layersScmOverrides. The local build can be used without this. Of course you need to use -lc layers_overrides for every bob command on the CI than - but I don't see the problem with this? I'd suggest to use a run_bob wrapper an include this where needed.

The layers_overrides.yaml is generated using something like
...

Different tooling but same idea I'm currently using in our CI setup and using -lc "everywhere" is not an issue there. But when my setup tool is used locally (which it also supports), then the need for -lc (or the modification of the config.yaml to avoid -lc) may cause resistance from the local users.

Though I just realized, local use of my tool probably wouldn't include the command that does the CI setup. The need for using (layer) scm overrides locally is probably very limited. So this can probably be disregarded.

BTW: I also generate the scmOverrides in the same way every time. Does it work with to use CI_JOB_TOKEN in scmOverrides if CI_JOB_TOKEN is in the whitelist? 🤔

Yes, the env vars can be used directly in the scm overrides. Works beautifully and without the need for whitelist. We use even more levels of abstraction for semantic reasons and to reduce the places where changes have to be made:

• in the committed default.yaml (no modification necessary in the CI):

  environment:
      # SCM configuration environment
      SCM_GIT_ROOT_GROUP:         git@<hard-coded url of our main gitlab instance>:<root group on gitlab>
      SCM_GIT_BASEMENT_PLATFORM:  git@<hard-coded url of our main gitlab instance>:<root group on gitlab>/basement
  
  # other config
  
  include:
  # other includes
  - generated\gitlab-bob-ci
  

• in the generated\gitlab-bob-ci.yaml (only generated when running in the CI):

  environment:
      SCM_GIT_ROOT_GROUP: https://gitlab-ci-token:${CI_JOB_TOKEN}@${CI_SERVER_HOST}/<root group on gitlab>
      SCM_GIT_BASEMENT_PLATFORM: https://gitlab-ci-token:${CI_JOB_TOKEN}@${CI_SERVER_HOST}/<root group on gitlab>/basement
  # other config
  include:
  - scm_overrides
  

  Using CI_SERVER_HOST here also has the benefit of being independent from the specific gitlab instance, so the same CI setup can be used on different gitlab instances (e.g. prod and test).
• most of our recipes define their SCMs using these SCM_GIT_* variables so the generated\gitlab-bob-ci.yaml is sufficient for the CI setup
• any other overrides needed in the CI* go into generated\scm_overrides.yaml also using these SCM_GIT_* variables

* e.g. for using the correct branches for a merge request and its dependencies, see #629 for context about these merge dependencies

[jkloetzke]

In principle, config.yaml has those settings that should not be overridden by a Bob project user. That is why I have been hesitant to add an "include" feature to it. But layers configurations are special. They can already be used in config.yaml. So adding something like layersInclude sounds like a good idea. The included file is treated as layers configuration and can only have the layersWhitelist, layersScmOverrides and layersInclude keys.

Regarding the access to environment variables I'm yet unsure. See the old design discussion for some more insight. Back then, I thought that accessing the raw environment is not a good idea. The only place where this is allowed is the environment section in default.yaml. OTOH, layersScmOverrides is special anyway.

[mvAtWork]

So adding something like layersInclude sounds like a good idea. The included file is treated as layers configuration and can only have the layersWhitelist, layersScmOverrides and layersInclude keys.

Sounds good to me 👍

Regarding the access to environment variables I'm yet unsure. See the old design discussion for some more insight. Back then, I thought that accessing the raw environment is not a good idea. The only place where this is allowed is the environment section in default.yaml. OTOH, layersScmOverrides is special anyway.

I agree that access to the raw environment should be carefully considered. It could open the door to a whole lot of other problems.

So I just learned about layersWhitelist from your comment. 🙈 In a quick test with layersWhitelist: ["CI_JOB_TOKEN", "CI_SERVER_HOST"] using https://gitlab-ci-token:${CI_JOB_TOKEN}@${CI_SERVER_HOST} in the layersScmOverrides still didn't work, with and without -D CI_JOB_TOKEN=... added to the bob commands. Quite possible I didn't use/understand it correctly. How/where should I be using the whitelisted vars?

@rhubert: In our specific usecase where the env vars are needed for the scm URLs, maybe the change can be made in a different place. The git config has an insteadof option with one example being the switch to a different protocol. When generating this config in the CI, it should be possible to inject CI_JOB_TOKEN and CI_SERVER_HOST there. Doing this before invoking bob could be sufficient to solve our usecase. I haven't tested any of this. And it would be one additional thing to implement and test, making the whole setup more complex, which I would consider a drawback.

[jkloetzke]

So I just learned about layersWhitelist from your comment. 🙈 In a quick test with layersWhitelist: ["CI_JOB_TOKEN", "CI_SERVER_HOST"] using https://gitlab-ci-token:${CI_JOB_TOKEN}@${CI_SERVER_HOST} in the layersScmOverrides still didn't work, with and without -D CI_JOB_TOKEN=... added to the bob commands. Quite possible I didn't use/understand it correctly. How/where should I be using the whitelisted vars?

I made a quick test with a simple layersScmOverrides and it worked as expected:

bobMinimumVersion: "1.1"
layers:
    - name: foo
      scm: git
      url: server

layersScmOverrides:
    - match:
        url: server
      set:
        url: "$FOO"

If I call $ bob layers ls -D FOO=bar, I get the following, expected output:

foo:
...
        url: bar

Could you post your layersScmOverrides to see if there is some other problem?

Regarding the layersWhitelist: this is used for something else. Like the whitelist keyword, it is there to pass-through environment variables during the execution of the SCM, that is when git is running. Usually, it is used for things like SSH_AGENT_PID and SSH_AUTH_SOCK to allow authentication even with password protected SSH keys.

[jkloetzke]

I think #669 should solve the problem. It actually adds two things:

1. A new layersInclude key in config.yaml to include layer configuration files. It behaves like include in default.yaml. For reasons of consistency, also a layersRequire key is added.
2. Out of tree builds can have their own layer configuration file in the build tree.

The latter means that it does not need any modificatoins of the the recipes:

mkdir build
cd build
bob init ..
# create config.yaml with layersScmOverrides
bob dev/build ...

Let me know what you think.