Expose configmaps/secrets to build environment

[Mofef]

Hey, I was wondering if I could expose configmaps or secrets to build jobs aswell.
What I'm trying to do is add some custom apt sources along with a client cert in order to install some internal packages as dependencies.
Currently we work around this by installing some packages at runtime.

[mmourafiq]

That will be supported in v0.5.
For now, if the deps are almost the same for all builds, you can create a docker image, and use it as base image for all your builds.

[Mofef]

Oh very nice :)

We thought of this option, but in this case its easier for us to wait. Would you like to give a rough estimate for v0.5? (next month/quarter/year?)

[mmourafiq]

Pushing hard for a RC soon.

[Mofef]

Did this get lost? As far as I can see the configmaps/secrets are still not exposed in the build phase but only during runtime.

[Mofef]

I just found K8S_CONFIG_MAPS:BUILD_JOBS in the settings under Scheduling -> Build Jobs Scheduling... I guess this is what i'm searching for. Will report back in a min.

[Mofef]

Hm. No I can't figure it out. There seems to be little docu for those settings pages (maybe I just didn't find it) and just adding the name of the configmap to K8S_CONFIG_MAPS:BUILD_JOBS (resp. secret) doesn't seem to change anything. :/
@mouradmourafiq I'm happy to help with the documentation as soon as I understand it myself

[mmourafiq]

I think I only understood what you are trying to do, you don't need the configmaps/secrets for the build process, but rather for the dockerfile templating, is that correct?

[Mofef]

Let's put it like this: Before running an experiment I need to install some packages (dependencies of the experiment). To download those I need a client certificate. It would be nice if I could expose this client-cert as a secret (+ some config as config maps).

Honestly, I'm not sure how the build steps in polyaxon are different from dockerfile templating. As far as I understood it, the build steps of the polyaxonfile get translated to a Dockerfile, which is then used to build the container image used to run the experiment/job.

[mmourafiq]

Ok I see, it seems that in order to use the client-cert, a path must be mounted and the command must be updated, which is not supported in the v0.5.

A fully customizable build (and any job/service) is what we are aiming for the v0.6.

[Mofef]

um... not sure if we need to mount a path. What we do so far is sth. like

echo $secret_with_cert_data > /etc/apt/ssl/apt-ssl.crt

Just that we can't have that in the build steps because secret_with_cert_data is not exposed there. So we added it to the run steps before the actual run.

[Mofef]

@mouradmourafiq are you still watching this issue?

[mmourafiq]

Did you try your approach using kaniko as a build backend?

[Mofef]

I just tried it, but it doesn't seem to make a difference.

[mmourafiq]

Did you check if the config map gets mounted in the pod?

[Mofef]

Honestly I'm not sure how to check that :/ What would be the expected mount path?
I did check if the environment varaible is set (by running env as a build_step with nocache:true), and that is not the case.

[mmourafiq]

I will try it tomorrow and let you know if it's possible and what steps to do, if not possible, I will let you know if that could be possible in future releases.

[Mofef]

Great, thanks!

[Mofef]

I tried to make a minimal example of the problem with this polyaxon file:

version: 1
kind: experiment
environment:
  secret_refs: ["my-secret"]
build:
  image: ros:melodic
  build_steps:
    - echo $MySecretKey
run:
  cmd:
    - echo $MySecretKey

and this secret:

kubectl -n polyaxon create secret generic my-secret --from-literal=MySecretKey=test

Curiously now my builds fail with this error
"Could not start build job encountered an EnvFromRefFoundError exception."

Traceback (most recent call last): 
  File "/polyaxon/polyaxon/scheduler/dockerizer_scheduler.py", line 141, in start_dockerizer 
    reconcile_url=get_build_reconcile_url(build_job.unique_name)) 
  File "/polyaxon/polyaxon/polypod/dockerizer.py", line 232, in start_dockerizer 
    restart_policy=get_pod_restart_policy(max_restarts)) 
  File "/polyaxon/polyaxon/polypod/templates/resource_manager.py", line 418, in get_pod 
    sidecar_context_mounts=sidecar_context_mounts) 
  File "/polyaxon/polyaxon/polypod/templates/resource_manager.py", line 358, in get_pod_spec 
    restart_policy=restart_policy) File "/polyaxon/polyaxon/polypod/templates/resource_manager.py", line 227, in get_task_pod_spec ephemeral_token=ephemeral_token) 
  File "/polyaxon/polyaxon/polypod/templates/resource_manager.py", line 130, in get_pod_container
    env_from = get_pod_env_from(secret_refs=secret_refs, config_map_refs=config_map_refs) 
  File "/polyaxon/polyaxon/polypod/templates/env_vars.py", line 319, in get_pod_env_from 
    env_from += get_pod_env_from_secrets(secret_refs=secret_refs) 
  File "/polyaxon/polyaxon/polypod/templates/env_vars.py", line 283, in get_pod_env_from_secrets 
    'were provided but not defined in the config maps catalog'.format(validation)) 
polypod.templates.env_vars.EnvFromRefFoundError: The following secret refs `{'my-secret'}` were provided but not defined in the config maps catalog

Maybe I'm just missing what "catalog" is meant here. Also the docu doesn't specify it further as far as I can see. I can only guess that it refers to K8S_SECRETS:BUILD_JOBS / K8S_SECRETS:EXPERIMENTS on the settings page. Both of them I set to ["aptssl", "my-secret", "myothersecret"]

[mmourafiq]

secret_refs and config_map_refs need to be authorised in the UI by an admin before they can be used by an end user:

[Mofef]

Thanks a lot for the screenshot. I looked for that on the Settings page, but its in a separate Stores page that I forgot about.
So yes now I have a empty log line in the build an a line saying "test" in the execution. It also doesn'T work with a polyaxonfile of kind: build :/

[mmourafiq]

yes, now that I understood what you are trying to achieve and that I am thinking about it, it's not going to work actually, in order to pass an env var from the host to the dockerizer process, something like this docker -e must be done manually. At least for the native process, it will not work for the current version, next version the command and args for the dockerizer will be customizable and users can pass env vars from host to the docker process.

For kaniko, it seems that also you need to pass the envs as build args, though this issue mentions that this is not possible at the moment.

In both cases, the build component should be customized, and these type of customizations are only possible in the v0.6.

[Mofef]

Ok. Thank you so much for your time and help.
In this case I guess the easiest is still to pre build our base image outside of polyaxon and host it on a private container registry. I'll have a look at https://docs.polyaxon.com/integrations/gcr/ then. :)