K8s secrets reference from step

[zc-devs]

Part of #3582

Pipeline:

skip_clone: true
steps:
  secret-test:
    image: alpine
    commands:
      - echo $$AWS_ACCESS_KEY_ID
      - echo $$AWS_SECRET_ACCESS_KEY
    backend_options:
      kubernetes:
        secretNames:
          - test-secret

Test secret:

apiVersion: v1
kind: Secret
metadata:
  name: test-secret
  namespace: test-woodpecker-runtime
data:
  AWS_ACCESS_KEY_ID: N0lrZHNzb0xleGFtcGxlNkpJVnZDRXE=
  AWS_SECRET_ACCESS_KEY: TXk0WE5BYXNleGFtcGxldXRKRHNUWHc=
type: Opaque

1. Default Agent config.

In log I see warning:

{"level":"debug","time":"2024-04-27T18:17:49Z","caller":"/src/pipeline/backend/kubernetes/pod.go:166","message":"Secret names were defined in backend options, but its using disallowed by instance configuration "}

WP output:

+ echo $AWS_ACCESS_KEY_ID

+ echo $AWS_SECRET_ACCESS_KEY

2. Allowing native secrets.

Agent config:

data:
  WOODPECKER_BACKEND: kubernetes
  WOODPECKER_BACKEND_K8S_NATIVE_SECRETS_ALLOW_FROM_STEP: 'true'
  ...

WP output:

+ echo $AWS_ACCESS_KEY_ID
7IkdssoLexample6JIVvCEq
+ echo $AWS_SECRET_ACCESS_KEY
My4XNAasexampleutJDsTXw

[qwerty287]

Not sure how complex this is to implement, but if you use internal secrets and print them out to the logs, they're replaced with ********. Can we apply something similar here too? If not, this needs a warning in the docs.

Also, in my opinion WOODPECKER_BACKEND_K8S_NATIVE_SECRETS_ALLOW_FROM_STEP is way too long for an env name - wouldn't WOODPECKER_BACKEND_K8S_ALLOW_NATIVE_SECRETS be enough?

As an alternative to native support, maybe we could implement http service extension doing this? This feature is still in development and not supported yet. I understand however if you would like to support it natively.

[zc-devs]

if you use internal secrets and print them out to the logs, they're replaced with ********

I know, i know... Also thought about it and decided to warn in the docs too.
Now we don't have secrets to find them in the logs. However, when second part is implemented, this is achievable.

WOODPECKER_BACKEND_K8S_ALLOW_NATIVE_SECRETS

Will take it.

http service extension doing this?

Extension or external service?
I don't like extensions :) but like idea of external (micro)service. It might be useful in terms of duplication of masking secrets logic, at least.

Seems, I cannot manage it alone.
Is there some example / skeleton of service? I have ideas of service implementation, though. So I can do it, but need the Go "template". Also help with Server-side code might be needed.
How it should be organized? In this repo? Then probably POC branch is needed? Otherwise separate repo?

Edit
Can the external secrets work with internal at the same time?

[qwerty287]

As I wrote, external services are not yet available and still in development... They'll work similar to config extensions (https://woodpecker-ci.org/docs/administration/external-configuration-api) but with possibility so set on them global/org/user/repo level.
We can extend this to be able to use both internal and external secrets combined together.
But probably this feature will take some more time to complete...

[zc-devs]

But probably this feature will take some more time to complete

Let's give this PR a try then. Could you add build_image label, please?

@dominic-p, could you test then?

[dominic-p]

Thanks for working on this! I'm ready to test this PR as soon as we get an image published.

[zc-devs]

And this too

[dominic-p]

Ok, I was able to test this tonight, and it seems to be working as expected! I was able to successfully build and push my first image to my local repo! 🎉🎉🎉

I did run into one snag. As I mentioned, my main use case for this feature is to provide container image registry credentials to my build script. These credentials are currently stored in a Kubernetes registry cred secret. This kind of secret uses the hardcoded key .dockerconfigjson to reference its data. The issue is that with the current implementation we wind up with an environment variable with a dot in its name. I had a hard time using it since env variable names really aren't supposed to have dots in them.

I was able to work around the issue with printenv, but it's kind of awkward. I'm not sure if this problem is worth fixing in the current iteration of this feature or not. Maybe we can just add a short blurb to the documentation about it? Or, a simple solution might be to replace any special characters in the secret key names with underscores and then document that convention?

[zc-devs]

Could you provide an example?

1. Secret.
2. Build tool (kaniko, buildx, etc.) and its configuration.

I use kaniko debug image and plain secret:

    secrets:
      - docker_usr
      - docker_pwd

kaniko requires Docker auth file in /kaniko/.docker/config.json, so I create this file manually:

DOCKER_AUTHS=$(printf '{"auths":{"%s":{"auth":"%s"}}}' "$REGISTRY" "$AUTH")
printf '%s' "$DOCKER_AUTHS" > "/kaniko/.docker/config.json"

In my use-case it could be requirements:

1. Existing dockerconfigjson secret
2. Mount as file to /kaniko/.docker/config.json

Current implementation treat .dockerconfigjson as env var. Suppose it is DOCKER_AUTHS instead. What do you do next? Do you write it in a file and point your build tool to it?

[dominic-p]

Thanks for the detailed feedback.

Here's an example secret from my cluster:

apiVersion: v1
type: kubernetes.io/dockerconfigjson
kind: Secret
metadata:
  name: reg-cred
  namespace: woodpcker
data:
  .dockerconfigjson: <base64 encoded JSON auth file here>

And, here's a portion of the build script that would use it. I'm using buildah to build my container images via a standard shell script.

#!/bin/sh

# Create a working container
dev=$(buildah from docker.io/alpine)

# ... build the image

# Save credentials in temporary file

# You would like to do something like this, but, of course, it won't work
# echo "$.dockerconfigjson" > /tmp/.dockerconfigjson

printenv ".dockerconfigjson" > /tmp/.dockerconfigjson

# Save and push the image to our local registry
buildah commit $dev "$name"

buildah push --authfile /tmp/.dockerconfigjson "$name" "docker://$registry_url/$name"

So, the simplest workflow for me is to simply dump the reg-cred secret data into a temporary file and then reference it as shown above.

Of course, I could use a a regular Opaque secret and build the docker config JSON as you showed in your example (or reference the username and password with buildah push --creds), but since I already have the JSON in an existing secret it would be nice to leverage it. One benefit of doing it this way is that Kubernetes requires you to use this kind of secret in spec.imagePullSecrets. So, I can use the same secret to push to the local registry and pull from it.

At the end of the day, this is not a big deal. There are many usable workarounds. I just wanted to point out the awkwardness that you get when you wind up with an env variable that has a dot in its name.

[zc-devs]

Thanks for explanation. This PR have to be reworked then. Meantime I move it to a draft.

[dominic-p]

Sure thing. My personal preference would be to release this as-is and then iterate from there. Even in its current form it's enough for my use case. But, I can understand if you want to wait until it's a bit more complete.

[zc-devs]

iterate from there

Sure. But we have to decide on syntax.

[zc-devs]

Agent config:

  WOODPECKER_BACKEND_K8S_ALLOW_NATIVE_SECRETS: 'true'

Secret:

apiVersion: v1
kind: Secret
metadata:
  name: test-secret
  namespace: test-woodpecker-runtime
data:
  AWS_ACCESS_KEY_ID: N0lrZHNzb0xleGFtcGxlNkpJVnZDRXE=
  AWS_SECRET_ACCESS_KEY: TXk0WE5BYXNleGFtcGxldXRKRHNUWHc=
type: Opaque

Pipeline:

skip_clone: true
steps:
  secret-test:
    image: alpine
    commands:
      - echo $$AWS_ACCESS_KEY_ID
      - echo $$AWS_SECRET_ACCESS_KEY
    backend_options:
      kubernetes:
        secrets:
          - name: test-secret

Output:

+ echo $AWS_ACCESS_KEY_ID
7IkdssoLexample6JIVvCEq
+ echo $AWS_SECRET_ACCESS_KEY
My4XNAasexampleutJDsTXw

[qwerty287]

I have some kind of issue with the kubernetes approach of options that could be/are dangerous.

We currently have an env var option for everyone of them, just like WOODPECKER_BACKEND_K8S_ALLOW_NATIVE_SECRETS. The docker backend is doing it differently: You can't use the feature as long as the repo isn't trusted. If it's trusted you can use these features.

I prefer the docker approach. It's configurable from ui and can be set per repo. Also, kubernetes and docker backends should work in a similar way and why do we have the trusted option if we don't use, but rather add new env options?

Yes, I know that this is currently not really possible and we would have to do some refactoring for the backends first, but just wanted to point that out. So nothing to do here, just some discussion (probably it's better to move that to it's own thread).

[dominic-p]

Sure thing. It looks like the pipelines are currently waiting for approval. Once we have a new image built with the latest version of the PR I can test it.

[dominic-p]

I'm not sure why the docker publish failed.

Canceled: grpc: the client connection is closing

Maybe a network error of some kind?

[qwerty287]

Was successful now. @zc-devs please fix the linters: https://ci.woodpecker-ci.org/repos/3780/pipeline/17390/31

[qwerty287]

The formatting is still not successful: https://ci.woodpecker-ci.org/repos/3780/pipeline/17420/31#L230

[zc-devs]

File is not gci-ed with --skip-generated -s standard -s default -s prefix(go.woodpecker-ci.org/woodpecker) --custom-order (gci)

Which command should I run?

[qwerty287]

That's the import order/grouping. You can do that manually, never did that with a command. I can give you more details what you have to fix tomorrow

[lafriks]

Might be a bit late to the party but just a question why we do not implement defining such secrets in woodpecker secret section where instead of value there would be option to provide where to get secret value from (be it k8s secret, hashicorp vault etc) and then in pipeline just define secret as we already do.
With adding more and more runner specific options to pipeline syntax it will become less portable and harder to use

[6543]

☝️ if there is an api / interface for it - that would be best

I remember to have created an issue: "add more secret provider" or so ...

[6543]

anyway that's still a good enouth solution for now, and later we can move this into a more generic solution if it exists

[anbraten]

☝️ if there is an api / interface for it - that would be best

#3349

[lafriks]

In this case this probably can't be received at server side, it would still be for agent to bind when starting step so process would still be the same, my main point is that adding such definitions to pipeline makes them hard to maintain if things changes in future, so my suggestion would be just moving definition to server side but resolving value in this case would be still be on specific agent runner (like docker would not still be able to resolve k8s secrets)

[qwerty287]

This should fix gci

[woodpecker-bot]

Tearing down https://woodpecker-ci-woodpecker-pr-3655.surge.sh