Improve UX of getting credentials into Tasks

[ghost]

Expected Behavior

Reading the docs a user can quickly and easily discover how to:

1. declare credentials their Tasks accept
2. attach credentials to TaskRuns or PipelineRuns

Once a user has read the docs, there's a low hurdle to attaching credentials to their CI workloads.

Actual Behavior

Problems with Documentation and User Flow

Our auth.md doc describes a system for getting credentials out of Secrets and into Steps.

The process for users is documented as follows:

1. add the credential information to a Secret in k8s
2. add some annotations to that Secret to ensure it's recognized by Tekton
3. add the Secret to a Service Account
4. add the Service Account's name to your TaskRun.

The number of steps isn't too bad but there's a lot of room for confusion:

• the annotations and type information on Secrets has to be entered pretty precisely
• errors don't show up until a Task is running
• validation of the credentials is non-existent
• there's a disconnect between the Secret and the Run: the user adds the service account name to the TaskRun and then needs to hope the Secrets are picked up and parsed correctly with no other feedback from Tekton.

Problems with Implementation

On the controller-side Pod startup looks as follows with credentials:

1. a creds-init Image is injected as an init-container of the TaskRun Pod
2. creds-init receives Secrets as Volumes. Secrets must include specific type and annotations.
3. creds-init copies the credentials out of those Volumes into /tekton/home or /tekton/creds
4. entrypoint, before each Step, copies the credentials out of /tekton/creds into the Step Container's $HOME

There are a couple of challenges with this:

• Secrets get mounted by Kubernetes as in-memory filesystems but creds-init copies those credentials to disk (/tekton/home and /tekton/creds are emptyDir volumes). This diminishes a security feature built into the platform.

• Our guidance on credentials for Task Authors and Users is not very strong, despite how long auth.md is. It's not easy to figure out precisely what you should implement or expect, regardless if you're a Task Author or a Task User.

• Every Step gets access to all credentials. Auth.md states at the end: "For sensitive credentials that should not be made available to some steps, do not use the mechanisms outlined here. Instead, the user should declare an explicit Volume from the Secret and manually VolumeMount it into the Step." So we're kind of acknowledging here that there are problems with the existing flow and Tekton's treatment of creds.

Requirements and Constraints

Requirements

• Improve ease-of-use over the existing creds-init system.
• Using authenticated resources like private repos should be practically as simple as using public.
• Improve explicitness. Credentials should be a declarable dependency of a Task.
• Improve security of credentials. We currently advise users not to employ the existing mechanism if their creds are "sensitive". This should not be a thing anymore.
• Credentials should support being "optional".
  • For example, a Git task can work with both authenticated and public repos.

Constraints

• Solution(s) must not interfere with existing creds-init behaviour
• Solution(s) must work in containers running as root, non-root and randomized user.
• Solution(s) must not preclude any other creds systems that orgs are already using.
  • For example, Tekton's creds system should not prevent this stuff being achievable.
• Solution(s) must "play nice" with any external credential mechanism like Vault, SealedSecrets, ExternalSecrets, our own pipeline-as-code ideas etc...

Related Issues

I'm collecting creds-related issues as I spot them.

• [Bug Report] Cannot use ssh private key in task steps #1271
• Creds-init container process caused "invalid value "tekton-ssh-key={my own gitlab domain name}" for flag -ssh-git: exit status 1" #1350
• Proposal: Per-resource Service Accounts #1371
• git auth with multiple keys for the same provider fail #2407
• git basic authentication fail #2409
• Not able to push to git repository using ssh-auth #2638
• Issue when trying to access our GitHub repo #2689
• Enterprise Git clone fails in tekton taskrun #2703
• v0.13.0: unsuccessful cred copy: ".ssh" from "/tekton/creds" to "/root" #2791
• Git Credential Initializer doesn't include port when running ssh-keyscan #2801
• git-clone:0.1 - open /root/.ssh/known_hosts: too many levels of symbolic links catalog#531
• Explore entrypoint binary redacting secrets from logs #3373
• Unsuccessful cred copy: ".docker" at end of successful build #3399
• .ssh/* files do not have expected/required permissions when copied into custom home dir #3509

[ghost]

Design Idea 1 - Use Workspaces for Credentials

The fact that creds are often mounted as volumes and manifest as files on disk leads me to think that Workspaces may make a good primitive for credentials. This idea proposes the following changes to specs and the Pipelines controller to accomodate credentials in workspaces:

A Workspace can be declared as "optional" by a Task.

• An optional Workspace will not fail validation of a Run that doesn't bind a volume to it.
• An unbound optional Workspace results in Steps running with no volume mounted at that mountPath; it's just the container's regular filesystem.
• Example use case: git-clone expects ~/.ssh to be available for authenticated clones but works just fine without it for public repos.
• Existing workaround: Run has to provide emptyDir volumes for unneeded workspaces

A Workspace can be declared as "sensitive" by a Task.

• Sensitive Workspaces only accept Secret Volumes from Runs
• Sensitive Workspaces get permission 0400 by default (overridable? when? where?)
• Existing workaround: None exists to enforce Secret Volume usage. For permissions, Runs must declare them on the volume otherwise Secrets mount with 0644 by default.

Steps of a Task can declare the Workspaces they need access to

• Steps must declare if they're going to use a Sensitive Workspace
• The TaskRun reconciler will only add volumeMounts for a Sensitive Workspace to those Steps that declare an explicit dependency on them, instead of adding it to all Steps
• Example use case: Only some Steps in a Task should have access to sensitive creds.
• Existing workaround: User must figure out all the volume / volumeMount stuff on their own when writing the Steps of their Tasks. They can't use Workspaces for this currently.
• Here's what the YAML for Steps declaring workspaces might look like:

workspaces:
- name: my-sensitive-workspace
  description: Provide git credentials here.
  sensitive: true # only mounted if explicitly requested by step
  mountPath: /home/.ssh
- name: my-shared-output-workspace
  description: This is where the code will be put.
  mountPath: /source
steps:
- name: do-something-authenticated
  workspaces:
  - my-sensitive-workspace # requests sensitive workspace mount
  - my-shared-output-workspace # documents use but not required
  image: foo
  # ...

Problems with this design

• There's no good solution for credentials that are individual files (like ~/.git-credentials).
  • Secrets always mount as directories.
  • Projected Volumes allow individual items of multiple Secrets to be exposed together in a single directory but these also have problems.
• Doesn't provide a mechanism to expose credentials as environment variables
  • Quite common practice (cf. AWS_* env vars for the aws sdk)
• This doesn't solve problem of credentials getting into $HOME.
  • Secrets (as well as Projected Volumes) mount as read-only.
    • Can't mount a Projected Volume directly into $HOME with a combination of all the Secret credentials that a Task needs. $HOME would be read-only, breaking a large number of tools and scripts.
  • The $HOME variable doesn't really get resolved until runtime. Some platforms randomize the UID or enforce non-root users.

[ghost]

Design Idea 2 - Introduce Credentials Sections to CRDs

First, a quick example of how I imagine this syntax looking in a Task spec:

credentials:
- name: ssh-directory
  optional: true
  description: |
    An .ssh directory with id_rsa, known_hosts, etc...
  directory: $HOME/.ssh
- name: git-credentials-file
  optional: true
  description: |
    A file with each line formatted https://user:pass@example.com
  file: $HOME/.git-credentials
- name: git-password-envvar
  optional: true
  description: |
    Not a real git env var, just illustrating possible syntax.
  envVar: GIT_PASSWORD
steps:
- name: clone
  credentials:
  - ssh-directory
  - git-credentials-file
  - git-password-envvar
  image: foo
  # ...

And possible syntax for providing those credentials from a TaskRun spec:

credentials:
- name: ssh-directory
  secretName: my-secret
- name: git-credentials-file
  secretName: another-secret
  key: gitCredsContent # file, must specify one key from secret
- name: git-password-envvar
  secretName: yet-another-secret
  key: myPassword # envVar, must specify one key in secret

And a breakdown of the individual features:

Credentials are attachable as Directories, Files, or EnvVars

Depending on the choice of directory, file, or envVar in a Credential declaration, Tekton will attach Secrets in different ways:

• A directory credential would likely be mounted as a Secret volume.
  • Use case: A ~/.ssh directory from a Secret containing known_hosts, id_rsa keys.
• file credentials would be mounted as Secret volumes with individual item entries and then those mounted items would be symlinked (or, perhaps by configuration, copied) to their intended destination.
  • Use case: A ~/.git-credentials file with user/password lines.
• envVar credentials would be attached using Container's env.valueFrom to expose the Secret's key as the env var.
  • Use case: The AWS SDK for example utilizes a set of env vars like AWS_ACCESS_KEY_ID

Credentials can be declared as "optional"

• An optional credential does not fail validation of a TaskRun if the cred isn't provided.
• Example use case: A git task supporting a ~/.ssh dir, ~/.git-credentials file, or public repos without auth.

Credentials mount with very restrictive access control

• Files and directories are mounted with as reduced set of permissions as possible (0400 in Linux?)
• In cases where a cred must be copied out of a Secret volume mount (i.e. to put it in $HOME without making the entire $HOME dir read-only), Tekton should do so while maintaining the reduced permission bits.

Steps must explicitly request credentials

• Steps must declare if they're going to use a Credential
• The TaskRun reconciler will only add volumeMounts or env vars for a Credential to those Steps that declare an explicit dependency on them (cf. "sensitive" workspaces in the other design idea)
• Example use case: Only some Steps in a Task should have access to sensitive creds.
• Here's what the YAML for Steps declaring Credentials might look like:

credentials:
- name: ssh-directory
  optional: true
  directory: $HOME/.ssh
steps:
- name: do-something-authenticated
  credentials:
  - ssh-directory # this step receives the ssh-directory volume mount
  image: foo

Problems with this design

• Adds another section to our spec YAMLs
• Implies a reduced set of use cases vs workspaces. Changes to workspaces are more generic/flexible.

[ghost]

Related, I think it would be really cool if tkn supported something like:

tkn tr start my-git-task --credential git-ssh-directory="~/.ssh"

That automatically created a Secret with contents of file/directory and ran the TaskRun with it. Maybe even deleting the Secret afterwards?

[vdemeester]

I definitely prefer the design idea 2. It's more explicit and make a clear distinction from the workspaces

[ghost]

Excellent, I am building a POC of #2 first.

[jlpettersson]

It is great that we focus on improving UX here!

3. creds-init copies the credentials out of those Volumes into /tekton/home or /tekton/creds
4. entrypoint, before each Step, copies the credentials out of /tekton/creds into the Step Container's $HOME

There are a couple of challenges with this:

Secrets get mounted by Kubernetes as in-memory filesystems but creds-init copies those credentials to disk (/tekton/home and /tekton/creds are emptyDir volumes). This diminishes a security feature built into the platform.

I agree, it would be nice to avoid these "moving parts". From a security standpoint - in a best of worlds - I would like to be able to use readOnlyRootFilesystem in the pod security policy and instead have every thing declaratively exposed to the Task/Step.

To follow up from #2398. I think some use of projected volumes can be very clean and useful piece for this kind of solution. We will need some support for secret but also serviceAccountToken (the newer - more secure variant, see No More Forever-tokens) that projected volumes has. We currently seem to use workspaces for many things, but here we could move some things to projected volumes instead (possibly in combination with task-specific syntax)

It's more explicit and make a clear distinction from the workspaces

I agree with that.

Projected Volumes and Secrets get mounted as read-only.

But I see this as a positive thing, it is easier to reason about it from a security standpoint as well.

So a git Task that asks for a single Secret to mount into ~ would not be able to clone a repo into ~/src.

There are many use cases. But for e.g. git it is usually possible to configure credentials path and for e.g. aws cli using AWS_CONFIG_FILE and for ssh it is possible to use -i for Identity key location.

[jlpettersson]

I earlier promised to come up with a proposal on how we can use "projected service accounts" - with tokens that are regularly rotated by the kubelet. (see No more forever tokens )

I have tried this out. Actually, it works as-is with Tekton, without any changes. By using a projected volume. Also, secrets (also optional) and configmaps, specific for a step can be used with the same solution. See my comment with example of secret from projected volume

And this is:

• Without annotating secrets
• Without using the creds-init and its drawbacks

What we lack is:

• better documentation about this method

Here is an example of using a "regularly rotated token" (but for us - the value is that it is a token that expire - and single-audience) in a Task:

apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
  name: task
spec:
  steps:
  - name: lister
    image: <my-custom-pod-lister-image>
    command: list-pods
    volumeMounts:
    - mountPath: /var/run/secrets/rotating-tokens
      name: rotating-token
  volumes:
  - name: rotating-token
    projected:
      sources:
      - serviceAccountToken:
          path: token
          expirationSeconds: 7200
---
apiVersion: tekton.dev/v1beta1
kind: TaskRun
metadata:
  generateName: taskrun-serviceaccount
spec:
  serviceAccountName: lister
  taskRef:
    name: task
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: lister

For a non-root pod, this need to be added to the template:

  securityContext:
    supplementalGroups: [ 65534 ]
    fsGroup: 65534

From the Kubernetes documentation it looks like this solution plays well with a service like Vault - it is used as an example.

If wanted, I can provide a more extensive example or also contribute to documentation about this.

[ghost]

Very interesting! Would there ever be a reason that a TaskRun would want to override the audience? I have to admit I don't fully understand how the audience field works.

This definitely seems worthwhile to document (at least to me!)

Edit to add: I would be really curious to see a real-world example which uses one of these tokens to perform some operation.

[jlpettersson]

Very interesting! Would there ever be a reason that a TaskRun would want to override the audience? I have to admit I don't fully understand how the audience field works.

Yes! When a serviceAccountToken from a projected volume is without audience it is intended to be used for requests to the ApiServer in the same cluster.

By setting a different audience, e.g. audience: vault or audience: docker-registry or audience: deploy-to-cloud-run, the token can not be used for requests to the ApiServer (or any service other than written in audience).

This is great for security. If I have a normal service-account-token today (not from projected volume) and use if from Task-A to authenticate to Service-B. Then if Service-B is compromised, it could use the received token and initiate requests to the ApiServer and do the operations that Task-A is authorized to do (perhaps create namespaces and deploy pods?). The token could aslo be logged and this is dangerous since it never expire - but tokens from projected volumes will expire after minutes/hours.

These different tokens: audience: vault or audience: docker-registry or audience: deploy-to-cloud-run can now only be used for authenticating to the intended services. e.g. within Google Cloud, by using workload identity service accounts can now be "linked" to GCP IAM service-accounts, so that there is no need to copy and distribute secrets (as what I understand, haven't tried).

Essentially, using service accounts to authenticating to different services, is a lot more easy than creating passwords and other credentials that then need to be distributed. ServiceAccounts with tokens that are often automatically rotated and has a single-audience is a very secure way to solve authentication. I would love to only have serviceAccountTokens in my CI/CD pipeline - without any need to handle and distribute secrets.

[wlynch]

Hopping in late, but here's an idea to add on to approach 2, but relies more on the existing k8s Secrets API.

Idea

Treat secrets like workspaces by providing a mechanism to bind them in tasks and allow users to reference them by name to let users use them with the Secrets API.

Example

Task:

secretBinding:
- name: foo
  config: github
...
spec:
  ...
  env:
  - name: SECRET_USERNAME
    valueFrom:
      secretKeyRef:
        name: $(secrets.foo.name)
        key: username
  volumes:
  - name: secret-volume
    secret:
      secretName: $(secrets.foo.name)

TaskRun:

secretBinding:
- name: foo
  ref: my-secret  # Reference to secret within the Namespace 

Config labels

In the task definition, users can specify a user configurable label for what secret to default to if not provided. This allows for most operators to set a credential once, and be able to use the same one automatically across multiple tasks (e.g. remove the burden of having to specify the GitHub token secret for all runs).

If needed, this can be configured per namespace, or overridden at Task/PipelineRun time by providing secretBinding.

e.g.

apiVersion: v1
kind: ConfigMap
metadata:
  name: tekton-creds-config. # Name TBD, but should be some Tekton identifiable config for the namespace.
data:
  github: github-token. # Reference to Secret in the namespace.
  external-sa: sa-config
  ...

Most importantly, this also avoids and list permissions needed for secrets, as long as the controller has access to the configmap (which I expect to not be an issue).

Comparison

The main advantage of configuring things by reference is that it unlocks the full range of the Secret API within tasks, allowing normal pod-like configuration for both file and env var secrets. For example, this would allow Tasks to individually configure projected volumes as requested by @jlpettersson.

Coupling this with a namespaced default configuration means that in most cases, users shouldn't need to actually provide credentials in their task runs assuming they've been set up once for the namespace. Since Secrets are also configured per namespace, this should be a reasonable to do, and we could possibly make this easier to do via tkn (e.g. tkn secrets config <label> <secretRef>?)

This has some drawbacks - notably this takes away some of the ease of setup for secrets within Tasks, and makes the optional secrets difficult since the expectation of most tasks is that if it requires a secret that it will be provided. That said, I'm not sure how common this behavior would be in practice.

At minimum, I think it'd be great to add the config defaulting behavior to make it easier to provide credentials without explicit user configuration.
Open question whether the direct usage of the Secrets API is better than the credentials setup proposed in approach 2.

Thoughts? :D

[jlpettersson]

@sbwsg I created an alternative git-clone task in Catalog, tektoncd/catalog#309 and wrote a README about how to setup SSH authentication to e.g. GitHub, solely by creating Secrets and mount them with a projected volume, and configure the git command to use paths to located them.

It shows an authentication:

• without custom annotations on Secrets
• without modification of ServiceAccounts
• without using credentials initialization process
• without moving or copying any sensitive data

It's different. But I can see how the current git-clone originated from a PipelineResource. I don't know the requirements or history of the current one, but this was an initiative to PoC an alternative.

[ghost]

@sbwsg I created an alternative git-clone task in Catalog, tektoncd/catalog#309 and wrote a README about how to setup SSH authentication to e.g. GitHub, solely by creating Secrets and mount them with a projected volume, and configure the git command to use paths to located them.

It took me way too long to fully understand the implications of this. Essentially I think we're talking about removing any credentials-specific code from Tekton and simply relying on existing Kubernetes mechanisms (and/or external systems like Vault) for credential handling. Is that right? I can definitely see the desirability of this both from a user perspective (don't need to learn something Tekton specific) and from an implementer's view (dont need to maintain something).

[jlpettersson]

@sbwsg I created an alternative git-clone task in Catalog, tektoncd/catalog#309 and wrote a README about how to setup SSH authentication to e.g. GitHub, solely by creating Secrets and mount them with a projected volume, and configure the git command to use paths to located them.

It took me way too long to fully understand the implications of this. Essentially I think we're talking about removing any credentials-specific code from Tekton and simply relying on existing Kubernetes mechanisms (and/or external systems like Vault) for credential handling. Is that right? I can definitely see the desirability of this both from a user perspective (don't need to learn something Tekton specific) and from an implementer's view (dont need to maintain something).

Is that right?

Yes, thats how I meant about it. But I also see that it is not a very clear way forward.

I brought this up in last weeks API WG, and elaborated about this in #2680

I also create a new git-clone-ssh PR tektoncd/catalog@92fb4a0 that includes instructions on how to configure authentication for this alternative task. But this only address ssh-setup.

Note that tektoncd/catalog#332 contains improvements (using TaskRunSpec introduced in #2389) that was not avaliable when I created the tasks in earlier comments above here. E.g. my PR tektoncd/catalog#309 - as how I mounted the volume there is "deprecated"

Also imagePullSecrets was added to TaskRunSpec in #2547

@sbwsg we could create a design doc on the way forward here? As you commented in slack, the way above does not have a clear way forward for ENV, but that could possibly be added to TaskRunSpec as well? We could also elaborate about a "migration path" from the auth we have today, to a more k8s native?

[ghost]

As you commented in slack, the way above does not have a clear way forward for ENV

Looking at the variable replacement code it seems we might have a way to do this today as well.

A Task could declare:

params:
- name: access-key-secret-name
- name: access-key-secret-key
steps:
- name: use-access-key
  image: # something
  env:
    name: SECRET_ACCESS_KEY
    valueFrom:
      secretKeyRef:
        name: $(params.access-key-secret-name)
        key: $(params.access-key-secret-key)
  script: |
    # Do some stuff with $SECRET_ACCESS_KEY

I'm starting to come around to the idea that we might not need to provide Tekton-specific mechanisms for getting credentials into Tasks and Runs. There are still quite a lot of questions in my mind about this - What do errors look like for missing or invalid Secrets? How well do external systems like Vault inject Secrets into Runs? - but it does seem quite desirable to lean on the existing ecosystem for this problem area rather than invent another system.

I also really like the idea of documenting migration away from today's auth/creds-init workflow!

[ghost]

@sbwsg we could create a design doc on the way forward here?

What would be the best way to approach this? Document the existing possibilities (creds-init, Workspace Secrets, "k8s-native" approach that you've described) and then suggest recommendations which we could create documentation for? Beyond documentation do you think there's more we could do?

[jlpettersson]

@sbwsg we could create a design doc on the way forward here?

What would be the best way to approach this? Document the existing possibilities (creds-init, Workspace Secrets, "k8s-native" approach that you've described) and then suggest recommendations which we could create documentation for? Beyond documentation do you think there's more we could do?

I am not sure. I was thinking, maybe we could generate Event from creds-init if the user is using e.g. Secrets with annotations and inform that the user is using a feature that will be deprecated/removed. At the same time we must have a good documentation story about how to do it the "other" way? Eventually we could hide creds-init behind a toggle, that is initially on by default, and a release later it could be off by default (but still manually enabled) and later remove the older auth?

A few streps for current users. But new users could follow and use a newer workflow for configuring auth.

We possibly have to think about how to do this in a smooth way for git-init-task.

Any other ideas?

[ghost]

Ah, nice idea about the Events! I hadn't considered that.

The other thing, and it might be totally impossible, would be to try and introduce guidelines for catalog tasks so that they all uniformly accept credentials in the same way. I'm unsure if that would be feasible though.

The one major benefit of the creds-init approach is that it is a cross-cutting style compared to Workspaces and k8s-native. A Task does not need to specify credentials accepted, the system just implicitly "knows" the way to wire things up. Unfortunately it is just so difficult to debug effectively when it goes wrong.

[jlpettersson]

The one major benefit of the creds-init approach is that it is a cross-cutting style compared to Workspaces and k8s-native. A Task does not need to specify credentials accepted, the system just implicitly "knows" the way to wire things up. Unfortunately it is just so difficult to debug effectively when it goes wrong.

I think this is just "so so true". The fact is that few Catalog tasks uses "credentials". It mostly is git-init - but that can use credentials for both SSH or token.

All Tasks that interact with an external system should be possible to configure with authentication, a Secret or a ServiceAccountToken.

E.g. the kubectl task does not seem to have "credentials" managed by creds-init.

There is probably a lot of use cases to use the curl task that require authentication.

Same for e.g. the gcloud task

The other thing, and it might be totally impossible, would be to try and introduce guidelines for catalog tasks so that they all uniformly accept credentials in the same way. I'm unsure if that would be feasible though.

Yes, it would be great to have common auth guidelines for tasks. I think a problem is that there are multiple kinds of authentication types, e.g. certificates, passwords and most modern is probably serviceAccountTokens. But maybe we could document the different cases.

[ghost]

One additional idea I have is to audit the catalog and make a tally of the existing tasks' creds usage:

• credential mechanism task expects (creds-init, workspace, k8s-native, smthg else)
• manifestation of credential (file, env var, other)
• type of credential (ssh key, certificate, username/pass, api token)
• is cred used in single step or multiple steps?

[ghost]

OK, here's a breakdown of the catalog by Task. For each entry I've recorded the kinds of credential mechanisms supported, how the creds appear in the container (env var / file / etc...), the "type" of cred (user/pass/token/key), whether the cred is exposed to multiple steps or isolated to an individual step, and then any notes i recorded along the way.

https://docs.google.com/spreadsheets/d/1lESYHGDYnx4RfYAs3BQZXrCw5Aa81EWHCaUuxLpp_Sw/edit#gid=0

I haven't processed the data at all yet but it looks like the most common pattern right now is "k8s-native", by which I mean the task explicitly mounts a Secret rather than exposing a workspace or relying on creds-init.

[ghost]

So, made a couple of quick charts from the catalog repo.

First, the breakdown of credential "mechanism". Does the Task expose a workspace? Does it use creds-init? Does it use a platform-dependent mechanism like GKE's Workload Identity? A Task may support multiple mechanisms and so there can be double counts here. It's immediately obvious that the largest grouping are "k8s-native" mechanisms where they mount a Secret as an Env Var or Volume. Some of these Tasks expose params to configure precisely which Secret and Secret Key is used while many do not and simply rely on fixed Secret names and keys.

Second, how are the credentials manifested inside the container? Are they files? Env vars? Arguments passed to a CLI tool? Again a Task may support multiple mechanisms and the mechanisms might manifest differently so there could be double counts. The largest group of Tasks are relying on credentials written to files. There are a number of tasks that are relying on arguments passed to command line tools to provide e.g. username / password. These are a bit worrying to me because it means their usage will expose those credentials in logs / TaskRuns.

Third, what "type" of credential is being supplied? I've categorized these as "userpass", "token", "key" (which includes ssh keys), "cert", "platform-dependent", and "serviceaccount". "platform-dependent" is an interesting case - a Task can expose a workspace and through that accept either files structured for e.g. GKE (.json file) or files structured for e.g. AWS (.aws directory w/ config/credential files). Again caveats about multiple types being supported simultaneously - one task can be counted multiple times.

Edit: I'm going to put some thoughts together and present these at the API WG on Monday.

[ghost]

Over the past few weeks I've been running a survey related to user's knowledge and usage of creds-init. I plan to present the results at the API WG later today but am summarizing them here to make the information more accessible / searchable. We had 21 responses in total, which isn't a huge amount, but I think it's worth being transparent with the data regardless.

Raw data follows but TL;DR here are the highlights from these responses:

1. A large majority of respondents are aware of Tekton's built-in creds mechanism.
2. One fifth of respondents agree that Tekton provides for all of their organization's credential use-cases. The majority of respondents think that Tekton's existing creds mechanism services some -but not all- of their organization's use-cases.
3. Concerns were raised with the level of permissions required by the Tekton controller wrt Secrets, with the opaque nature of the creds-init mechanism, with systems having to support multiple credential mechanisms alongside creds-init due to its limited types, and with the quality of documentation around docker credentials.

---

First question in the survey asked whether users were aware of Tekton's built-in credential mechanism. Response was 90% in the affirmative:

Second question in the survey asked whether the user or their organization uses Tekton's built-in credentials mechanism. 80% said that they did:

The third question asked how well Tekton's creds-init mechanism supports the user's organization's use-cases more broadly. 24% responded that Tekton supported all of their existing use-cases. 67% responded that some of their use-cases were met by Tekton. 5% responded that they do not use Tekton in their organization and 5% requested improved documentation specifically around the Kaniko task's use of Docker credentials:

The final question was a request for any further comments. I've paraphrased them a bit here to avoid identifying info:

We absolutely depend on this feature because it makes it way easier for customers to bring their own auth credentials and wire them up in to their pipelines. It would be nice if this could be expanded to allow a user to configure multiple keys into the same Tekton task.

Works fine tbh, not had any gripes or complaints from users in the field using <our product> which incorporates the Dashboard. Once you've done it for the first time it becomes natural. Anything else is syntactic sugar IMO.

In Openshift console you can change the service account and I might not have permission to edit secrets. In this case I would not use tekton's built in auth and just use config maps.

I'd prefer to use an explicit binding to declare what credentials should be used at the TaskRun/Resource level, similar to how the ClusterResource and PRResource auth work. The implicit one (annotation hints on secrets) is hard to understand and requires the controller to run with more permissions that it should need (list secrets in all namespaces).

We use Tekton's existing auth mechanism in <our product> although when Kubernetes secret values change they are not updated in the build pod because they are not mounted into each container. Another thing is our builds often need a number of secrets, maven settings file, npm tokens, third party keys etc - we'd like to avoid giving the build pod service account RBAC permission to access the namespace secrets so we tend to mount these via a secrets volume, so we use two mechanisms but maybe there is a better way to consolidate approaches?

Auth plugins for docker seem tricky to get right. Not sure I have a better solution, but they seem to cause headaches. I don't know if a sidecar/proxy model would work better for developers (it would be more of a pain for managing, and might run into other issues, since it would effectively MITM the tasks from the registry/git).

(Related to the way Kaniko's support for docker creds works) Docker authentication using docker config file ~/.docker/config.json is something I would like to see it as documented here: https://github.com/tektoncd/pipeline/blob/master/docs/auth.md#basic-authentication-docker in addition to username and password.

[ghost]

Something @bobcatfish raised today while we were talking about credentials is the idea of conformance and platforms conforming to the Tekton API.

In a world where creds-init is part of Tekton conformance it's very difficult to imagine a system being Tekton-compatible on anything but Kubernetes. This is down to the heavy reliance of creds-init on specific k8s concepts like Secrets, Annotations, Service Accounts, etc. But Tekton as an API has always had a goal of being implementable independent of the underlying infra.

[tekton-robot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale with a justification.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close with a justification.
If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/lifecycle stale

Send feedback to tektoncd/plumbing.

[ghost]

/remove-lifecycle stale
/lifecycle frozen

We're quite close to having this issue wrapped up, so keeping it around.