Secrets Provider - Push to File Mode

Table of Contents
Overview
How Push to File Works
Certification Level
Set up Secrets Provider for Push to File
Reference Table of Configuration Annotations
Example Common Policy Path
Example Secret File Formats
Custom Templates for Secret Files
Configuring Pod Volumes and Container Volume Mounts
Secret File Attributes
Deleting Secret Files
Decoding Base64 Encoded Secrets
Upgrading Existing Secrets Provider Deployments
Troubleshooting

Overview

The push to file feature detailed below allows Kubernetes applications to consume Conjur secrets directly through one or more files accessed through a shared, mounted volume. Providing secrets in this way should require zero application changes, as reading local files is a common, platform agnostic delivery method.

The Secrets Provider can be configured to create and write multiple files containing Conjur secrets. Each file is configured independently as a group of Kubernetes Pod Annotations, collectively referred to as a "secret group".

Using annotations for configuration is new to Secrets Provider with this feature and provides a more idiomatic deployment experience.

How Push to File Works

The Secrets Provider, deployed as a Kubernetes init container in the same Pod as your application container, starts up and parses Pod annotations from a Kubernetes Downward API volume. The Pod annotations are organized in secret groups, with each secret group indicating to the Secrets Provider:
- The policy paths from which Conjur secrets should be retrieved.
- The format of the secret file to be rendered for that group.
- How retrieved Conjur secret values should be mapped to fields in the rendered secret file.
The Secrets Provider authenticates to the Conjur server using the Kubernetes Authenticator (authn-k8s).
The Secrets Provider reads all Conjur secrets required across all secret groups.
The Secrets Provider renders secret files for each secret group, and writes the resulting files to a volume that is shared with your application container.
The Secrets Provider init container runs to completion.
Your application container starts and consumes the secret files from the shared volume.

Certification Level

The Secrets Provider push to File feature is a Certified level project. Certified projects have been reviewed and are supported by CyberArk. For more detailed information on our certification levels, see our community guidelines.

Set up Secrets Provider for Push to File

This section describes how to set up the Secrets Provider for Kubernetes for Push to File operation.

Before you begin
- Make sure that a Kubernetes Authenticator has been configured and enabled. For more information, contact your Conjur admin, or see Enable Authenticators for Applications.
  
  In this procedure, we will use dev-cluster for the Kubernetes Authenticator.
- This procedure assumes you have a configured Kubernetes namespace, with a service account for your application. The namespace must be configured with the Namespace Preparation Helm chart.
  
  In this procedure, we will use test-app-namespace for the namespace, and test-app-sa for the service account.
- This procedure assumes that you are familiar with loading K8s manifests into your workspace.
Define the application as a Conjur host in policy

Conjur admin: To enable the Secrets Provider for Kubernetes (cyberark-secrets-provider-for-k8s init container) to retrieve Conjur secrets, it first needs to authenticate to Conjur.
- In this step, you define a Conjur host used to authenticate the cyberark-secrets-provider-for-k8s container with the Kubernetes Authenticator.
  
  The Secrets Provider for Kubernetes must be defined by its namespace and authentication container name, and can also be defined by its service account. These definitions are defined in the host annotations in the policy. For guidelines on how to define annotations, see Application Identity in Kubernetes.
  
  The following policy:
  - Defines a Conjur identity for test-app by its namespace, test-app-namespace, authentication container name, cyberark-secrets-provider-for-k8s, as well as by its service account, test-app-sa.
  - Gives test-app permissions to authenticate to Conjur using the dev-cluster Kubernetes Authenticator.
  Save the policy as apps.yml:
```
- !host
  id: test-app
  annotations:
    authn-k8s/namespace: test-app-namespace
    authn-k8s/service-account: test-app-sa
    authn-k8s/authentication-container-name: cyberark-secrets-provider-for-k8s

- !grant
  roles:
  - !group conjur/authn-k8s/dev-cluster/consumers
  members:
  - !host test-app
```
  NOTE: The value of the host's authn-k8s/authentication-container-name annotation states the container name from which it authenticates to Conjur. When you create the application deployment manifest below, verify that the CyberArk Secrets Provider for Kubernetes init container has the same name.
- Load the policy file to root.
```
$ conjur policy load -f apps.yml -b root
```
Define variables to hold the secrets for your application, and grant the access to the variables

Conjur admin: In this step, you define variables (secrets) to which the Secrets Provider for Kubernetes needs access.
- Save the following policy as app-secrets.yml.
  
  This policy defines Conjur variables and a group that has permissions on the variables.
  
  In the following example, all members of the consumers group are granted permissions on the username and password variables:
```
- !policy
  id: secrets
  body:
    - !group consumers
    - &variables
      - !variable username
      - !variable password
    - !permit
      role: !group consumers
      privilege: [ read, execute ]
      resource: *variables
- !grant
  role: !group secrets/consumers
  member: !host test-app
```
- Load the policy file to root.
```
$ conjur policy load -f app-secrets.yml -b root
```
- Populate the variables with secrets, for example myUser and MyP@ssw0rd!:
```
$ conjur variable set -i secrets/username -v myUser
$ conjur variable set -i secrets/password -v MyP@ssw0rd!
```

Set up the application deployment manifest

Application developer: In this step you set up an application deployment manifest that includes includes an application container, myorg/test-app, and an init container that uses the cyberark/secrets-provider-for-k8s image. The deployment manifest also includes Pod Annotations to configure the Secrets Provider for Kubernetes Push to File feature. The annotations direct the Secrets Provider to generate and write a secret file containing YAML key/value settings into a volume that is shared with the application container.

Copy the following manifest and load it to the application namespace, test-app-namespace.

NOTE: The mountPath values used for the cyberark-secrets-provider-for-k8s container must appear exactly as shown in the manifest below, i.e.:

/conjur/podinfo for the podinfo volume.
/conjur/secrets for the conjur-secrets volume.

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: test-app
  name: test-app
  namespace: test-app-namespace
spec:
  replicas: 1
  selector:
    matchLabels:
      app: test-app
  template:
    metadata:
      labels:
        app: test-app
      annotations:
        conjur.org/authn-identity: host/test-app
        conjur.org/container-mode: init
        conjur.org/secrets-destination: file
        conjur.org/conjur-secrets-policy-path.test-app: secrets/
        conjur.org/conjur-secrets.test-app: |
          - admin-username: username
          - admin-password: password
        conjur.org/secret-file-path.test-app: "./credentials.yaml"
        conjur.org/secret-file-format.test-app: "yaml"
    spec:
      serviceAccountName: test-app-sa
      containers:
      - name: test-app
        image: myorg/test-app
        volumeMounts:
          - name: conjur-secrets
            mountPath: /opt/secrets/conjur
            readOnly: true
      initContainers:
      - name: cyberark-secrets-provider-for-k8s
        image: 'cyberark/secrets-provider-for-k8s:latest'
        imagePullPolicy: Never
        env:
        - name: MY_POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: MY_POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        envFrom:
        - configMapRef:
            name: conjur-connect
        volumeMounts:
          - name: podinfo
            mountPath: /conjur/podinfo
          - name: conjur-secrets
            mountPath: /conjur/secrets
      volumes:
        - name: podinfo
          downwardAPI:
            items:
              - path: "annotations"
                fieldRef:
                  fieldPath: metadata.annotations
        - name: conjur-secrets
          emptyDir:
            medium: Memory

The Secrets Provider will create a secret file in the conjur-secrets shared volume that will appear in the test-app container at location /opt/secrets/conjur/credentials.yaml, with contents as follows:

"admin-username": "myUser"
"admin-password": "myP@ssw0rd!"

Reference Table of Configuration Annotations

Below is a list of Annotations that are used for basic Secrets Provider configuration and to write the secrets to file. All annotations begin with conjur.org/ so they remain unique. Push to File Annotations are organized by "secret groups". A secrets group is a logical grouping of application secrets, typically belonging to a particular component of an application deployment (e.g. all secrets related to a backend database). Each group of secrets is associated with a specific destination file.

Please refer to the Secrets Provider documentation for a description of each environment variable setting:

K8s Annotation	Equivalent Environment Variable	Description, Notes
`conjur.org/authn-identity`	`CONJUR_AUTHN_LOGIN`	Required value. Example: `host/conjur/authn-k8s/cluster/apps/inventory-api`
`conjur.org/container-mode`	`CONTAINER_MODE`	Allowed values: `init` `application` Defaults to `init`. Must be set (or default) to `init`for Push to File mode. NOTE: `sidecar` is supported for Secrets Rotation
`conjur.org/secrets-destination`	`SECRETS_DESTINATION`	Allowed values: `file` `k8s_secrets`
`conjur.org/k8s-secrets`	`K8S_SECRETS`	This list is ignored when `conjur.org/secrets-destination` annotation is set to `file`
`conjur.org/retry-count-limit`	`RETRY_COUNT_LIMIT`	Defaults to 5
`conjur.org/retry-interval-sec`	`RETRY_INTERVAL_SEC`	Defaults to 1 (sec)
`conjur.org/log-level`	`LOG_LEVEL`	Allowed values: `debug` `info` `warn` `error` Defaults to `info`.
`conjur.org/conjur-secrets.{secret-group}`	Note*	List of secrets to be retrieved from Conjur. Each entry can be either: A Conjur variable path A key/value pairs of the form `<alias>:<Conjur variable path>` `[content-type: <type>]` where the `alias` represents the name of the secret to be written to the secrets file and the optional `content-type` is either text or base64, defaulting to text. See Decoding Base64 Encoded Secrets for more information.
`conjur.org/conjur-secrets-policy-path.{secret-group}`	Note*	Defines a common Conjur policy path, assumed to be relative to the root policy. When this annotation is set, the policy paths defined by `conjur.org/conjur-secrets.{secret-group}` are relative to this common path. When this annotation is not set, the policy paths defined by `conjur.org/conjur-secrets.{secret-group}` are themselves relative to the root policy. (See Example Common Policy Path for an explicit example of this relationship.)
`conjur.org/secret-file-path.{secret-group}`	Note*	Relative path for secret file or directory to be written. This path is assumed to be relative to the respective mount path for the shared secrets volume for each container. If the `conjur.org/secret-file-format.{secret-group}` is set to `template`, then this secret file path defaults to `{secret-group}.out`. For example, if the secret group name is `my-app`, the the secret file path defaults to `my-app.out`. Otherwise, this secret file path defaults to `{secret-group}.{secret-group-file-format}`. For example, if the secret group name is `my-app`, and the secret file format is set for YAML, the the secret file path defaults to `my-app.yaml`.
`conjur.org/secret-file-permissions.{secret-group}`	Note*	Explicitly defines secret file permissions. Defaults to `-rw-r--r--` (Octal `644`) Values must be formatted as a valid permission string (Directory bit is optional). For example: `-rw-rw-r--` `rw-rw-r--` Owner must have at a minimum read/write permissions (`-rw-------`)
`conjur.org/secret-file-format.{secret-group}`	Note*	Allowed values: yaml (default) json dotenv bash template This annotation must be set to `template` when using custom templates. (See Example Secret File Formats for example output files.)
`conjur.org/secret-file-template.{secret-group}`	Note*	Defines a custom template in Golang text template format with which to render secret file content. See dedicated Custom Templates for Secret Files section for details.

Note*: These Push to File annotations do not have an equivalent environment variable setting. The Push to File feature must be configured using annotations.

Example Common Policy Path

Given the relationship between conjur.org/conjur-secrets.{secret-group} and conjur.org/conjur-secrets-policy-path.{secret-group}, the following sets of annotations will eventually retrieve the same secrets from Conjur:

conjur.org/conjur-secrets.db: |
  - url: policy/path/api-url
  - policy/path/username
  - policy/path/password

conjur.org/conjur-secrets-policy-path.db: policy/path/
conjur.org/conjur-secrets.db: |
  - url: api-url
  - username
  - password

Example Secret File Formats

Example YAML Secret File

Here is an example YAML format secret file. This format is rendered when the conjur.org/secret-file-format.{secret-group} annotation is set to yaml:

"api-url": "dev/redis/api-url"
"admin-username": "dev/redis/username"
"admin-password": "dev/redis/password"

Example JSON Secret File

Here is an example JSON format secret file. This format is rendered when the conjur.org/secret-file-format.{secret-group} annotation is set to json:

{"api-url":"dev/redis/api-url","admin-username":"dev/redis/username","admin-password":"dev/redis/password"}

Example Bash Secret File

Here is an example bash format secret file. This format is rendered when the conjur.org/secret-file-format.{secret-group} annotation is set to bash:

export api-url="dev/redis/api-url"
export admin-username="dev/redis/username"
export admin-password="dev/redis/password"

Example dotenv Secret File

Here is an example dotenv format file secret file. This format is rendered when the conjur.org/secret-file-format.{secret-group} annotation is set to dotenv:

api-url="dev/redis/api-url"
admin-username="dev/redis/username"
admin-password="dev/redis/password"

Custom Templates for Secret Files

In addition to offering standard file formats, Push to File allows users to define their own custom secret file templates. These templates adhere to Go's text template formatting. Using custom templates requires the following:

The annotation conjur.org/secret-file-format.{secret-group} must be set to template.
The annotation conjur.org/secret-file-path.{secret-group} must be set, and must specify a file name.

The custom template must be provided by only one of the methods listed below, either by Pod annotation or by volume-mounted ConfigMap. Providing a template by both or neither method fails before retrieving secrets from Conjur.

Pod annotation

Custom templates can be defined with the Pod annotation conjur.org/secret-file-template.{secret-group}. The following annotations describe a valid secret group that uses a custom template defined in Pod annotations:

conjur.org/secret-group.example: |
  - admin-username: <variable-policy-path>
  - admin-password: <variable-policy-path>
conjur.org/secret-file-format.example: "template"
conjur.org/secret-file-template.example: |
  "database": {
    "username": {{ secret "admin-username" }},
    "password": {{ secret "admin-password" }},
  }

Volume-mounted ConfigMap

Custom templates can be provided as fields in a ConfigMap. This feature requires the following:
- The ConfigMap containing template files must be created before the Secrets Provider container.
- The ConfigMap must be mounted in the Secrets Provider container at /conjur/templates.
- Each template's key in the ConfigMap's data field must be formatted as {secret-group}.tpl.
Creating a ConfigMap for Secret File Templates

Defining Templates in a ConfigMap Manifest

The following is an example of a ConfigMap manifest defining a custom Go template for a secret group example:
```
apiVersion: v1
kind: ConfigMap
metadata:
  name: my-custom-template
data:
  example.tpl: |
    "database": {
      "username": {{ secret "admin-username" }},
      "password": {{ secret "admin-password" }},
    }
```
If custom secret file templates are required for multiple secret groups, add entries to the data field, adhering to the {secret-group}.tpl key pattern.

Creating a ConfigMap from an Existing Template File

A ConfigMap can also be created from an existing, compatible template file, allowing secret file templates to be checked into version control independent from a K8s manifest.

Given the following template file, example.tpl:
```
"database": {
  "username": {{ secret "admin-username" }},
  "password": {{ secret "admin-password" }},
}
```
Create a ConfigMap from the template files:
```
kubectl create configmap my-custom-template --from-file=/path/to/example.tpl
```
The resulting ConfigMap will be functionally identical to one created from the example ConfigMap manifest shown above.

If secret file templates are required for multiple secret groups, a ConfigMap can be created from a directory of template files, each adhering to the {secret-group}.tpl naming pattern:
```
kubectl create configmap my-custom-template --from-file=/path/to/template/dir/
```
Configuring Secrets Provider to Consume ConfigMap-based Templates

The following annotations describe a valid secret group that uses the custom template created by either of the previously described methods:
```
conjur.org/secret-group.example: |
  - admin-username: <variable-policy-path>
  - admin-password: <variable-policy-path>
conjur.org/secret-file-format.example: "template"
```
In order for Secrets Provider to consume the templates, another volume needs to be added to the application's deployment manifest, populated with the contents of the ConfigMap:
```
volumes:
- name: conjur-templates
  configMap:
    name: my-custom-template
```
Then, mount the ConfigMap-populated volume to the Secrets Provider init container at /conjur/templates:
```
volumeMounts:
- name: conjur-templates
  mountPath: /conjur/templates
```

Using Conjur Secrets in Custom Templates

Injecting Conjur secrets into custom templates requires using the built-in custom template function secret. The action shown below renders the value associated with <secret-alias> in the secret-file.

{{ secret "<secret-alias>" }}

Global Template Functions

Custom templates support global functions native to Go's text/template package. The following is an example of using template function to HTML escape/encode a secret value.

{{ secret "alias" | html }}
{{ secret "alias" | urlquery }}

If the value retrieved from Conjur for alias is "<Hello@World!>", the following file content will be rendered, each HTML escaped and encoded, respectively:

&lt;Hello;@World!&gt;
%3CHello%40World%21%3E

For a full list of global Go text template functions, reference the official text/template documentation.

Additional Template Functions

Custom templates also support a limited number of additional functions. Currently supported functions are:

b64enc: Base64 encode a value.
b64dec: Base64 decode a value.

These can be used as follows:

{{ secret "alias" | b64enc }}
{{ secret "alias" | b64dec }}

When using the b64dec function, an error will occur if the value retrieved from Conjur is not a valid Base64 encoded string.

Execution "Double-Pass"

To avoid leaking sensitive secret data to logs, and to ensure that a misconfigured Push to File workflow fails fast, Push to File implements a "double-pass" execution of custom templates. The template "first-pass" runs before secrets are retrieved from Conjur, and validates that the provided custom template successfully executes given "REDACTED" for each secret value. Redacting secret values allows for secure, complete error logging for malformed templates. The template "second-pass" runs when rendering secret files, and error messages during this stage are sanitized. Custom templates that pass the "first-pass" and fail the "second-pass" require experimenting locally to identify bugs.

NOTE: Custom templates should not branch conditionally on secret values. This may result in a template first-pass execution that doesn't validate all branches of the custom template.

Example Custom Templates: Direct reference to secret values

The following is an example of using a custom template to render secret data by referencing secrets directly using the custom template function secret.

conjur.org/secret-file-path.direct-reference: ./direct.txt
conjur.org/secret-file-template.direct-reference: |
  username | {{ secret "db-username" }}
  password | {{ secret "db-password" }}

Assuming that the following secrets have been retrieved for secret group direct-reference:

db-username: admin
db-password: my$ecretP@ss!

Secrets Provider will render the following content for the file /conjur/secrets/direct.txt:

username | admin
password | my$ecretP@ss!

Example Custom Templates: Iterative approach

The following is an example of using a custom template to render secret data using an iterative process instead of referencing all variables directly.

conjur.org/secret-file-path.iterative-reference: ./iterative.txt
conjur.org/secret-file-template.iterative-reference: |
  {{- range $index, $secret := .SecretsArray -}}
  {{- $secret.Alias }} | {{ $secret.Value }}
  {{- end -}}

Here, .SecretsArray is a reference to Secret Provider's internal array of secrets that have been retrieved from Conjur. For each entry in this array, there is a secret Alias and Value field that can be referenced in the custom template.

Assuming that the following secrets have been retrieved for secret group iterative-reference:

db-username: admin
db-password: my$ecretP@ss!

Secrets Provider will render the following content for the file /conjur/secrets/iterative.txt:

db-username | admin
db-password | my$ecretP@ss!

Example Custom Templates: PostgreSQL connection string

The following is an example of using a custom template to render a secret file containing a Postgres connection string. For a secret group described by the following annotations:

conjur.org/secret-file-path.postgres: ./pg-connection-string.txt
conjur.org/secret-file-template.postgres: |
  postgresql://{{ secret "dbuser" }}:{{ secret "dbpassword" }}@{{ secret "hostname" }}:{{ secret "hostport" }}/{{ secret "dbname" }}??sslmode=require

Assuming that the following secrets have been retrieved for secret group postgres:

dbuser:     "my-user"
dbpassword: "my-secret-pa$$w0rd"
dbname:     "postgres"
hostname:   "database.example.com"
hostport:   5432

Secrets Provider will render the following content for the file /conjur/secrets/pg-connection-string.txt:

postgresql://my-user:my-secret-pa$$w0rd@database.example.com:5432/postgres??sslmode=require

Example Custom Templates: Spring Boot configuration

Many Spring Boot applications use externalized configuration loaded at start-up, usually as an application.yaml or .properties file. For Spring Boot applications deployed in Kubernetes, Secrets Provider's Push-to-File feature can generate properly formatted config files, populated with secret data, using custom templates.

This example assumes that the target Conjur server has been loaded with the following secrets, representing a PostgreSQL database and its credentials:

backend/pg/url
backend/pg/username
backend/pg/password

These secrets will be used by a secret group, spring-app, to render a Spring Boot configuration file.

Create a file defining a Spring Boot configuration template, saved as spring-app.tpl. The following represent template examples for an application.yml template:

spring:
  datasource:
    platform: postgres
    url: jdbc:{{ secret "url" }}
    username: {{ secret "username" }}
    password: {{ secret "password" }}
  jpa:
    generate-ddl: true
    hibernate:
      ddl-auto: update

...and an application.properties template:

spring.datasource.platform: postgres
spring.datasource.url: jdbc:{{ secret "url" }}
spring.datasource.username: {{ secret "username" }}
spring.datasource.password: {{ secret "password" }}
spring.jpa.generate-ddl: true
spring.jpa.hibernate.ddl-auto: update

Create a ConfigMap from the new template file:

kubectl create configmap spring-boot-templates \
  --from-file=/path/to/spring-app.tpl \
  --namespace test-app-namespace

Now that the spring-boot-templates ConfigMap is deployed to the desired namespace, set up a Deployment manifest, with:

a Secrets Provider init container configured for Push-to-File with custom secret file templates,
a Spring Boot application container, configured to consume the newly-generated configuration file with the --spring.config.location flag.

Copy the following manifest and load it into the application namespace, test-app-namespace. Configuration unique to this example is marked in bold.

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: test-app
  name: test-app
  namespace: test-app-namespace
spec:
  replicas: 1
  selector:
    matchLabels:
      app: test-app
  template:
    metadata:
      labels:
        app: test-app
      annotations:
        conjur.org/authn-identity: host/test-app
        conjur.org/container-mode: init
        conjur.org/secrets-destination: file
        conjur.org/conjur-secrets-policy-path.spring-app: backend/pg/
        conjur.org/conjur-secrets.spring-app: |
          - url
          - username
          - password
        conjur.org/secret-file-path.spring-app: "./application.{yaml|properties}"
        conjur.org/secret-file-format.spring-app: "template"
    spec:
      serviceAccountName: test-app-sa
      containers:
      - name: test-app
        image: myorg/test-springboot-app
        command: ["java", "-jar", "/app.jar", "--spring.config.location=file:/opt/secrets/conjur/application.{yaml|properties}"]
        volumeMounts:
          - name: conjur-secrets
            mountPath: /opt/secrets/conjur
            readOnly: true
      initContainers:
      - name: cyberark-secrets-provider-for-k8s
        image: 'cyberark/secrets-provider-for-k8s:latest'
        imagePullPolicy: Never
        env:
        - name: MY_POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: MY_POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        envFrom:
        - configMapRef:
            name: conjur-connect
        volumeMounts:
          - name: podinfo
            mountPath: /conjur/podinfo
          - name: conjur-secrets
            mountPath: /conjur/secrets
          - name: conjur-templates
            mountPath: /conjur/templates
      volumes:
        - name: podinfo
          downwardAPI:
            items:
              - path: "annotations"
                fieldRef:
                  fieldPath: metadata.annotations
        - name: conjur-secrets
          emptyDir:
            medium: Memory
        - name: conjur-templates
          configMap:
            name: spring-boot-templates

Configuring Pod Volumes and Container Volume Mounts for Push to File

Enabling Push to File on your application Pod requires the addition of several Volumes and VolumeMounts. Some example Volume/VolumeMount configurations are shown below.

Example Pod Volumes:

volumes:
  - name: podinfo
    downwardAPI:
      items:
        - path: annotations
          fieldRef:
            fieldPath: metadata.annotations
  - name: conjur-secrets
    emptyDir:
      medium: Memory

Example volume mounts for the Secrets Provider init container. The mountPath values must appear exactly as shown below:

volumeMounts:
  - name: podinfo
    mountPath: /conjur/podinfo
  - name: conjur-secrets
    mountPath: /conjur/secrets

Example volume mounts for the application container. The mountPath to use is dependent upon where your application expects to find secret files:
```
volumeMounts:
  - name: conjur-secrets
    mountPath: <your-desired-path-to-secret-files>
```

Secret File Attributes

By default, the Secrets Provider will create secrets files with the following file attributes:

Attribute	Value	Notes
User	`secrets-provider`
Group	`root`	OpenShift requires that any files/directories that are shared between containers in a Pod must use a GID of 0 (i.e. GID for the root group) The Secrets Provider uses a GID of 0 for secrets files even for non-OpenShift platforms, for simplicity.
UID	`777`
GID	`0`
Permissions	`rw-rw-r--`	As shown in the table table, the Secrets Provider will create secrets files that are world readable. This means that the files will be readable from any other container in the same Pod that mounts the Conjur secrets shared Volume.

The file attributes can be overridden by defining a PodSecurityContext for the application Pod.

For example, to have containers run as the nobody user (UID 65534) and nobody group (GID 65534), and have secrets files created with the corresponding GID, the Pod SecurityContext would be as follows:

    securityContext:
      runAsUser: 65534
      runAsGroup: 65534
      runAsNonRoot: true
      fsGroup: 65534

Deleting Secret Files

Currently, it is recommended that applications do not delete secret files after consuming the files. Kubernetes does not currently restart init containers when primary (i.e. non-init) containers crash and cause liveness or readiness probe failures. Because the Secrets Provider is run as an init container for the Push to File feature, this means that it is not restarted, and therefore secret files are not recreated, following liveness or readiness failures.

Decoding Base64 Encoded Secrets

Secrets can be stored in Conjur as base64 encoded strings. If the applications consuming the secrets cannot be modified to decode the secrets, those secrets become unusable. Secrets Provider can decode the secrets before handing it over to the applications for consumption.

Decoding Base64 with Push to File

Secrets Provider in push to file mode is configured using annotations. The conjur.org/conjur-secrets.{secret-group} annotation will need to be modified to decode secret. Add the content-type annotation. Note the alias must also be defined with the path.

Given the following secrets annotation:

conjur.org/conjur-secrets.db: |
  - url: policy/path/api-url
  - policy/path/username
  - policy/path/password

Update with base64 decode as below:

conjur.org/conjur-secrets.db: |
  - url: policy/path/api-url
  - policy/path/username
  - password: policy/path/password
    content-type: base64

With this annotation the contents of password will be base64 decoded. If the contents cannot be decoded, a warning is displayed in the log files and the contents retrieved will not be decoded.

Decoding Base64 with Kubernetes Secrets

Secrets Provider in Kubernetes Secrets mode is configured using annotations and by configuring a kubernetes secret with a conjur-map.

Given the following Kubernetes Secret configuration

apiVersion: v1
kind: Secret
metadata:
  name: test-app-secrets-provider-k8s-secret
type: Opaque
stringData:
  conjur-map: |-
    DB_URL: test-secrets-provider-k8s-app-db/url
    DB_USERNAME: test-secrets-provider-k8s-app-db/username
    DB_PASSWORD: test-secrets-provider-k8s-app-db/password

To add base64 decoding to the test-secrets-provider-k8s-app-db/password modify the DB_PASSWORD with an id: and content-type: as below.

apiVersion: v1
kind: Secret
metadata:
  name: test-app-secrets-provider-k8s-secret
type: Opaque
stringData:
  conjur-map: |-
    DB_URL: test-secrets-provider-k8s-app-db/url
    DB_USERNAME: test-secrets-provider-k8s-app-db/username
    DB_PASSWORD:
      id: test-secrets-provider-k8s-app-db/password
      content-type: base64

If the contents cannot be decoded, a warning is displayed in the log files and the contents retrieved will not be decoded.

Upgrading Existing Secrets Provider Deployments

At a high level, converting an existing Secrets Provider deployment to use annotation-based configuration and/or Push to File mode can be done in two ways:

Inspect the existing application Deployment manifest (if available) or inspect a live deployment with:
```
kubectl get deployment $DEPLOYMENT_NAME -o yaml
```
Update the manifest with the changes described below, and then re-apply then configuration:
```
kubectl apply -f <updated-manifest-filepath>
```
Edit a live deployment, applying the changes described below, with the interactive command:
```
kubectl edit deployment $DEPLOYMENT_NAME
```

To convert Secrets Provider from K8s Secrets mode to Push to File mode, make the following changes to the deployment:

Convert the Service Provider container/Conjur environment variable settings to the equivalent annotation-based settings. Use the Annotation Reference Table for envvar-to-annotation mappings.
- conjur.org/secrets-destination: file enables Push to File mode.
- conjur.org/conjur-secrets.{group} defines Conjur policy paths previously defined in Kubernetes Secrets.
  - For each existing Kubernetes Secret, you may wish to create a separate secrets group.
  - Inspect the manifests for the existing Kubernetes Secret(s). The manifests should contain a stringData section that contains secrets key/value pairs. Map the stringData entries to a YAML list value for conjur-secrets, using the secret names as aliases.
  - Alternatively, for existing deployments, this mapping can be obtained with the command
```
kubectl get secret $SECRET_NAME -o jsonpath={.data.conjur-map} | base64 --decode
```
- conjur.org/secret-file-path.{group} configures a target file destination.
- conjur.org/secret-file-format.{group} configures a desired file type, depending on how the application will consume the secrets file.
- Remove environment variables used for Secrets Provider configuration from the init container (see Annotation Reference table).
- Remove environment variables referencing Kubernetes Secrets from the application container.

Add push-to-file volumes:

volumes:
  - name: podinfo
    downwardAPI:
      items:
        - path: annotations
          fieldRef:
            fieldPath: metadata.annotations
  - name: conjur-secrets
    emptyDir:
      medium: Memory
  - name: conjur-templates
    emptyDir:
      medium: Memory

Add push-to-file volume mounts to the Secrets Provider init container:

volumeMounts:
  - name: podinfo
    mountPath: /conjur/podinfo
  - name: conjur-secrets
    mountPath: /conjur/secrets
  - name: conjur-templates
    mountPath: /conjur/templates

Add push-to-file volume mount to the application container:

volumeMounts:
  - name: conjur-secrets
    mountPath: /conjur/secrets

Delete existing Kubernetes Secrets or their manifests:
- If using Helm, delete Kubernetes Secrets manifests and do a helm upgrade ...
- Otherwise, kubectl delete ... the existing Kubernetes Secrets
Modify application to consume secrets as files:
- Modify application to consume secrets files directly, or...
- Modify the Deployment's spec for the app container so that the command entrypoint includes sourcing of a bash-formatted secrets file.

Troubleshooting

This section describes how to troubleshoot common Secrets Provider for Kubernetes issues.

Enable debug logs

To enable debug logs, add the debug parameter to the application deployment manifest.

annotations:
    conjur.org/log-level  "debug"

For details, see Reference Table of Configuration Annotations

Display logs

To display the Secrets Provider logs in Kubernetes.

Go to the app namespace

kubectl config set-context --current --namespace=<namespace>

Find the pod name

kubectl get pods

Display the logs

kubectl logs <pod-name> -c <init-container-name>

To display the Secrets Provider logs in Openshift.

Go to the app namespace

oc project <project-name>

Find the pod name

oc get pods

Display the logs

oc logs <pod-name> -c <init-container-name>

Common issues displayed in the logs and resolutions

Issue	Error code	Resolution
Secrets Provider for Kubernetes failed to update secrets in file mode	`CSPFK039E`	Check that the secret key pairs have been configured correctly.
Failed to open annotations file	`CSPFK041E`	The annotations file cannot be opened by the Secrets provider. Check the volume mounts and downward API configuration.
Annotation 'x' does not accept value	`CSPFK042E`	This annotation requires a specific type for its value. The value provided is the wrong type.
Annotation 'x' does not accept value	`CSPFK043E`	This annotation only accepts one of a specific set of values. The value provided is the wrong value.
Annotation file line 'x' is malformed:	`CSPFK045E`	The annotation described in the message is malformed. Check the annotation has a valid key and value
Secret Store Type needs to be configured	`CSPFK046E`	Either the conjur.org/secrets-destination annotation has not been set, or the downward API volumes have not been configured as required. See the Configuring Pod Volumes and Container Volume Mounts section above for more information.
Secrets Provider in Push-to-File mode can only be configured with Pod annotations	`CSPFK047E`	To configure push-to-file the Secrets store type must have the `conjur.org/secrets-destination` with pod annotations and cannot be set with the `SECRETS_DESTINATION` environment variable. If the secrets-destination is set via annotations verify that the pod volumes and volumeMounts are configured correctly. See the Configuring Pod Volumes and Container Volume Mounts section above for more information.
Secrets Provider in K8s Secrets mode requires either the 'K8S_SECRETS' environment variable or 'conjur.org/k8s-secrets'	`CSPFK048E`	If the `secrets-destination` is set to `k8s_secrets` then the `K8S_SECRETS` environment variable or `conjur.org/k8s-secrets` needs to be configured. This does not apply to push-to-file.
Failed to validate Pod annotations	`CSPFK049E`	The service provider was unable to successfully parse the annotations. This could be due to a previous error. Check the logs for a specific error before this.
Unable to initialize Secrets Provider: unable to create secret group collection	`CSPFK053E`	Secrets provider could be initialized. Check futher back in the log file for any specific configuration errors.
Unable to initialize Secrets Provider: unrecognized Store Type	`CSPFK054E`	The `secrets-destination` value, either defined by the `SECRETS_DESTINATION` environment variable or the `conjur.org/secrets-destination` is an invalid value. `conjur.org/secrets-destination` should be `files` or `k8s-secrets`.

Other Common issues resolutions

Issue	Resolution
There are no files created and no errors in the logs	Verify there is no SECRETS_DESTINATION environmental variable set and the volumes are set up correctly.

Files

PUSH_TO_FILE.md

Latest commit

History