k8s-resource

A Concourse resource for retrieving resources from a kubernetes cluster, along with a general purpose put for running any kubectl command.

Table of Contents

• Source Configuration
• Behavior
  • check: Check for new k8s resource(s)
  • in: Retrieve the k8s resource
  • out: Execute a kubectl command
• Examples
  • get Resources
  • put Resources
  • put with await
  • put with await leveraging check filters
  • put with await custom conditions
  • get and put Resources

Source Configuration

• url: Required. The kubernetes server URL, e.g. "https://my-cluster:8443".

• token: Required. Authorization token for the api server.

• certificate_authority: Required. The certificate authority for the api server.

    certificate_authority: |
      -----BEGIN CERTIFICATE-----
      ...
      -----END CERTIFICATE-----

• insecure_skip_tls_verify: Optional. If true, ignores certificate_authority and skips the validity check of the kubernetes server’s certificate. Default is false.

  🔥	Use with caution. This makes the HTTPS connection insecure!

• resource_types: Optional. Comma separated list of resource type(s) to retrieve (defaults to just pod).

    resource_types: deployment,service,pod

• namespace: Optional. The namespace to restrict the query to.
  For check/get, this will default to all namespaces (--all-namespaces).
  For put, this will default to remaining unset (not specified), but can be overridden with a step param (see below).

• filter: Optional. Can contain any/all the following criteria:

  • selector: Specify a label --selector for the query. Can use any valid label selector expression. Defaults to empty.

    source:
      filter:
        selector: app=my-app,app.kubernetes.io/component in (frontend, backend)

    ℹ️	Selectors are the only filter that are performed server-side, and can help cull down the response before it passes through the rest of the filters (below). This can speed up check operations when dealing with a potentially large volume of resources.

  • name: Matches against the metadata.name of the resource. Supports both literal (my-ns-1) and regular expressions ("my-ns-[0-9]*$").

    source:
      filter:
        name: "my-ns-[0-9]*$"

  • olderThan: Time in seconds that the metadata.creationTimestamp must be older than (ex: 86400 for 24hrs).

    source:
      filter:
        olderThan: 86400

  • youngerThan: Time in seconds that the metadata.creationTimestamp must be younger than (ex: 3600 for 1hr).

    source:
      filter:
        youngerThan: 3600

  • phases: List of status.phase value(s) the resource must match at least one of. This varies depending on the resource. For example, a pod’s status can be one of Pending, Running, Succeeded, Failed or Unknown. To retrieve only Failed or Unknown pods:

    source:
      filter:
        phases:
          - Failed
          - Unknown

  • jq: Apply any other custom filter(s) on the JSON returned by kubectl. For example, to retrieve only resources whose image comes from the registry registry.acme.io:

    source:
      filter:
        jq:
          - '.spec.containers[] | .image | startswith("registry.acme.io")'

    We can add multiple filters that are (by default) "OR’d" together. For example, to also look for any resource with a container that has restarted more than 10 times:

    source:
      filter:
        jq:
          - '.spec.containers[] | .image | startswith("registry.acme.io")'
          - '.status.containerStatuses[] | .restartCount > 10'

    💡	This is advanced usage and requires good knowledge about jq and the Kubernetes API. Best approach to writing such queries is to experiment directly with kubectl and jq: kubectl ... get ... -o json | jq <query>

    When at least one jq filter is present, the following additional options can be configured (optionally):

    • jq_operator: Defaults to , - the basic identity operator which combines them as "OR". It can be any filter joining operation jq understands, including + and - (see jq Manual: Basic Filters). Building on the example above, if we wanted to constrain the matches to only resources that matched both filters together, we can use the and operator:

      source:
        filter:
          jq:
            - '.spec.containers[] | .image | startswith("registry.acme.io")'
            - '.status.containerStatuses[] | .restartCount > 10'
          jq_operator: 'and'

    • jq_transform: Defaults to empty (do nothing) - specifies a final JSON transform of the result(s) matched by the list of jq queries. It can be used to alter the structure of the matched json or even produce a completely new json. The interactive equivalent to this is:

      kubectl ... get ... -o json | jq "[.[] | select( $MATCH_QUERY ) ] | unique $TRANSFORM_QUERY"

      ⚠️

      Use with caution. Whatever the transformation is, it should also include the metadata: {uid: "…​", resourceVersion: "…​"} structure, because this is reported to Concourse as the result of the check. See here for an example. The empty result [] appears to not be considered a new version by Concourse (does not trigger a job) - the transform query can make use of that in a condition where it does not want to produce a new version.

• sensitive: Optional. If true, the resource content will be considered sensitive and not show up in the logs or Concourse UI. Can be overridden as a param to each get step. Default is false.

Behavior

check: Check for new k8s resource(s)

The current list of resource_types resources are fetched from the cluster, and filtered against any filter criteria configured. Each matching resource is emitted as a separate version, uniquely identified by its uid/resourceVersion pair.

New versions will be triggered by encountering any of:

• new uid not seen before

• new resourceVersion for a uid (that was previously seen at a different resourceVersion)

ℹ️

Due to the way Concourse treats the versions from the first check, this resource will emit only a single initial resource version (or zero if none match). It will be the first resource in the list returned from the query. All subsequent check invocations after that will always emit the full batch of resources as individual versions. This is done to give pipelines the opportunity to run across each k8s resource. Otherwise, if all versions were emitted from the first initial check, Concourse would only trigger on the last version in the list.

in: Retrieve the k8s resource

Retrieve the single resource as JSON (-o json) and writes it to a file resource.json.

{
  "apiVersion": "v1",
  "kind": "...",
  "metadata": {...},
  ...
}

Parameters

• sensitive: Optional. Overrides the source configuration’s value for this particular get.

out: Execute a kubectl command

General purpose execution of kubectl with args provided as a param to put.

Parameters

• kubectl: Required. The args to pass directly to kubectl.

  ℹ️	The --server, --token, --certificate-authority and --namespace will all be implicitly included in the command based on the source configuration.

• namespace: Optional. Overrides the source configuration’s value for this particular put step.

• await: Optional. Configures the put step to poll the cluster (after running the kubectl command) for resources and await certain conditions before succeeding. Has the following configuration:

  • timeout: Required. Must be a positive integer to enable waiting (anything else disables waiting). Measured in seconds.

  • interval: Optional. Polling interval, measured in seconds (defaults to 3).

  • resource_types: Optional. Overrides the source config resources_types for what to retrieve from the cluster and run through the conditions.

  • conditions: Optional. List of zero or more jq expressions to evaluate. If none are given, default expressions are inferred based on the resource_types being retrieved.

Wait Conditions

Wait conditions are expressed as jq expressions listed under await.conditions in the put step params (similar to the filter.jq list in source configuration of the resource). The conditions are given each resource’s root JSON object.

- put: k8s
  params:
    kubectl: create deployment my-nginx --image=nginx
    await:
      timeout: 30 # seconds
      resource_types: deployment
      conditions:
        - select(.spec.replicas > 0) | .status.readyReplicas > 0

• Can list zero or more conditions (see defaults below for when none are given).

• Each expression must evaluate to a boolean result (true or false), all other results are ignored.

• All conditions must produce at least one true result, and no false results.

• If the timeout is reached before the conditions are satisfied, put will fail.

❗	Be sure to craft your expressions to safely filter out or ignore any resources you don’t care about, taking note of the resource_types you are querying for. Any false result will in any condition will prevent the wait from succeeding.

Default Wait Conditions

If no conditions are given, wait will attempt to infer sensible default conditions based on the resource_types. The table below list the conditions that are used by default.

resource_types	Default Condition
pod, pods, po	select(.kind == "Pod") | .status.containerStatuses[] | .ready
deployment, deployments, deploy	select(.kind == "Deployment") | select(.spec.replicas > 0) | .spec.replicas == .status.readyReplicas
replicaset, replicasets, rs	select(.kind == "ReplicaSet") | select(.spec.replicas > 0) | .spec.replicas == .status.readyReplicas
statefulset, statefulsets, sts	select(.kind == "StatefulSet") | select(.spec.replicas > 0) | .spec.replicas == .status.readyReplicas

ℹ️	The default source config resource_types is pod.

Examples

get Resources

The pipeline below checks for kubernetes namespaces named my-ns-<number> created more than 24 hours ago.

resource_types:
  - name: k8s-resource
    type: docker-image
    source:
      repository: jgriff/k8s-resource

resources:
  - name: expired-namespace
    type: k8s-resource
    icon: kubernetes
    source:
      url: ((k8s-server))
      token: ((k8s-token))
      certificate_authority: ((k8s-ca))
      resource_types: namespaces        (1)
      filter:
        name: "my-ns-[0-9]*$"           (2)
        olderThan: 86400                (3)

jobs:
  - name: view-expired-namespaces
    plan:
      - get: expired-namespace
        version: every
        trigger: true
      - task: take-a-look
        config:
          platform: linux
          image_resource:
            type: registry-image
            source: { repository: busybox }
          inputs:
            - name: expired-namespace
          run:
            path: cat
            args: ["expired-namespace/resource.json"]

1. are namespaces.

2. are named my-ns-<number> (e.g my-ns-1, my-ns-200, etc).

3. have existed for longer than 24 hours (86400 seconds).

Each k8s resource that matches the above criteria is emitted individually from the expired-namespace resource, and then the take-a-look task echoes the contents of the retrieved resource file (for demonstration purposes).

ℹ️	Be sure to include version: every in your get step so you get every k8s resource that matches your query. Otherwise, Concourse will only trigger on the latest resource to be emitted (the last one in the list that comes back from the query).

put Resources

The pipeline below demonstrates using the put operation to deploy a resource file deploy.yaml from a git repo my-k8s-repo (config not shown).

resource_types:
  - name: k8s-resource
    type: docker-image
    source:
      repository: jgriff/k8s-resource

resources:
  - name: k8s
    type: k8s-resource
    icon: kubernetes
    source:
      url: ((k8s-server))
      token: ((k8s-token))
      certificate_authority: ((k8s-ca))

jobs:
  - name: deploy-prod
    plan:
      - get: my-k8s-repo
        trigger: true
      - put: k8s
        params:
          kubectl: apply -f my-k8s-repo/deploy.yaml
          namespace: prod

put with await

Here’s the same example as above, with the added await behavior where put will wait up to 2 minutes for the deployment to come up. If the deployment isn’t ready after 2 minutes, put will fail.

resource_types:
  - name: k8s-resource
    type: docker-image
    source:
      repository: jgriff/k8s-resource

resources:
  - name: k8s
    type: k8s-resource
    icon: kubernetes
    source:
      url: ((k8s-server))
      token: ((k8s-token))
      certificate_authority: ((k8s-ca))

jobs:
  - name: deploy-prod
    plan:
      - get: my-k8s-repo
        trigger: true
      - put: k8s
        params:
          kubectl: apply -f my-k8s-repo/deploy.yaml
          namespace: prod
          await:
            timeout: 120
            resource_types: deployment

put with await leveraging check filters

Since await uses check to retrieve the resources, all the source.filter options are available to you when querying for resources to check against your conditions.

For example:

resources:
  - name: k8s
    type: k8s-resource
    icon: kubernetes
    source:
      url: ((k8s-server))
      token: ((k8s-token))
      certificate_authority: ((k8s-ca))
      namespace: prod
      resource_types: deployment
      filter:
        selector: app=my-app,app.kubernetes.io/component in (frontend, backend)
        name: "my-*"

jobs:
  - name: deploy-prod
    plan:
      - get: my-k8s-repo
        trigger: true
      - put: k8s
        params:
          kubectl: apply -f my-k8s-repo/deploy.yaml
          await:
            timeout: 120

This will:

1. Apply our deployment from deploy.yaml.

2. Then wait at most 2 minutes for all deployments to reach a ready state (default condition for deployment resource types) whose:

   • name starts with "my-".

   • have a metadata label "app.kubernetes.io/component" of either "frontend" or "backend".

put with await custom conditions

You can supply any custom condition to await on.

jobs:
  - name: deploy-prod
    plan:
      - get: my-k8s-repo
        trigger: true
      - put: k8s
        params:
          kubectl: apply -f my-k8s-repo/deploy.yaml
          namespace: prod
          await:
            timeout: 120
            resource_types: deployment,statefulset
            conditions:
              - select(.metadata.name == "my-deployment") | .status.readyReplicas > 0
              - select(.metadata.name == "my-statefulset") | .status.readyReplicas > 0

get and put Resources

The pipeline below demonstrates using both get and put in the same pipeline.

⚠️

Don’t use the same k8s-resource instance for both get and put operations! The put step emits a meaningless version (it’s just the kubectl command that was executed). The problem is Concourse will include that (meaningless) version in the version history for the resource. It will then be offered to your get step which will be unable to retrieve the nonsensical version and then fail. So the best way to deal with this is to use one resource instance for the resources you are get'ing, and another instance for general purpose put'ing things.

Here’s an example that combines the previous 2 examples into a single pipeline that watches for expired namespaces, and then deletes them.

k8s-resource-source-config: &k8s-resource-source-config
  url: ((k8s-server))
  token: ((k8s-token))
  certificate_authority: ((k8s-ca))

resource_types:
  - name: k8s-resource
    type: docker-image
    source:
      repository: jgriff/k8s-resource

resources:
  - name: k8s
    type: k8s-resource
    icon: kubernetes
    source:
      << : *k8s-resource-source-config

  - name: expired-namespace
    type: k8s-resource
    icon: kubernetes
    source:
      << : *k8s-resource-source-config
      resource_types: namespaces
      filter:
        name: "my-ns-[0-9]*$"
        olderThan: 86400
        phases: [Active]

jobs:
  - name: delete-expired-namespaces
    plan:
      - get: expired-namespace
        version: every
        trigger: true
      - load_var: expired-namespace-resource
        file:     expired-namespace/resource.json
      - put: k8s
        params:
          kubectl: delete namespace ((.:expired-namespace-resource.metadata.name))