teleport-cluster-auth-test job failing when upgrading from 15.2.5 to 15.3.0

[PierreBesson]

Expected behavior:

The upgrade should be successful and not fail.

Current behavior:

Test job failing with error message:

FAIL /etc/teleport/teleport.yaml
ERROR: failed parsing the config file: yaml: unmarshal errors:  line 1: field Error not found in type config.FileConfig

The faulty confimap teleport-cluster-auth-test is rendered as:

Name:         teleport-cluster-auth-test
Namespace:    teleport
Labels:       app.kubernetes.io/component=auth
              app.kubernetes.io/instance=teleport-cluster
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=teleport-cluster
              app.kubernetes.io/version=15.3.0
              argocd.argoproj.io/instance=teleport-cluster-public
              helm.sh/chart=teleport-cluster-15.3.0
              teleport.dev/majorVersion=15
Annotations:  helm.sh/hook: pre-install,pre-upgrade
              helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
              helm.sh/hook-weight: 4

Data
====
apply-on-startup.yaml:
----
kind: token
version: v2
metadata:
  name: teleport-cluster-proxy
  expires: "3000-01-01T00:00:00Z"
spec:
  roles: [Proxy]
  join_method: kubernetes
  kubernetes:
    allow:
      - service_account: "teleport:teleport-cluster-proxy"

teleport.yaml:
----
Error: 'error converting YAML to JSON: yaml: line 20: did not find expected key'

[omitted our config from `auth.teleportConfig`]

BinaryData
====

From our debugging this is caused by the faulty configmap merge in this line: https://github.com/gravitational/teleport/blob/branch/v15/examples/chart/teleport-cluster/templates/auth/predeploy_config.yaml#L34

The issue was probably introduce by this change f431c3b

[hugoShaka]

[omitted our config from auth.teleportConfig]

To validate and reproduce the issue we need your values.yaml with the potential secrets redacted, and the generated auth configmap which contans invalid YAML.

[PierreBesson]

@hugoShaka thank you for the response. The teleport-cluster-auth-test configmap was posted in the original issue.

Here is our values.yaml for our teleport-cluster helm release:

teleport-cluster:
  enterprise: true
  proxyProtocol: "off"
  auth:
    teleportConfig:
      auth_service:
        client_idle_timeout: 4h
        web_idle_timeout: 4h
  authentication:
    type: oidc
    secondFactor: "on"
  proxyListenerMode: multiplex
  exposeDiagPort: false
  sessionRecording: "'off'"
  operator:
    enabled: false
  chartMode: gcp
  gcp:
    backendTable: teleport-backend
    auditLogTable: teleport-auditlog
    auditLogMirrorOnStdout: true
    sessionRecordingBucket: redacted-session-recordings
    credentialSecretName: ""
  podMonitor:
    enabled: true
  highAvailability:
    replicaCount: 2
    requireAntiAffinity: true
    podDisruptionBudget:
      enabled: true
      minAvailable: 1
    certManager:
      enabled: true
      addCommonName: true
      issuerName: letsencrypt-dns01
      issuerKind: ClusterIssuer
      issuerGroup: cert-manager.io
  resources:
    requests:
      cpu: 500m
      memory: 1Gi
    limits:
      memory: 2Gi
  log:
    level: INFO
    output: stderr
    format: json
    extraFields: ["timestamp", "level", "component", "caller"]
  priorityClassName: high-priority
  exposeDiagPort: true
  authentication:
    localAuth: true
  clusterName: teleport.example.com
  gcp:
    projectId: example
    sessionRecordingBucket: example-session-recordings
  global:
    license: redacted_teleport_license

[hugoShaka]

sessionRecording: "'off'"

This is the broken bit, you're relying on the previous bug and are working around by encapsulating a string in a string. Since f431c3b and as indicated in the release notes, the typing bug is solved and you no longer have to hack in the chart for the field to stay string type.

The correct value to disable session recording is

sessionRecording: 'off'

[PierreBesson]

Thank you @hugoShaka, I confirm this fixes our issue with the upgrade.

You help is very much appreciated 👍 !

[kriegster108]

just wanted to say, thankyou for this feedback. resolved upgrading from 15.0.0 -> 16.0.1