Skip to content

ERROR TLS setup failed: failed to load CA: (null) #1215

Open
#1271
Open
ERROR TLS setup failed: failed to load CA: (null)#1215
#1271

Description

@Erilbeth
Erilbeth
opened on Dec 2, 2024

Summary:
When client_tls settings were configured liked below, it gives these errors:

2024-12-02 12:12:56.629 UTC [1] ERROR TLS setup failed: failed to load CA: (null)
2024-12-02 12:12:56.629 UTC [1] FATAL TLS setup failed

          client_tls_sslmode = require
          client_tls_key_file = /etc/ssl/certs/tls.key
          client_tls_cert_file = /etc/ssl/certs/tls.crt

Cert is created by cert-manager(let's encrypt) and verified that it's working on an nginx ingress.

deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: pgbouncer-test
  name: pgbouncer-test
  namespace: default
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: pgbouncer-test
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: pgbouncer-test
    spec:
      containers:
      - env:
        - name: PGBOUNCER_PORT
          value: "6432"
        - name: POSTGRESQL_USERNAME
          valueFrom:
            secretKeyRef:
              key: PGUSER
              name: pg-credentials
        - name: POSTGRESQL_PASSWORD
          valueFrom:
            secretKeyRef:
              key: PGPASSWORD
              name: pg-credentials
        - name: POSTGRESQL_HOST
          value: 127.0.0.1
        - name: POSTGRESQL_PORT
          value: "5432"
        image: docker.io/bitnami/pgbouncer:latest
        imagePullPolicy: IfNotPresent
        name: pgbouncer-test
        ports:
        - containerPort: 6432
          protocol: TCP
        resources:
          limits:
            cpu: 500m
            memory: 256Mi
          requests:
            cpu: 100m
            memory: 128Mi
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /bitnami/pgbouncer/conf/
          name: pgbouncer-config
        - mountPath: /etc/ssl/certs
          name: ssl-certs
        - mountPath: /var/log/pgbouncer
          name: pgbouncer-log
        - mountPath: /var/run/pgbouncer
          name: pgbouncer-run
        - mountPath: /etc/ssl/le
          name: pgbouncer-le-ca-cert
      - command:
        - /cloud-sql-proxy
        - -c=/secrets/credentials.json
        - "this section contains sensitive information so i've removed."
        image: gcr.io/cloud-sql-connectors/cloud-sql-proxy:2.14.0
        imagePullPolicy: IfNotPresent
        name: cloud-sql-proxy
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /secrets
          name: cloudsql-instance-credentials
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
      volumes:
      - configMap:
          defaultMode: 420
          name: pgbouncer-config
        name: pgbouncer-config
      - name: ssl-certs
        secret:
          defaultMode: 420
          secretName: smart-mycompany-com-tls
      - name: pgbouncer-le-ca-cert
        secret:
          defaultMode: 420
          secretName: pgbouncer-le-ca-cert
      - name: cloudsql-instance-credentials
        secret:
          defaultMode: 420
          secretName: cloudsql-instance-credentials
      - emptyDir: {}
        name: pgbouncer-log
      - emptyDir: {}
        name: pgbouncer-run

configmap.yaml

apiVersion: v1
data:
  pgbouncer.ini: |
    [databases]
    * = host=127.0.0.1 port=5432

    [pgbouncer]
    verbose=2
    listen_addr = 0.0.0.0
    listen_port = 6432
    auth_type = trust
    auth_file = /bitnami/pgbouncer/conf/userlist.txt
    client_tls_sslmode = require
    client_tls_key_file = /etc/ssl/certs/tls.key
    client_tls_cert_file = /etc/ssl/certs/tls.crt
    logfile = /var/log/pgbouncer/pgbouncer.log
    pidfile = /var/run/pgbouncer/pgbouncer.pid
    admin_users = pgbouncer_user
    ignore_startup_parameters = extra_float_digits
  userlist.txt: |
    "pgbouncer" "sensitive"
kind: ConfigMap

Full logs, verbose=2

Defaulted container "pgbouncer-test" out of: pgbouncer-test, cloud-sql-proxy
pgbouncer 12:33:20.32 INFO  ==>
pgbouncer 12:33:20.32 INFO  ==> Welcome to the Bitnami pgbouncer container
pgbouncer 12:33:20.33 INFO  ==> Subscribe to project updates by watching https://github.com/bitnami/containers
pgbouncer 12:33:20.33 INFO  ==> Submit issues and feature requests at https://github.com/bitnami/containers/issues
pgbouncer 12:33:20.33 INFO  ==> Upgrade to Tanzu Application Catalog for production environments to access custom-configured and pre-packaged software components. Gain enhanced features, including Software Bill of Materials (SBOM), CVE scan result reports, and VEX documents. To learn more, visit https://bitnami.com/enterprise
pgbouncer 12:33:20.34 INFO  ==>
pgbouncer 12:33:20.40 INFO  ==> ** Starting PgBouncer setup **
pgbouncer 12:33:20.42 INFO  ==> Validating settings in PGBOUNCER_* env vars...
pgbouncer 12:33:20.43 INFO  ==> Initializing PgBouncer...
pgbouncer 12:33:20.44 INFO  ==> Found configuration files in
pgbouncer 12:33:20.50 INFO  ==> Waiting for PostgreSQL backend to be accessible
pgbouncer 12:33:20.52 INFO  ==> Backend 127.0.0.1:5432 accessible
pgbouncer 12:33:20.53 INFO  ==> Configuring credentials
pgbouncer 12:33:20.53 INFO  ==> Creating configuration file
pgbouncer 12:33:20.54 INFO  ==> Loading custom scripts...
pgbouncer 12:33:20.54 INFO  ==> ** PgBouncer setup finished! **

pgbouncer 12:33:20.61 INFO  ==> ** Starting PgBouncer **
2024-12-02 12:33:20.627 UTC [1] DEBUG parse_ini_file: 'verbose' = '2' ok:1
2024-12-02 12:33:20.627 UTC [1] DEBUG parse_ini_file: 'listen_addr' = '0.0.0.0'
2024-12-02 12:33:20.627 UTC [1] DEBUG parse_ini_file: 'listen_addr' = '0.0.0.0' ok:1
2024-12-02 12:33:20.627 UTC [1] DEBUG parse_ini_file: 'listen_port' = '6432'
2024-12-02 12:33:20.627 UTC [1] DEBUG parse_ini_file: 'listen_port' = '6432' ok:1
2024-12-02 12:33:20.627 UTC [1] DEBUG parse_ini_file: 'auth_type' = 'trust'
2024-12-02 12:33:20.627 UTC [1] DEBUG parse_ini_file: 'auth_type' = 'trust' ok:1
2024-12-02 12:33:20.627 UTC [1] DEBUG parse_ini_file: 'auth_file' = '/bitnami/pgbouncer/conf/userlist.txt'
2024-12-02 12:33:20.627 UTC [1] DEBUG parse_ini_file: 'auth_file' = '/bitnami/pgbouncer/conf/userlist.txt' ok:1
2024-12-02 12:33:20.627 UTC [1] DEBUG parse_ini_file: 'client_tls_sslmode' = 'require'
2024-12-02 12:33:20.627 UTC [1] DEBUG parse_ini_file: 'client_tls_sslmode' = 'require' ok:1
2024-12-02 12:33:20.627 UTC [1] DEBUG parse_ini_file: 'client_tls_key_file' = '/etc/ssl/certs/tls.key'
2024-12-02 12:33:20.627 UTC [1] DEBUG parse_ini_file: 'client_tls_key_file' = '/etc/ssl/certs/tls.key' ok:1
2024-12-02 12:33:20.627 UTC [1] DEBUG parse_ini_file: 'client_tls_cert_file' = '/etc/ssl/certs/tls.crt'
2024-12-02 12:33:20.627 UTC [1] DEBUG parse_ini_file: 'client_tls_cert_file' = '/etc/ssl/certs/tls.crt' ok:1
2024-12-02 12:33:20.627 UTC [1] DEBUG parse_ini_file: 'logfile' = '/var/log/pgbouncer/pgbouncer.log'
2024-12-02 12:33:20.627 UTC [1] DEBUG parse_ini_file: 'logfile' = '/var/log/pgbouncer/pgbouncer.log' ok:1
2024-12-02 12:33:20.627 UTC [1] DEBUG parse_ini_file: 'pidfile' = '/var/run/pgbouncer/pgbouncer.pid'
2024-12-02 12:33:20.627 UTC [1] DEBUG parse_ini_file: 'pidfile' = '/var/run/pgbouncer/pgbouncer.pid' ok:1
2024-12-02 12:33:20.627 UTC [1] DEBUG parse_ini_file: 'admin_users' = 'pgbouncer_user'
2024-12-02 12:33:20.627 UTC [1] DEBUG parse_ini_file: 'admin_users' = 'pgbouncer_user' ok:1
2024-12-02 12:33:20.627 UTC [1] DEBUG parse_ini_file: 'ignore_startup_parameters' = 'extra_float_digits'
2024-12-02 12:33:20.627 UTC [1] DEBUG parse_ini_file: 'ignore_startup_parameters' = 'extra_float_digits' ok:1
2024-12-02 12:33:20.628 UTC [1] DEBUG loading auth_file: "/bitnami/pgbouncer/conf/userlist.txt"
2024-12-02 12:33:20.633 UTC [1] ERROR TLS setup failed: failed to load CA: (null)
2024-12-02 12:33:20.633 UTC [1] FATAL TLS setup failed

Activity

  1. emelsimsek commented on Feb 6, 2025

    @emelsimsek
    emelsimsek
    on Feb 6, 2025
    Contributor

    @Erilbeth what is your connection string to pgbouncer? Do you use verify-full or verify-ca ?

  2. Erilbeth commented on Feb 6, 2025

    @Erilbeth
    Erilbeth
    on Feb 6, 2025
    Author

    @Erilbeth what is your connection string to pgbouncer? Do you use verify-full or verify-ca ?

    I would use this as a connection string:
    psql "host=test.mycompany.com port=6432 dbname=postgres user=pgbouncer password=pgbouncer123! sslmode=require"

    I think connection string is not relevant, because pgbouncer doesn't even start.

  3. JelteF commented on Feb 6, 2025

    @JelteF
    JelteF
    on Feb 6, 2025
    Member

    I think you also need to configure client_tls_ca_file

  4. Erilbeth commented on Feb 6, 2025

    @Erilbeth
    Erilbeth
    on Feb 6, 2025
    Author

    I think you also need to configure client_tls_ca_file

    In that case i would need a ca file, if i create a self-signed cert i could set client_tls_ca_file.
    But i want to use a lets encrypt cert, so i don't have a client_tls_ca_file.
    Also client_tls_ca_file is optional. lack of client_tls_ca_file should not break the startup of pgbouncer.

  5. JelteF commented on Feb 6, 2025

    @JelteF
    JelteF
    on Feb 6, 2025
    Member

    I think for self signed certs you'd still have a CA, just a self signed CA. It might even work to put the same file there as you put in client_tls_cert_file.

    I think it's optional, only if you don't provide client_tls_cert_file/client_tls_key_file. Docs could definitely be clearer on that though.

  6. Erilbeth commented on Feb 6, 2025

    @Erilbeth
    Erilbeth
    on Feb 6, 2025
    Author

    I think for self signed certs you'd still have a CA, just a self signed CA. It might even work to put the same file there as you put in client_tls_cert_file.

    I think it's optional, only if you don't provide client_tls_cert_file/client_tls_key_file. Docs could definitely be clearer on that though.

    +1
    Since this issue is created 2 months ago, i remember i've downloaded ca file and defined like the way you mentioned. but it didn't work. though i don't remember the error. It would be nice someone try it with lets encrypt cert to see if it's really working. I would like to do it but we already left pgbouncer for this error and i don't have any desire to deal more with pgbouncer.

  7. emelsimsek commented on Feb 7, 2025

    @emelsimsek
    emelsimsek
    on Feb 7, 2025
    Contributor

    @Erilbeth,

    It would be nice someone try it with lets encrypt cert to see if it's really working.

    Did you create the pgbouncer certs using Let's Encrypt?

    auth_type = trust

    in your config implies that the client authentication is not via certs, right?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions