Unable to add pgbouncer using Helm chart

[jayon-niravel]

Official Helm Chart version

1.10.0 (latest released)

Apache Airflow version

2.6.0

Kubernetes Version

1.23

Helm Chart configuration

VALUES.YAML

# Otherwise pass connection values in metadataConnection: 
user: postgres 
pass: postgres 
protocol: postgresql 
host: airflow-postgresql 
port: 5432 
db: airflow 
sslmode: disable 
# resultBackendConnection defaults to the same database as metadataConnection 
# resultBackendConnection: ~ 
# or, you can use a different database 
# resultBackendConnection: 
#   user: postgres 
#   pass: postgres 
#   protocol: postgresql 
#   host: ~ 
#   port: 5432 
#   db: postgres 
#   sslmode: disable 
# Note: brokerUrl can only be set during install, not upgrade brokerUrl: ~

  pgbouncer:
  # Enable PgBouncer
  enabled: true
  # Number of PgBouncer replicas to run in Deployment
  replicas: 1
  # Max number of old replicasets to retain
  revisionHistoryLimit: ~
  # Command to use for PgBouncer(templated).
  command: ["pgbouncer", "-d", "/etc/pgbouncer/pgbouncer.ini"]
  # Args to use for PgBouncer(templated).
  args: ~
  auth_type: md5
  auth_file: /etc/pgbouncer/users.txt

  # annotations to be added to the PgBouncer deployment
  annotations: {}

  podAnnotations: {}

  # Create ServiceAccount
  serviceAccount:
    # Specifies whether a ServiceAccount should be created
    create: true
    # The name of the ServiceAccount to use.
    # If not set and create is true, a name is generated using the release name
    name: ~

    # Annotations to add to worker kubernetes service account.
    annotations: {}

  # Additional network policies as needed
  extraNetworkPolicies: []

  # Pool sizes
  metadataPoolSize: 10
  resultBackendPoolSize: 5

  # Maximum clients that can connect to PgBouncer (higher = more file descriptors)
  maxClientConn: 100

  # supply the name of existing secret with pgbouncer.ini and users.txt defined
  # you can load them to a k8s secret like the one below
  #  apiVersion: v1
  #  kind: Secret
  #  metadata:
  #    name: pgbouncer-config-secret
  #  data:
  #     pgbouncer.ini: <base64_encoded pgbouncer.ini file content>
  #     users.txt: <base64_encoded users.txt file content>
  #  type: Opaque
  #
  #  configSecretName: pgbouncer-config-secret
  #
  configSecretName: airflow-pgbouncer-config

  # PgBouncer pod disruption budget
  podDisruptionBudget:
    enabled: false

    # PDB configuration
    config:
      maxUnavailable: 1

  # Limit the resources to PgBouncer.
  # When you specify the resource request the k8s scheduler uses this information to decide which node to
  # place the Pod on. When you specify a resource limit for a Container, the kubelet enforces those limits so
  # that the running container is not allowed to use more of that resource than the limit you set.
  # See: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/
  # Example:
  #
  # resource:
  #   limits:
  #     cpu: 100m
  #     memory: 128Mi
  #   requests:
  #     cpu: 100m
  #     memory: 128Mi
  resources: {}

  service:
    extraAnnotations: {}

  # https://www.pgbouncer.org/config.html
  verbose: 0
  logDisconnections: 0
  logConnections: 0

  sslmode: "prefer"
  ciphers: "normal"

  ssl:
    ca: ~
    cert: ~
    key: ~

  # Add extra PgBouncer ini configuration in the databases section:
  # https://www.pgbouncer.org/config.html#section-databases
  extraIniMetadata: ~
  extraIniResultBackend: ~
  # Add extra general PgBouncer ini configuration: https://www.pgbouncer.org/config.html
  extraIni: ~

  # Mount additional volumes into pgbouncer.
  extraVolumes: []
  extraVolumeMounts: []

  # Select certain nodes for PgBouncer pods.
  nodeSelector: {}
  affinity: {}
  tolerations: []
  topologySpreadConstraints: []

  priorityClassName: ~

  uid: 1001180000

  metricsExporterSidecar:
    resources: {}
    #  limits:
    #   cpu: 100m
    #   memory: 128Mi
    #  requests:
    #   cpu: 100m
    #   memory: 128Mi
    sslmode: "disable"

    livenessProbe:
      initialDelaySeconds: 10
      periodSeconds: 10
      timeoutSeconds: 1

    readinessProbe:
      initialDelaySeconds: 10
      periodSeconds: 10
      timeoutSeconds: 1
  
  

Docker Image customizations

pgbouncer.ini

`[databases]
airflow-metadata = host=airflow-postgresql dbname=airflow port=5432 pool_size=10 auth_user=postgres
airflow-result-backend = host=airflow-postgresql dbname=airflow port=5432 pool_size=5 auth_user=postgres

[pgbouncer]
logfile = /var/log/pgbouncer/pgbouncer.log
pidfile = /var/run/pgbouncer/pgbouncer.pid
pool_mode = transaction
listen_port = 6543
listen_addr = *
auth_type = md5
auth_file = /etc/pgbouncer/users.txt
ignore_startup_parameters = extra_float_digits
max_client_conn = 100
admin_users = postgres
verbose = 0
log_disconnections = 0
log_connections = 0

server_tls_sslmode = prefer
server_tls_ciphers = normal`



users.txt
`users.txt
`

What happened

Pgbouncer container does not start and there are no error logs.

If I use below command 
command: ["pgbouncer", "-u", "postgres", "/etc/pgbouncer/pgbouncer.ini"]
then I would get "FATAL Error: Unable switch to user postgres"

What you think should happen instead

Pgbouncer container should have start running but it keeps crashing. Am I missing any configuration?

How to reproduce

Please try using above mentioned YAML and values

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[potiuk]

You should take a look at kubernetes logs, why your pgbouncer does not start - not the airflow logs. I assume that you already run helm install --dry-run and inspected that the resources for pgbouncer are present, then you also see pgbouncer pod that attempts to start and crash, if you see that after your custom modifications, you need to use whatever you usually use to check erros from crashing pods in your k8s installation (I for une use k9s tool - very useful) to diagnose your configuration . We are using standard pgbouncer image so if you customized it an it crashes, you should see the error logs and see why.

This kind of issue is more of troubleshooting question - you should open a discussion instead (issue template is quite clear about it) - please use it next time for similar things. I am converting it to a discussion now.

[jayon-niravel]

@potiuk : I get below logs

Defaulted container "pgbouncer" out of: pgbouncer, metrics-exporter

airflow-pgbouncer-855c74465b-zt2zz 0/2 CrashLoopBackOff 1262 (97s ago)

[jayon-niravel]

and also I did not customize the pgbouncer, if you see my above configuration

[potiuk]

airflow-pgbouncer-855c74465b-zt2zz 0/2 CrashLoopBackOff 1262 (97s ago)

Yes you need to see what cotainer logs tell you when crashing. Without it you will not be able to figure out what's wrong. It's impossible to help you when you tell us that things are crashing and do not provide information why. There are ways how you can check it (you can google it) depending on the tools/K8s you use (I recommend k9s) and if you want to manage application on k8s you need to learn how to do it for your platform because it will be happening more often

and also I did not customize the pgbouncer, if you see my above configuration

Could be. I have no time for all issues that someone raises, to get the exact combination of helm/airfow/configurateion and compare it. Generally the rule is that when you reach for troubleshooting help, it's a good idea you make it as easy as possible for someone - who (in their free time, without having to do it - doing that because they want to help) - tries to help you. So if you have not made any modifications, it's worth mentioning it when you raise issue/open discussion.

But there are other reasons why things might not work - your networking configuration, your kubernetes capabilities, your resources, etc. etc. There is no guarantee that if you "don't change anything" - things will be working in your particular case. There are various environmental circumstances. And the way how to find out what's wrong in your case is to see (and only you can do it) why you have hte crash loopback - the crashing container should produce logs (again you nee to learn how to get it in the way that your environment supports) that should tell you the reason. Guessing it makes no sense.