Atmosphere Operator: optional deployment of ingress-nginx

[runlevel-six]

Our deployment of Atmosphere includes a slightly different architecture for Kubernetes, including a custom deployment of ingress-nginx, and one that uses metal-loadbalancer instead of keepalived. Prior to the move of ingress-nginx into the Atmosphere operator this was simple enough to manage, as we just commented out the deployment of keepalived and ingress-nginx in the openstack Ansible playbook.

Per the conversation we had last Friday (September 30), it would be ideal if instead of maintaining a fork of the atmosphere operator with the flows.py code commented out for the deployment of ingress-nginx, if instead this could be managed much like memcached (with a simple config override config.memcached.enabled) then the operator would be a more flexible option for us.

[mnaser]

@runlevel-six can you checkout if that PR does initially what you're looking for? you can feel free to test that branch while we wait for it to pass tests.

[runlevel-six]

@mnaser I checked out the feat/cfg-overrides branch and set the following in roles/atmosphere/defaults/main.yml:

atmosphere_image: quay.io/vexxhost/atmosphere:0.4.1 # x-release-please-version

atmosphere_config:
  memcached:
    secret_key: "{{ openstack_helm_endpoints_memcached_secret_key }}"
    overrides: "{{ openstack_helm_infra_memcached_values | default({}) }}"
  issuer: "{{ atmosphere_issuer_config }}"
  ingress_nginx:
    enabled: false

atmosphere_issuer_config:
  type: self-signed

When the atmosphere operator starts up, it fails almost immediately with the following error:

❯ kubectl -n openstack logs atmosphere-operator-55dd6b995d-6kw48
2022-10-06 12:32.06 [info     ] Starting Atmosphere operator
Traceback (most recent call last):
  File "/app/.venv/bin/atmosphere-operator", line 6, in <module>
    sys.exit(main())
  File "/app/atmosphere/cmd/operator.py", line 12, in main
    cfg = config.Config.from_file()
  File "/app/atmosphere/models/config.py", line 113, in from_file
    return cls.from_toml(data)
  File "/app/atmosphere/models/config.py", line 104, in from_toml
    c = cls(data, validate=validate)
  File "/app/.venv/lib/python3.10/site-packages/schematics/models.py", line 232, in __init__
    data = self._convert(raw_data,
  File "/app/.venv/lib/python3.10/site-packages/schematics/models.py", line 299, in _convert
    return func(self._schema, self, raw_data=raw_data, oo=True, context=context, **kwargs)
  File "/app/.venv/lib/python3.10/site-packages/schematics/validate.py", line 67, in validate
    raise DataError(errors, data)
schematics.exceptions.DataError: {"ingress_nginx": "Rogue field"}

A description of the pod in case it helps show a configuration error (on my end or not):

❯ kubectl -n openstack describe po atmosphere-operator-55dd6b995d-6kw48
Name:             atmosphere-operator-55dd6b995d-6kw48
Namespace:        openstack
Priority:         0
Service Account:  atmosphere
Node:             clt-a-a06-19-2-sr-blade-b/10.101.5.77
Start Time:       Thu, 06 Oct 2022 08:31:40 -0400
Labels:           application=atmosphere
                  component=operator
                  pod-template-hash=55dd6b995d
Annotations:      <none>
Status:           Running
IP:               10.204.2.82
IPs:
  IP:           10.204.2.82
Controlled By:  ReplicaSet/atmosphere-operator-55dd6b995d
Containers:
  operator:
    Container ID:   containerd://fd77249a288e1e2bb4d6e49f93d2edcca980c0bf10785e8f8b2a83dd382d53a0
    Image:          quay.io/vexxhost/atmosphere:0.4.1
    Image ID:       quay.io/vexxhost/atmosphere@sha256:0021ba019bd6ad4c35c5ebfff39b2a180d4692dd871a0bf37a6294f966b3b7ee
    Port:           <none>
    Host Port:      <none>
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Thu, 06 Oct 2022 08:34:37 -0400
      Finished:     Thu, 06 Oct 2022 08:34:39 -0400
    Ready:          False
    Restart Count:  5
    Environment:    <none>
    Mounts:
      /etc/atmosphere from config (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-4r5sw (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  config:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  atmosphere-config
    Optional:    false
  kube-api-access-4r5sw:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              openstack-control-plane=enabled
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason           Age                   From               Message
  ----     ------           ----                  ----               -------
  Normal   Scheduled        3m23s                 default-scheduler  Successfully assigned openstack/atmosphere-operator-55dd6b995d-6kw48 to atmosphere-1
  Normal   Pulling          3m22s                 kubelet            Pulling image "quay.io/vexxhost/atmosphere:0.4.1"
  Normal   Pulled           3m19s                 kubelet            Successfully pulled image "quay.io/vexxhost/atmosphere:0.4.1" in 3.19819974s
  Normal   Created          108s (x5 over 3m19s)  kubelet            Created container operator
  Normal   Started          108s (x5 over 3m19s)  kubelet            Started container operator
  Normal   Pulled           108s (x4 over 3m15s)  kubelet            Container image "quay.io/vexxhost/atmosphere:0.4.1" already present on machine
  Warning  BackOff          77s (x9 over 3m13s)   kubelet            Back-off restarting failed container

This could be that I just misread a change necessary to make this work, so I will go back and more carefully review the branch changes.

[mnaser]

I think the issue here is that this image is not the up to date/fixed one:

atmosphere_image: quay.io/vexxhost/atmosphere:0.4.1 # x-release-please-version

You'll have to build the image and upload it manually somewhere, or use tty.sh similar to how we use it in Molecule and point to it. However, I'll cut the release shortly anyhow that includes this.

[mnaser]

@runlevel-six just cut 0.5.0 off, so you should be able to point the collection to that and get the fixes in :)

FYI, you could totally use the overrides and have Atmosphere manage the ingress component for you, just with your own overrides, if that works for you.

[runlevel-six]

@mnaser this worked well for us. Thank you!

And you are right - this is something we are discussing. The customization on our end is mainly security focused, which would be easily adopted into the overrides available in Atmosphere. The only other change we make is running ingress-nginx in its own namespace which is not a Helm variable, if I remember right.

[mnaser]

31e528b

once all this pass tests and we cut a release based on that, I think you should be able to fully override it and let atmosphere manage it.