_init_projects_container_image image not customizable

[samweisgamdschie]

Please confirm the following

• I agree to follow this project's code of conduct.
• I have checked the current issues for duplicates.
• I understand that the AWX Operator is open source software provided for free and that I might not receive a timely response.

Bug Summary

We definitely need to set custom image repos for every image because of our company proxy. The new variable _init_projects_container_image has no corresponding variable in the AWX.spec nor it is configurable during installation.

error validating data: ValidationError(AWX.spec): unknown field "_init_projects_container_image" in com.ansible.awx.v1beta1.AWX.spec;

AWX Operator version

1.1.2

AWX version

latest

Kubernetes platform

kubernetes

Kubernetes/Platform version

Server Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.15", GitCommit:"8f1e5bf0b9729a899b8df86249b56e2c74aebc55", GitTreeState:"clean", BuildDate:"2022-01-19T17:23:01Z", GoVersion:"go1.15.15", Compiler:"gc", Platform:"linux/amd64"}

Modifications

no

Steps to reproduce

kubectl delete ns myteam-tower
kubectl apply -f tower_pvc.yaml -f tower_2.yaml
namespace/myteam-tower unchanged
customresourcedefinition.apiextensions.k8s.io/awxbackups.awx.ansible.com unchanged
customresourcedefinition.apiextensions.k8s.io/awxrestores.awx.ansible.com unchanged
customresourcedefinition.apiextensions.k8s.io/awxs.awx.ansible.com unchanged
serviceaccount/awx-operator-controller-manager created
role.rbac.authorization.k8s.io/awx-operator-awx-manager-role created
role.rbac.authorization.k8s.io/awx-operator-leader-election-role created
clusterrole.rbac.authorization.k8s.io/awx-operator-metrics-reader unchanged
clusterrole.rbac.authorization.k8s.io/awx-operator-proxy-role unchanged
rolebinding.rbac.authorization.k8s.io/awx-operator-awx-manager-rolebinding created
rolebinding.rbac.authorization.k8s.io/awx-operator-leader-election-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/awx-operator-proxy-rolebinding unchanged
configmap/awx-operator-awx-manager-config created
service/awx-operator-controller-manager-metrics-service created
deployment.apps/awx-operator-controller-manager created
error: error validating "tower_2.yaml": error validating data: ValidationError(AWX.spec): unknown field "_init_projects_container_image" in com.ansible.awx.v1beta1.AWX.spec; if you choose to ignore these errors, turn validation off with --validate=false

Expected results

No errors during installation.

Actual results

see above.

Additional information

No response

Operator Logs

No response

[renanguilhermef]

same issue, i cannot deploy new version because I need to use local registry

  Name:         awx-7c9958dffb-vpfsm
  Namespace:    awxm
  Priority:     0
  Node:         2c4a4d03-4bbb-4ca4-9076-30370cf9da0a/172.24.22.6
  Start Time:   Wed, 21 Dec 2022 12:31:12 -0300
  Labels:       app.kubernetes.io/component=awx
                app.kubernetes.io/managed-by=awx-operator
                app.kubernetes.io/name=awx
                app.kubernetes.io/operator-version=1.1.3
                app.kubernetes.io/part-of=awx
                app.kubernetes.io/version=21.4.0_vuln_fix
                pod-template-hash=7c9958dffb
  Annotations:  <none>
  Status:       Pending
  IP:           198.18.10.227
  IPs:
    IP:           198.18.10.227
  Controlled By:  ReplicaSet/awx-7c9958dffb
  Init Containers:
    init:
      Container ID:  docker://4e515b3601079f20c9c1618640e04464efc468bdc7ec6a7f0ed14fd73f58f49f
      Image:         <MY_REGYSTRY>:latest
      Image ID:      docker-pullable://<MY_REGYSTRY>@sha256:1b8b3955365fef83074c5d7b69a0b61063f95a3a353f72667d6273a92ff7ad37
      Port:          <none>
      Host Port:     <none>
      Command:
        /bin/sh
        -c
        hostname=$MY_POD_NAME
        receptor --cert-makereq bits=2048 commonname=$hostname dnsname=$hostname nodeid=$hostname outreq=/etc/receptor/tls/receptor.req outkey=/etc/receptor/tls/receptor.key
        receptor --cert-signreq req=/etc/receptor/tls/receptor.req cacert=/etc/receptor/tls/ca/receptor-ca.crt cakey=/etc/receptor/tls/ca/receptor-ca.key outcert=/etc/receptor/tls/receptor.crt verify=yes
        
      State:          Waiting
        Reason:       CrashLoopBackOff
      Last State:     Terminated
        Reason:       Error
        Exit Code:    127
        Started:      Wed, 21 Dec 2022 12:37:23 -0300
        Finished:     Wed, 21 Dec 2022 12:37:23 -0300
      Ready:          False
      Restart Count:  6
      Requests:
        cpu:     100m
        memory:  128Mi
      Environment:
        MY_POD_NAME:  awx-7c9958dffb-vpfsm (v1:metadata.name)
      Mounts:
        /etc/receptor/tls/ from awx-receptor-tls (rw)
        /etc/receptor/tls/ca/receptor-ca.crt from awx-receptor-ca (ro,path="tls.crt")
        /etc/receptor/tls/ca/receptor-ca.key from awx-receptor-ca (ro,path="tls.key")
        /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-5x5hb (ro)
    init-projects:
      Container ID:  
      Image:         quay.io/centos/centos:stream9
      Image ID:      
      Port:          <none>
      Host Port:     <none>
      Command:
        /bin/sh
        -c
        chmod 775 /var/lib/awx/projects
        chgrp 1000 /var/lib/awx/projects
        
      State:          Waiting
        Reason:       PodInitializing
      Ready:          False
      Restart Count:  0
      Environment:
        MY_POD_NAME:  awx-7c9958dffb-vpfsm (v1:metadata.name)
      Mounts:
        /var/lib/awx/projects from awx-projects (rw)
        /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-5x5hb (ro)
  Containers:
    redis:
      Container ID:  
      Image:         <MY_REGISTRY>/platform_eng_public/redis:latest
      Image ID:      
      Port:          <none>
      Host Port:     <none>
      Args:
        redis-server
        /etc/redis.conf
      State:          Waiting
        Reason:       PodInitializing
      Ready:          False
      Restart Count:  0
      Requests:
        cpu:        50m
        memory:     64Mi
      Environment:  <none>
      Mounts:
        /data from awx-redis-data (rw)
        /etc/redis.conf from awx-redis-config (ro,path="redis.conf")
        /var/run/redis from awx-redis-socket (rw)
        /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-5x5hb (ro)
    awx-web:
      Container ID:  
      Image:         <MY_REGYSTRY>awx:21.4.0_vuln_fix
      Image ID:      
      Port:          8052/TCP
      Host Port:     0/TCP
      Args:
        /usr/bin/launch_awx.sh
      State:          Waiting
        Reason:       PodInitializing
      Ready:          False
      Restart Count:  0
      Requests:
        cpu:     100m
        memory:  128Mi
      Environment:
        MY_POD_NAMESPACE:  awxm (v1:metadata.namespace)
        UWSGI_MOUNT_PATH:  /
      Mounts:
        /etc/nginx/nginx.conf from awx-nginx-conf (ro,path="nginx.conf")
        /etc/receptor/signing/work-public-key.pem from awx-receptor-work-signing (ro,path="work-public-key.pem")
        /etc/receptor/tls/ca/receptor-ca.crt from awx-receptor-ca (ro,path="tls.crt")
        /etc/receptor/tls/ca/receptor-ca.key from awx-receptor-ca (ro,path="tls.key")
        /etc/tower/SECRET_KEY from awxsmt-secret-key-config (ro,path="SECRET_KEY")
        /etc/tower/conf.d/credentials.py from awx-application-credentials (ro,path="credentials.py")
        /etc/tower/conf.d/execution_environments.py from awx-application-credentials (ro,path="execution_environments.py")
        /etc/tower/conf.d/ldap.py from awx-application-credentials (ro,path="ldap.py")
        /etc/tower/settings.py from awx-settings (ro,path="settings.py")
        /var/lib/awx/projects from awx-projects (rw)
        /var/lib/awx/rsyslog from rsyslog-dir (rw)
        /var/run/awx-rsyslog from rsyslog-socket (rw)
        /var/run/redis from awx-redis-socket (rw)
        /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-5x5hb (ro)
        /var/run/supervisor from supervisor-socket (rw)
    awx-task:
      Container ID:  
      Image:         <MY_REGYSTRY>awx:21.4.0_vuln_fix
      Image ID:      
      Port:          <none>
      Host Port:     <none>
      Args:
        /usr/bin/launch_awx_task.sh
      State:          Waiting
        Reason:       PodInitializing
      Ready:          False
      Restart Count:  0
      Requests:
        cpu:     100m
        memory:  128Mi
      Environment:
        SUPERVISOR_WEB_CONFIG_PATH:  /etc/supervisord.conf
        AWX_SKIP_MIGRATIONS:         1
        MY_POD_UID:                   (v1:metadata.uid)
        MY_POD_IP:                    (v1:status.podIP)
        MY_POD_NAMESPACE:            awxm (v1:metadata.namespace)
      Mounts:
        /etc/receptor/ from awx-receptor-config (rw)
        /etc/receptor/signing/work-private-key.pem from awx-receptor-work-signing (ro,path="work-private-key.pem")
        /etc/tower/SECRET_KEY from awxsmt-secret-key-config (ro,path="SECRET_KEY")
        /etc/tower/conf.d/credentials.py from awx-application-credentials (ro,path="credentials.py")
        /etc/tower/conf.d/execution_environments.py from awx-application-credentials (ro,path="execution_environments.py")
        /etc/tower/conf.d/ldap.py from awx-application-credentials (ro,path="ldap.py")
        /etc/tower/settings.py from awx-settings (ro,path="settings.py")
        /var/lib/awx/projects from awx-projects (rw)
        /var/lib/awx/rsyslog from rsyslog-dir (rw)
        /var/run/awx-rsyslog from rsyslog-socket (rw)
        /var/run/receptor from receptor-socket (rw)
        /var/run/redis from awx-redis-socket (rw)
        /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-5x5hb (ro)
        /var/run/supervisor from supervisor-socket (rw)
    awx-ee:
      Container ID:  
      Image:         <MY_REGYSTRY>smt-awx-ee:latest
      Image ID:      
      Port:          <none>
      Host Port:     <none>
      Args:
        /bin/sh
        -c
        if [ ! -f /etc/receptor/receptor.conf ]; then
          cp /etc/receptor/receptor-default.conf /etc/receptor/receptor.conf
          sed -i "s/HOSTNAME/$HOSTNAME/g" /etc/receptor/receptor.conf
        fi
        exec receptor --config /etc/receptor/receptor.conf
        
      State:          Waiting
        Reason:       PodInitializing
      Ready:          False
      Restart Count:  0
      Requests:
        cpu:        100m
        memory:     64Mi
      Environment:  <none>
      Mounts:
        /etc/receptor/ from awx-receptor-config (rw)
        /etc/receptor/receptor-default.conf from awx-default-receptor-config (rw,path="receptor.conf")
        /etc/receptor/signing/work-private-key.pem from awx-receptor-work-signing (ro,path="work-private-key.pem")
        /etc/receptor/tls/ from awx-receptor-tls (rw)
        /etc/receptor/tls/ca/receptor-ca.crt from awx-receptor-ca (ro,path="tls.crt")
        /var/lib/awx/projects from awx-projects (rw)
        /var/run/receptor from receptor-socket (rw)
        /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-5x5hb (ro)
  Conditions:
    Type              Status
    Initialized       False 
    Ready             False 
    ContainersReady   False 
    PodScheduled      True 
  Volumes:
    awx-application-credentials:
      Type:        Secret (a volume populated by a Secret)
      SecretName:  awx-app-credentials
      Optional:    false
    awx-receptor-tls:
      Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
      Medium:     
      SizeLimit:  <unset>
    awx-receptor-ca:
      Type:        Secret (a volume populated by a Secret)
      SecretName:  awx-receptor-ca
      Optional:    false
    awx-receptor-work-signing:
      Type:        Secret (a volume populated by a Secret)
      SecretName:  awx-receptor-work-signing
      Optional:    false
    awxsmt-secret-key-config:
      Type:        Secret (a volume populated by a Secret)
      SecretName:  awxsmt-secret-key-config
      Optional:    false
    awx-settings:
      Type:      ConfigMap (a volume populated by a ConfigMap)
      Name:      awx-awx-configmap
      Optional:  false
    awx-nginx-conf:
      Type:      ConfigMap (a volume populated by a ConfigMap)
      Name:      awx-awx-configmap
      Optional:  false
    awx-redis-config:
      Type:      ConfigMap (a volume populated by a ConfigMap)
      Name:      awx-awx-configmap
      Optional:  false
    awx-redis-socket:
      Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
      Medium:     
      SizeLimit:  <unset>
    awx-redis-data:
      Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
      Medium:     
      SizeLimit:  <unset>
    supervisor-socket:
      Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
      Medium:     
      SizeLimit:  <unset>
    rsyslog-socket:
      Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
      Medium:     
      SizeLimit:  <unset>
    receptor-socket:
      Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
      Medium:     
      SizeLimit:  <unset>
    rsyslog-dir:
      Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
      Medium:     
      SizeLimit:  <unset>
    awx-receptor-config:
      Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
      Medium:     
      SizeLimit:  <unset>
    awx-default-receptor-config:
      Type:      ConfigMap (a volume populated by a ConfigMap)
      Name:      awx-awx-configmap
      Optional:  false
    awx-projects:
      Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
      ClaimName:  awx-projects-claim
      ReadOnly:   false
    kube-api-access-5x5hb:
      Type:                    Projected (a volume that contains injected data from multiple sources)
      TokenExpirationSeconds:  3607
      ConfigMapName:           kube-root-ca.crt
      ConfigMapOptional:       <nil>
      DownwardAPI:             true
  QoS Class:                   Burstable
  Node-Selectors:              <none>
  Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                               node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
  Events:
    Type     Reason                  Age                   From                     Message
    ----     ------                  ----                  ----                     -------
    Normal   Scheduled               10m                   default-scheduler        Successfully assigned awxm/awx-7c9958dffb-vpfsm to 2c4a4d03-4bbb-4ca4-9076-30370cf9da0a
    Normal   SuccessfulAttachVolume  10m                   attachdetach-controller  AttachVolume.Attach succeeded for volume "pvc-62fa8940-4aaa-46bf-9c2c-b10f623c94c2"
    Normal   Pulled                  8m27s (x5 over 10m)   kubelet                  Container image "<MY_REGYSTRY>:latest" already present on machine
    Normal   Created                 8m26s (x5 over 10m)   kubelet                  Created container init
    Normal   Started                 8m26s (x5 over 10m)   kubelet                  Started container init
    Warning  BackOff                 4m58s (x26 over 10m)  kubelet                  Back-off restarting failed container

[samweisgamdschie]

As described in the linked issue ansible/aws#13378 it should be solved here in awx-operator.

Are there any efforts in this case? A fact is, that many companies does not allow servers to communicate into the internet (as mine) and works with local registries. Therefore every used image URL in this operator needs to be configured at every time. This means that every version which has a problem with that is not usable for these companies!

As we try installing and using awx-operator for the last half year in our company which uses:

• SSL Offloading proxy with a self signed certificate
• local registry (JFrog Artifactory) which mainly proxies images from official registries into the intranet (which means the images have the same id/checksum but are reached via a different URL)
• Proxy and Router rules to block every internet access except the ones allowed (but then with AD username/password)

I will report our efforts of the next 2-3 weeks trying to install awx-operator and setup a production environment here. But it seems you are often not sensitive to the above facts. Please correct me, if I'm wrong.

[mbutton77]

Also facing the issue.

[himadriganguly]

Facing the same issue. Is there any update on that issue. I also want to install awx-operator on K8s cluster without internet access.

I am also getting the following error in the log of init container

/bin/sh: line 1: receptor: command not found
/bin/sh: line 2: receptor: command not found

[samweisgamdschie]

Any news? It seems to be an operator issue and we cant use any of the new versions of awx-operator until this is fixed (for every image). This is a major issue imho!

[mbutton77]

On my side, I have tried to implement the same mechanism as the one used for the var init_project_container_image.
I have a feeling that I am not far but after rebuilding the image and having deployed it on my private repo, I am now struggling with the kustomization to add my own ImagePullSecret.
I will keep you posted and create a PR if my test is successful.

[samweisgamdschie]

Maybe the author @FlorianLaunay is able to update his extension from the pull request #1078 ? @FlorianLaunay please! Because of such an issue we may be forced to switch to the "oracle linux automation manager". Or maybe @shanemcd who approved and merged the commit?

[mbutton77]

@samweisgamdschie I clearly understand your frustration, but I don't see how threatening the guys working on this project to switch towards OLAM is going to help. (+ I really think they could not care less)

Remember : this is an open-source project and if you are not happy with this bug fixing process, try to fix it yourself.
It might be quicker and on top of that, it will help the whole community.

By the way, I pushed a PR : #1248

(If you are in a rush, you can always use the code to create your custom awx-operator image)

[samweisgamdschie]

@mbutton77 many thanks for this commit, it looks pretty much the same we thought it could work. The reason why we had not created a pull request is because we cannot test it if we try to create our own containers because setting up the pipeline for building it has also need to happen behind our annoying proxy (which in case is a chicken-egg-problem we say here in austria ;) ).

And I think you missunderstood me, I am the one who is threatened about OLAM ... Of course this is an open source project and I love open source since many years now. It's also just the only way to install awx now and if you have our experience with an omnipresent proxy, I guess you would understand me better.

[mbutton77]

You are very welcome @samweisgamdschie. As you said, I was also blocked by this.

Ok, I guess communication problems were likely to happen between an Austrian and a French ! :)
What matters is that we are now one step closer to a durable fix. We will just have to wait for a reviewer to take a look at that PR (and hopefully merge it !)

[FlorianLaunay]

Thanks @mbutton77 for this fix. I hadn't thought to make this parameter customizable initially. My bad.

BTW I'm glad to see other French people to contribute to this project 🤩

[FlorianLaunay]

@samweisgamdschie thumbs up #1248 to make it more visible (already done for me 😉 )

[mbutton77]

I also put a thumb up, even if it's like laughing to my own joke, but well ... :)

@himadriganguly and @renanguilhermef, as you were also concerned by this issue, could you please put a thumb up as well on the PR ? Following @FlorianLaunay's hint should prioritize the review process and then the possible merge.
Thanks guys !