Add bin/show-docker-images.py

[danielhoherd]

Description

Add a python script that parses the output of helm template and finds all the container specs and prints the images that are used in them. This avoids printing containers that are inside of nested yaml like the houston configmap.

$ bin/show-docker-images.py --help
usage: show-docker-images.py [-h] [--private-registry] [-v]

options:
  -h, --help          show this help message and exit
  --private-registry
  -v, --verbose

Related Issues

https://github.com/astronomer/issues/issues/6113
https://github.com/astronomer/issues/issues/4862 - fixing this issue as well with this PR

Testing

I've tested this extensively while developing, which is enough. This is not a product feature and is not customer facing so no QA is needed.

Merging

This should be merged everywhere.

[danielhoherd]

@pgvishnuram this is currently showing 4 objects that use a non-private-registry auth-sidecar:

$ bin/show-docker-images.py --private-registry
Deployment grafana uses quay.io
Deployment kibana uses quay.io
StatefulSet alertmanager uses quay.io
StatefulSet prometheus uses quay.io
https://example.com/the-private-registry/ap-alertmanager            example.com/the-private-registry/ap-alertmanager:0.26.0
https://example.com/the-private-registry/ap-astro-ui                example.com/the-private-registry/ap-astro-ui:0.34.0
https://example.com/the-private-registry/ap-awsesproxy              example.com/the-private-registry/ap-awsesproxy:1.5.0-6
https://example.com/the-private-registry/ap-base                    example.com/the-private-registry/ap-base:3.18.5
https://example.com/the-private-registry/ap-blackbox-exporter       example.com/the-private-registry/ap-blackbox-exporter:0.24.0-5
https://example.com/the-private-registry/ap-cli-install             example.com/the-private-registry/ap-cli-install:0.26.21
https://example.com/the-private-registry/ap-commander               example.com/the-private-registry/ap-commander:0.34.0
https://example.com/the-private-registry/ap-configmap-reloader      example.com/the-private-registry/ap-configmap-reloader:0.12.0
https://example.com/the-private-registry/ap-curator                 example.com/the-private-registry/ap-curator:8.0.8-6
https://example.com/the-private-registry/ap-db-bootstrapper         example.com/the-private-registry/ap-db-bootstrapper:0.31.9
https://example.com/the-private-registry/ap-default-backend         example.com/the-private-registry/ap-default-backend:0.28.22
https://example.com/the-private-registry/ap-elasticsearch-exporter  example.com/the-private-registry/ap-elasticsearch-exporter:1.7.0
https://example.com/the-private-registry/ap-elasticsearch           example.com/the-private-registry/ap-elasticsearch:8.11.4
https://example.com/the-private-registry/ap-fluentd                 example.com/the-private-registry/ap-fluentd:1.16.3
https://example.com/the-private-registry/ap-grafana                 example.com/the-private-registry/ap-grafana:10.2.3
https://example.com/the-private-registry/ap-houston-api             example.com/the-private-registry/ap-houston-api:0.34.1
https://example.com/the-private-registry/ap-init                    example.com/the-private-registry/ap-init:3.18.5
https://example.com/the-private-registry/ap-kibana                  example.com/the-private-registry/ap-kibana:8.11.4
https://example.com/the-private-registry/ap-kube-state              example.com/the-private-registry/ap-kube-state:2.10.1
https://example.com/the-private-registry/ap-nats-exporter           example.com/the-private-registry/ap-nats-exporter:0.14.0
https://example.com/the-private-registry/ap-nats-server             example.com/the-private-registry/ap-nats-server:2.10.9-1
https://example.com/the-private-registry/ap-nats-streaming          example.com/the-private-registry/ap-nats-streaming:0.25.6
https://example.com/the-private-registry/ap-nginx-es                example.com/the-private-registry/ap-nginx-es:1.25.3
https://example.com/the-private-registry/ap-nginx                   example.com/the-private-registry/ap-nginx:1.9.5
https://example.com/the-private-registry/ap-node-exporter           example.com/the-private-registry/ap-node-exporter:1.7.0
https://example.com/the-private-registry/ap-openresty               example.com/the-private-registry/ap-openresty:1.25.3
https://example.com/the-private-registry/ap-pgbouncer-krb           example.com/the-private-registry/ap-pgbouncer-krb:1.17.0-9
https://example.com/the-private-registry/ap-postgres-exporter       example.com/the-private-registry/ap-postgres-exporter:0.15.0-3
https://example.com/the-private-registry/ap-postgresql              example.com/the-private-registry/ap-postgresql:15.4.0-1
https://example.com/the-private-registry/ap-prometheus              example.com/the-private-registry/ap-prometheus:2.45.2
https://example.com/the-private-registry/ap-registry                example.com/the-private-registry/ap-registry:3.18.5-1
https://quay.io/astronomer/ap-auth-sidecar                          quay.io/astronomer/ap-auth-sidecar:1.25.2-3

If you run it with -v it'll show the template.spec in JSON. Do you want to check if that's a valid error that we need to fix in the chart or something that can be skipped in this script?

[pgvishnuram]

this can fixed by adding a template in templates/ dir and calling in all places where auth-sidecar is used
i can add the changes in the same PR

[pgvishnuram]

[pgvishnuram]

this was generated by our old scrip

this output was generated with old make file - new script seems to skip vector images

[danielhoherd]

@pgvishnuram i added a --with-houston argument that includes those images. Do we want to make that the default behavior for make show-docker-images and make-show-docker-images-with-private-registry?

Help looks like:

$ bin/show-docker-images.py --help
usage: show-docker-images.py [-h] [--private-registry] [-v] [--with-houston]

options:
  -h, --help          show this help message and exit
  --private-registry  show images when using private registry
  -v, --verbose       be verbose
  --with-houston      include images from houston configmap

Here we can see vector being included from the houston configmap:

$ bin/show-docker-images.py --with-houston | grep vector
https://quay.io/astronomer/ap-vector                  quay.io/astronomer/ap-vector:0.32.2-1