Helm chart images: conda removed -> pip only, usage disclaimer added, minimized Dockerfile complexity

[consideRatio]

Discussion for this change was made in #524 (comment).

1. dask/dask-gateway-server, an image used by the Helm chart's controller and api pod, should not be something end users derive their own image from. If they do for some unknown reason though, they shouldn't expect this project to respect backward compatibility or changelogs etc.
2. dask/dask-gateway, an image used by the Helm chart as a sample image for a clusters scheduler/worker pods, should be treated as a sample image that people shouldn't derive their own scheduler/worker images from. Instead, they should build their scheduler/worker images in any way according to documentation even though that documentation may require an update.
3. The example Dockerfile, just an example and not an image we build, is probably a bit too complicated to maintain and keep updated. As outdated examples are sometimes problematic I think the best course of action is to rely on the documentation on how to create your own image, and suggest asking users to look at the dask/dask-gateway image for practical inspiration on how you may do it.

Change summary

• Documents that the dask/dask-gateway image is meant for testing purposes and possibly inspiration of writing your own image. It also removes the example Dockerfile that had grown a bit outdated and tweaks documentation to reference the sample Dockerfile.
• Reduces the complexity of the Helm chart's Dockerfiles very significantly, cutting a CI job step down to around 68 and 80 seconds from around 284 and 316 seconds: a reduction of ~75% or ~3m46s. As the k8s tests were the by far the slowest in our test suite, this reduces our CI test duration with that amount.
• conda and mamba is no longer installed as part of these images, something I'm quite strongly opinionated about at the moment. If the Helm chart was a separate project that could build these images after dask-gateway and dask-gateway-server has been released and packaged to conda-forge, I'd think differently as the complexity comes mainly from that. I think this change is so problematic as there is now a more explicit expectation.
  • The dask/dask-gateway-server running in the api pod and controller pod isn't something users have overridden historically I think
  • The dask/dask-gateway image for scheduler/worker pods is documented as a sample image used for testing with an expectation that users define their own image.

Closes #524
Closes #489 - by declaring that the user is to make their own images for scheduler/workers if they have specific needs.
Closes #432 - by going for plain pip instead.

[martindurant]

Minor thoughts only. I wonder if you can compare the resultant pip-based image size with conda - is it a significant savings?

One reason for including conda (/mamba) is that users might expect it to be something they can call from within workers. Or maybe not.

[consideRatio]

Thanks a lot for your review @martindurant!!!!

I'm headed away so I can't go through it fully now so I'll return to it tomorrow/later. Cheers!

[jcrist]

Just a few small nits, otherwise this looks good to me. Thanks Erik!

[consideRatio]

@martindurant @jcrist thanks for your review! I've responded to the review comments! I know that this and #538 will have merge conflicts so I'll need to do some work as soon as one is merged and I'm quite anxious to keep moving.

Btw, I recommend using the "files changed" section to see the discussion as GitHub doesn't seem to make various comments show up in a sensible manner otherwise.

I added 7eca0c6 to include OCI standardized labels in all the Dockerfiles we maintain. source label helps GitHub setup permissions for this image and url is probably helpful to have alongside source that has to point to the root of the github repo based on GitHub docs.

[consideRatio]

Minor thoughts only. I wonder if you can compare the resultant pip-based image size with conda - is it a significant savings?

No savings in size on the image that I think matters to compare, dask-gateway-server. The build time of both images are significantly reduced though.

dg                                            latest               d0b41e6c4250   2 minutes ago        548MB
dg-new                                        latest               a0e4c7c492a8   28 seconds ago       163MB

dgs                                           latest               636d4a82d66b   About a minute ago   189MB
dgs-new                                       latest               c5a1bd202a43   4 seconds ago        193MB

One reason for including conda (/mamba) is that users might expect it to be something they can call from within workers. Or maybe not.

Yes maybe, but I see it as a feature of this PR to make a clear break of the potential expectations someone may have on this image - it is to be used for testing purposes.

[martindurant]

Big win on one, and slightly worse on the other. Interesting! The big win is a good argument to go ahead.

[consideRatio]

Hey @martindurant, thanks for your attention!

This PR blocks work to fix merge conflicts in #538 and would help having merged to continue development in #514 where I have a not yet understood test failure and I hope to rule out it's related to the old Dockerfiles. Please ping me for any followup questions or and I'll work to address them with haste!

[martindurant]

All good, happy with those conversations. Feel free to pick your favourite wording around the "production" image

[consideRatio]

Thank you @martindurant!! I updated USAGE DISCLAIMER: to be PURPOSE: instead and removed the part about not recommending use of the image as a base-image / in production entirely.

[martindurant]

👍