Skip to content

Conversation

@consideRatio
Copy link
Collaborator

@consideRatio consideRatio commented Nov 4, 2022

Summary

  • A bugfix of dask-gateway-server image for arm64 wasn't built or published
  • A new feature of providing the dask-gateway image (demo for scheduler/workers) for arm64 as well. Now the entire helm chart is arm64 compatible.
    • This involved adding a build stage where compilers etc were available to build wheels (psutils specifically), and then using them in the final stage of the Dockerfile. Note there is a quite cool trick involved here, were we avoid a COPY operation, but instead using --mount for the RUN command, making things from the previous stage accessible without creating another layer that takes up space in the final image.

Background

There is a bug in chartpress making us fail to build and publish the arm64 variant of the dask-gateway-server image. This PR makes us avoid that bug by making the dask-gateway image also support arm64.

@consideRatio consideRatio marked this pull request as draft November 4, 2022 15:10
@consideRatio consideRatio force-pushed the pr/trial-making-everything-arm64-compatible branch from 278fd11 to e8161cf Compare November 4, 2022 15:28
@consideRatio consideRatio changed the title Debugging arm64 compatibility dask-gateway image: make buildable and build for arm64 Nov 4, 2022
@consideRatio consideRatio force-pushed the pr/trial-making-everything-arm64-compatible branch 3 times, most recently from 60dd631 to 4996798 Compare November 4, 2022 16:18
@consideRatio consideRatio changed the title dask-gateway image: make buildable and build for arm64 Fix failure to build and publish arm64 images Nov 4, 2022
@consideRatio consideRatio force-pushed the pr/trial-making-everything-arm64-compatible branch from 4996798 to e96e3da Compare November 4, 2022 16:22
@consideRatio consideRatio marked this pull request as ready for review November 4, 2022 16:23
@consideRatio consideRatio added bug Something isn't working codebase:helm-chart and removed new labels Nov 4, 2022
@consideRatio consideRatio force-pushed the pr/trial-making-everything-arm64-compatible branch from e96e3da to e4c9631 Compare November 4, 2022 16:31
Copy link
Member

@jacobtomlinson jacobtomlinson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Neat!

# Install dask-gateway
COPY --chown=dask:dask . /opt/dask-gateway
RUN pip install --no-cache-dir \
ARG PIP_CACHE_DIR=/tmp/pip-cache
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this need to be set a second time?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yepp, another stage, another built-time env var needed to configure pip where to store/read cache

@consideRatio consideRatio merged commit 19a3644 into dask:main Nov 7, 2022
@consideRatio
Copy link
Collaborator Author

Going for a self-merge and a changelog for review/merge -> release

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working codebase:helm-chart

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Build arm64 dask-gateway-server image

2 participants