Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Build arm images #1305

Merged
merged 10 commits into from
Nov 4, 2022
Merged

Build arm images #1305

Show file tree
Hide file tree
Changes from 9 commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
63ce3e4
Build arm64 images
andrewwdye Nov 3, 2022
15c1a18
Comment out build/plugins
andrewwdye Nov 3, 2022
2a1b1e8
Fix build args
andrewwdye Nov 3, 2022
2e68b65
Fix build args extra quotation
andrewwdye Nov 3, 2022
bc77f94
Hardcode existing version
andrewwdye Nov 3, 2022
e3c71cd
Build phik wheel
andrewwdye Nov 3, 2022
193362c
Include build essentials in image
andrewwdye Nov 4, 2022
df52105
Fix sqlalchemy Dockerfile path
andrewwdye Nov 4, 2022
e6ebd80
Revert hacks for testing
andrewwdye Nov 4, 2022
b397a0d
Install fsspec for aws and gcp
andrewwdye Nov 4, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
72 changes: 51 additions & 21 deletions .github/workflows/pythonpublish.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -65,31 +65,61 @@ jobs:
- uses: actions/checkout@v2
with:
fetch-depth: "0"
- name: Build & Push Flytekit Python${{ matrix.python-version }} Docker Image to Github Registry
uses: whoan/docker-build-with-cache-action@v5
- name: Set up QEMU
uses: docker/setup-qemu-action@v1
- name: Set up Docker Buildx
id: buildx
uses: docker/setup-buildx-action@v1
- name: Login to GitHub Container Registry
if: ${{ github.event_name == 'release' }}
uses: docker/login-action@v1
with:
# https://docs.github.com/en/packages/learn-github-packages/publishing-a-package
registry: ghcr.io
username: "${{ secrets.FLYTE_BOT_USERNAME }}"
password: "${{ secrets.FLYTE_BOT_PAT }}"
image_name: ${{ github.repository_owner }}/flytekit
image_tag: py${{ matrix.python-version }}-latest,py${{ matrix.python-version }}-${{ github.sha }},py${{ matrix.python-version }}-${{ needs.deploy.outputs.version }}
push_git_tag: true
push_image_and_stages: true
registry: ghcr.io
build_extra_args: "--compress=true --build-arg=VERSION=${{ needs.deploy.outputs.version }} --build-arg=DOCKER_IMAGE=ghcr.io/flyteorg/flytekit:py${{ matrix.python-version }}-${{ needs.deploy.outputs.version }} --build-arg=PYTHON_VERSION=${{ matrix.python-version }}"
- name: Prepare Flytekit Image Names
id: flytekit-names
uses: docker/metadata-action@v3
with:
images: |
ghcr.io/${{ github.repository_owner }}/flytekit
tags: |
py${{ matrix.python-version }}-latest
py${{ matrix.python-version }}-${{ github.sha }}
py${{ matrix.python-version }}-${{ needs.deploy.outputs.version }}
- name: Build & Push Flytekit Python${{ matrix.python-version }} Docker Image to Github Registry
uses: docker/build-push-action@v2
with:
context: .
dockerfile: Dockerfile
platforms: linux/arm64, linux/amd64
push: ${{ github.event_name == 'release' }}
tags: ${{ steps.flytekit-names.outputs.tags }}
build-args: |
VERSION=${{ needs.deploy.outputs.version }}
DOCKER_IMAGE=ghcr.io/${{ github.repository_owner }}/flytekit:py${{ matrix.python-version }}-${{ needs.deploy.outputs.version }}
PYTHON_VERSION=${{ matrix.python-version }}
file: Dockerfile
cache-from: type=gha
cache-to: type=gha,mode=max
- name: Prepare SQLAlchemy Image Names
id: sqlalchemy-names
uses: docker/metadata-action@v3
with:
images: |
ghcr.io/${{ github.repository_owner }}/flytekit
tags: |
py${{ matrix.python-version }}-sqlalchemy-latest
py${{ matrix.python-version }}-sqlalchemy-${{ github.sha }}
py${{ matrix.python-version }}-sqlalchemy-${{ needs.deploy.outputs.version }}
- name: Push SQLAlchemy Image to GitHub Registry
uses: whoan/docker-build-with-cache-action@v5
uses: docker/build-push-action@v2
with:
# https://docs.github.com/en/packages/learn-github-packages/publishing-a-package
username: "${{ secrets.FLYTE_BOT_USERNAME }}"
password: "${{ secrets.FLYTE_BOT_PAT }}"
image_name: ${{ github.repository_owner }}/flytekit
image_tag: py${{ matrix.python-version }}-sqlalchemy-latest,py${{ matrix.python-version }}-sqlalchemy-${{ github.sha }},py${{ matrix.python-version }}-sqlalchemy-${{ needs.deploy.outputs.version }}
push_git_tag: true
push_image_and_stages: true
registry: ghcr.io
build_extra_args: "--compress=true --build-arg=VERSION=${{ needs.deploy.outputs.version }}"
context: "./plugins/flytekit-sqlalchemy/"
dockerfile: Dockerfile.py${{ matrix.python-version }}
platforms: linux/arm64, linux/amd64
push: ${{ github.event_name == 'release' }}
tags: ${{ steps.sqlalchemy-names.outputs.tags }}
build-args: |
VERSION=${{ needs.deploy.outputs.version }}
file: ./plugins/flytekit-sqlalchemy/Dockerfile.py${{ matrix.python-version }}
cache-from: type=gha
cache-to: type=gha,mode=max
10 changes: 6 additions & 4 deletions Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,11 +10,13 @@ ENV PYTHONPATH /root
ARG VERSION
ARG DOCKER_IMAGE

RUN apt-get update && apt-get install build-essential -y
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we could bring the builder back in a later PR.


# Pod tasks should be exposed in the default image
RUN pip install -U flytekit==$VERSION \
flytekitplugins-pod==$VERSION \
flytekitplugins-deck-standard==$VERSION \
flytekitplugins-data-fsspec==$VERSION \
scikit-learn
flytekitplugins-pod==$VERSION \
flytekitplugins-deck-standard==$VERSION \
flytekitplugins-data-fsspec==$VERSION \
andrewwdye marked this conversation as resolved.
Show resolved Hide resolved
scikit-learn

ENV FLYTE_INTERNAL_IMAGE "$DOCKER_IMAGE"