Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CIBUILD] Upgrade cibuild docker images to ubuntu22.04 with cuda11.7. #457

Open
wants to merge 3 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ on:
- '*.md'

env:
IMAGE: alideeprec/deeprec-cicd:deeprec-dev-cpu-cibuild-py36-ubuntu18.04
IMAGE: alideeprec/deeprec-cicd:deeprec-dev-cpu-cibuild-py38-ubuntu22.04
JOBNAME: deeprec-ci-cpu-${{ github.run_id }}

jobs:
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/ubuntu18.04-py3.6-cibuild-build-wheel.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ on:
- '*.md'

env:
IMAGE: alideeprec/deeprec-cicd:deeprec-dev-cpu-cibuild-py36-ubuntu18.04
IMAGE: alideeprec/deeprec-cicd:deeprec-dev-cpu-cibuild-py38-ubuntu22.04
JOBNAME: deeprec-ci-cpu-${{ github.run_id }}

jobs:
Expand Down Expand Up @@ -52,4 +52,4 @@ jobs:
- name: Remove Container
if: ${{ always() }}
run:
cibuild/remove_container.sh ${JOBNAME}
cibuild/remove_container.sh ${JOBNAME}
4 changes: 2 additions & 2 deletions .github/workflows/ubuntu18.04-py3.6-cibuild-c-unit-test.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ on:
- '*.md'

env:
IMAGE: alideeprec/deeprec-cicd:deeprec-dev-cpu-cibuild-py36-ubuntu18.04
IMAGE: alideeprec/deeprec-cicd:deeprec-dev-cpu-cibuild-py38-ubuntu22.04
JOBNAME: deeprec-ci-cpu-${{ github.run_id }}

jobs:
Expand Down Expand Up @@ -52,4 +52,4 @@ jobs:
- name: Remove Container
if: ${{ always() }}
run:
cibuild/remove_container.sh ${JOBNAME}
cibuild/remove_container.sh ${JOBNAME}
4 changes: 2 additions & 2 deletions .github/workflows/ubuntu18.04-py3.6-cibuild-cc-unit-test.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ on:
- '*.md'

env:
IMAGE: alideeprec/deeprec-cicd:deeprec-dev-cpu-cibuild-py36-ubuntu18.04
IMAGE: alideeprec/deeprec-cicd:deeprec-dev-cpu-cibuild-py38-ubuntu22.04
JOBNAME: deeprec-ci-cpu-${{ github.run_id }}

jobs:
Expand Down Expand Up @@ -52,4 +52,4 @@ jobs:
- name: Remove Container
if: ${{ always() }}
run:
cibuild/remove_container.sh ${JOBNAME}
cibuild/remove_container.sh ${JOBNAME}
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ on:
- '*.md'

env:
IMAGE: alideeprec/deeprec-cicd:deeprec-dev-cpu-cibuild-py36-ubuntu18.04
IMAGE: alideeprec/deeprec-cicd:deeprec-dev-cpu-cibuild-py38-ubuntu22.04
JOBNAME: deeprec-ci-cpu-${{ github.run_id }}

jobs:
Expand Down Expand Up @@ -52,4 +52,4 @@ jobs:
- name: Remove Container
if: ${{ always() }}
run:
cibuild/remove_container.sh ${JOBNAME}
cibuild/remove_container.sh ${JOBNAME}
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ on:
- '*.md'

env:
IMAGE: alideeprec/deeprec-cicd:deeprec-dev-cpu-cibuild-py36-ubuntu18.04
IMAGE: alideeprec/deeprec-cicd:deeprec-dev-cpu-cibuild-py38-ubuntu22.04
JOBNAME: deeprec-ci-cpu-${{ github.run_id }}

jobs:
Expand Down Expand Up @@ -52,4 +52,4 @@ jobs:
- name: Remove Container
if: ${{ always() }}
run:
cibuild/remove_container.sh ${JOBNAME}
cibuild/remove_container.sh ${JOBNAME}
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ on:
- '*.md'

env:
IMAGE: alideeprec/deeprec-cicd:deeprec-dev-cpu-cibuild-py36-ubuntu18.04
IMAGE: alideeprec/deeprec-cicd:deeprec-dev-cpu-cibuild-py38-ubuntu22.04
JOBNAME: deeprec-ci-cpu-${{ github.run_id }}

jobs:
Expand Down Expand Up @@ -52,4 +52,4 @@ jobs:
- name: Remove Container
if: ${{ always() }}
run:
cibuild/remove_container.sh ${JOBNAME}
cibuild/remove_container.sh ${JOBNAME}
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ on:
- '*.md'

env:
IMAGE: alideeprec/deeprec-cicd:deeprec-dev-cpu-cibuild-py36-ubuntu18.04
IMAGE: alideeprec/deeprec-cicd:deeprec-dev-cpu-cibuild-py38-ubuntu22.04
JOBNAME: deeprec-ci-cpu-${{ github.run_id }}

jobs:
Expand Down Expand Up @@ -52,4 +52,4 @@ jobs:
- name: Remove Container
if: ${{ always() }}
run:
cibuild/remove_container.sh ${JOBNAME}
cibuild/remove_container.sh ${JOBNAME}
4 changes: 2 additions & 2 deletions .github/workflows/ubuntu18.04-py3.6-cibuild-js-unit-test.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ on:
- '*.md'

env:
IMAGE: alideeprec/deeprec-cicd:deeprec-dev-cpu-cibuild-py36-ubuntu18.04
IMAGE: alideeprec/deeprec-cicd:deeprec-dev-cpu-cibuild-py38-ubuntu22.04
JOBNAME: deeprec-ci-cpu-${{ github.run_id }}

jobs:
Expand Down Expand Up @@ -52,4 +52,4 @@ jobs:
- name: Remove Container
if: ${{ always() }}
run:
cibuild/remove_container.sh ${JOBNAME}
cibuild/remove_container.sh ${JOBNAME}
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ on:
- '*.md'

env:
IMAGE: alideeprec/deeprec-cicd:deeprec-dev-cpu-cibuild-py36-ubuntu18.04
IMAGE: alideeprec/deeprec-cicd:deeprec-dev-cpu-cibuild-py38-ubuntu22.04
JOBNAME: deeprec-ci-cpu-${{ github.run_id }}

jobs:
Expand Down Expand Up @@ -52,4 +52,4 @@ jobs:
- name: Remove Container
if: ${{ always() }}
run:
cibuild/remove_container.sh ${JOBNAME}
cibuild/remove_container.sh ${JOBNAME}
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ on:
- '*.md'

env:
IMAGE: alideeprec/deeprec-cicd:deeprec-dev-cpu-cibuild-py36-ubuntu18.04
IMAGE: alideeprec/deeprec-cicd:deeprec-dev-cpu-cibuild-py38-ubuntu22.04
JOBNAME: deeprec-ci-cpu-${{ github.run_id }}

jobs:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ on:
- '*.md'

env:
IMAGE: alideeprec/deeprec-cicd:deeprec-dev-cpu-cibuild-py36-ubuntu18.04
IMAGE: alideeprec/deeprec-cicd:deeprec-dev-cpu-cibuild-py38-ubuntu22.04
JOBNAME: deeprec-ci-cpu-${{ github.run_id }}

jobs:
Expand Down Expand Up @@ -52,4 +52,4 @@ jobs:
- name: Remove Container
if: ${{ always() }}
run:
cibuild/remove_container.sh ${JOBNAME}
cibuild/remove_container.sh ${JOBNAME}
2 changes: 1 addition & 1 deletion .github/workflows/ubuntu18.04-py3.6-cibuild-unit-test.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ name: CPU Unit Tests
on: workflow_dispatch

env:
IMAGE: alideeprec/deeprec-cicd:deeprec-dev-cpu-cibuild-py36-ubuntu18.04
IMAGE: alideeprec/deeprec-cicd:deeprec-dev-cpu-cibuild-py38-ubuntu22.04
JOBNAME: deeprec-ci-cpu-${{ github.run_id }}
PODNAME: deeprec-ci-cpu-${{ github.run_id }}-chief-0
BAZEL_CACHE: ${{ secrets.BAZEL_CACHE }}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ on:
- '*.md'

env:
IMAGE: alideeprec/deeprec-cicd:deeprec-dev-gpu-cibuild-py36-cu116-ubuntu18.04
IMAGE: alideeprec/deeprec-cicd:deeprec-dev-gpu-cibuild-py38-cu117-ubuntu22.04
JOBNAME: deeprec-ci-gpu-${{ github.run_id }}

jobs:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ on:
- '*.md'

env:
IMAGE: alideeprec/deeprec-cicd:deeprec-dev-gpu-cibuild-py36-cu116-ubuntu18.04
IMAGE: alideeprec/deeprec-cicd:deeprec-dev-gpu-cibuild-py38-cu117-ubuntu22.04
JOBNAME: deeprec-ci-gpu-${{ github.run_id }}

jobs:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ on:
- '*.md'

env:
IMAGE: alideeprec/deeprec-cicd:deeprec-dev-gpu-cibuild-py36-cu116-ubuntu18.04
IMAGE: alideeprec/deeprec-cicd:deeprec-dev-gpu-cibuild-py38-cu117-ubuntu22.04
JOBNAME: deeprec-ci-gpu-${{ github.run_id }}

jobs:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ on:
- '*.md'

env:
IMAGE: alideeprec/deeprec-cicd:deeprec-dev-gpu-cibuild-py36-cu116-ubuntu18.04
IMAGE: alideeprec/deeprec-cicd:deeprec-dev-gpu-cibuild-py38-cu117-ubuntu22.04
JOBNAME: deeprec-ci-gpu-${{ github.run_id }}

jobs:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ on:
- '*.md'

env:
IMAGE: alideeprec/deeprec-cicd:deeprec-dev-gpu-cibuild-py36-cu116-ubuntu18.04
IMAGE: alideeprec/deeprec-cicd:deeprec-dev-gpu-cibuild-py38-cu117-ubuntu22.04
JOBNAME: deeprec-ci-gpu-${{ github.run_id }}

jobs:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ on:
- '*.md'

env:
IMAGE: alideeprec/deeprec-cicd:deeprec-dev-gpu-cibuild-py36-cu116-ubuntu18.04
IMAGE: alideeprec/deeprec-cicd:deeprec-dev-gpu-cibuild-py38-cu117-ubuntu22.04
JOBNAME: deeprec-ci-gpu-${{ github.run_id }}

jobs:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ on:
- '*.md'

env:
IMAGE: alideeprec/deeprec-cicd:deeprec-dev-gpu-cibuild-py36-cu116-ubuntu18.04
IMAGE: alideeprec/deeprec-cicd:deeprec-dev-gpu-cibuild-py38-cu117-ubuntu22.04
JOBNAME: deeprec-ci-gpu-${{ github.run_id }}

jobs:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ on:
- '*.md'

env:
IMAGE: alideeprec/deeprec-cicd:deeprec-dev-gpu-cibuild-py36-cu116-ubuntu18.04
IMAGE: alideeprec/deeprec-cicd:deeprec-dev-gpu-cibuild-py38-cu117-ubuntu22.04
JOBNAME: deeprec-ci-gpu-${{ github.run_id }}

jobs:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ on:
- '*.md'

env:
IMAGE: alideeprec/deeprec-cicd:deeprec-dev-gpu-cibuild-py36-cu116-ubuntu18.04
IMAGE: alideeprec/deeprec-cicd:deeprec-dev-gpu-cibuild-py38-cu117-ubuntu22.04
JOBNAME: deeprec-ci-gpu-${{ github.run_id }}

jobs:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ on:
- '*.md'

env:
IMAGE: alideeprec/deeprec-cicd:deeprec-dev-gpu-cibuild-py36-cu116-ubuntu18.04
IMAGE: alideeprec/deeprec-cicd:deeprec-dev-gpu-cibuild-py38-cu117-ubuntu22.04
JOBNAME: deeprec-ci-gpu-${{ github.run_id }}

jobs:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ name: GPU Unit Tests
on: workflow_dispatch

env:
IMAGE: alideeprec/deeprec-cicd:deeprec-dev-gpu-cibuild-py36-cu116-ubuntu18.04
IMAGE: alideeprec/deeprec-cicd:deeprec-dev-gpu-cibuild-py38-cu117-ubuntu22.04
JOBNAME: deeprec-ci-gpu-${{ github.run_id }}
PODNAME: deeprec-ci-gpu-${{ github.run_id }}-chief-0
BAZEL_CACHE: ${{ secrets.BAZEL_CACHE }}
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/ubuntu18.04-py3.6-modeltest.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ on:
required: false

env:
IMAGE: registry.cn-shanghai.aliyuncs.com/pai-dlc/tensorflow-developer:1.15deeprec-dev-cpu-cibuild-py36-ubuntu18.04
IMAGE: alideeprec/deeprec-cicd:deeprec-dev-cpu-cibuild-py38-ubuntu22.04
JOBNAME: deeprec-ci-cpu-${{ github.run_id }}
PODNAME: deeprec-ci-cpu-${{ github.run_id }}-chief-0
BAZEL_CACHE: ${{ secrets.BAZEL_CACHE }}
Expand Down
2 changes: 2 additions & 0 deletions cibuild/dockerfiles/Dockerfile.base-py3.8-cu117-ubuntu22.04
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,8 @@ RUN apt-get update && \
&& apt-get clean && \
&& rm -rf /var/lib/apt/lists/*

RUN apt-get install libsqlite3-dev

#install python
WORKDIR /home/workdir
RUN wget https://www.python.org/ftp/python/3.8.6/Python-3.8.6.tgz \
Expand Down
2 changes: 2 additions & 0 deletions cibuild/dockerfiles/Dockerfile.base-py3.8-ubuntu22.04
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,8 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/*

RUN apt-get install libsqlite3-dev

#install python
WORKDIR /home/workdir
RUN wget https://www.python.org/ftp/python/3.8.6/Python-3.8.6.tgz \
Expand Down
18 changes: 18 additions & 0 deletions cibuild/dockerfiles/Dockerfile.devel-py3.8-cu117-ubuntu22.04
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
FROM alideeprec/deeprec-base:deeprec-base-gpu-py38-cu117-ubuntu22.04

RUN apt-get install -y libhdf5-dev
RUN apt-get install -y libz-dev
RUN apt-get install -y openjdk-8-jdk

RUN pip install \
numpy==1.19.5 && \
h5py==2.10.0 && \
spicy==0.16.0 && \
portpicker==1.4.0 && \
sklearn==0.0 && \
tensorflow-estimator==1.15.0 && \
grpcio==1.47.0 && \
grpcio-tools==1.47.0 && \
pyarrow==2.0.0 && \
pandas==1.1.5 && \
fastparquet==0.6.1
18 changes: 18 additions & 0 deletions cibuild/dockerfiles/Dockerfile.devel-py3.8-ubuntu22.04
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
FROM alideeprec/deeprec-base:deeprec-base-cpu-py38-ubuntu22.04

RUN apt-get install -y libhdf5-dev
RUN apt-get install -y libz-dev
RUN apt-get install -y openjdk-8-jdk

RUN pip install \
numpy==1.19.5 && \
h5py==2.10.0 && \
spicy==0.16.0 && \
portpicker==1.4.0 && \
sklearn==0.0 && \
tensorflow-estimator==1.15.0 && \
grpcio==1.47.0 && \
grpcio-tools==1.47.0 && \
pyarrow==2.0.0 && \
pandas==1.1.5 && \
fastparquet==0.6.1
33 changes: 26 additions & 7 deletions docs/DeepRec-Compile-And-Install.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,17 +21,36 @@
| 7.5.0 | 3.6.9 | CUDA 11.7.1 | alideeprec/deeprec-base:deeprec-base-gpu-py36-cu117-ubuntu18.04 |
| 11.2.0 | 3.8.6 | CUDA 11.7.1 | alideeprec/deeprec-base:deeprec-base-gpu-py38-cu117-ubuntu22.04 |

**CPU Dev Docker**

| GCC Version | Python Version | IMAGE |
| ----------- | -------------- | --------------------------------------------------- |
| 7.5.0 | 3.6.9 | alideeprec/deeprec-dev:deeprec-cpu-py36-ubuntu18.04 |
| 11.2.0 | 3.8.6 | alideeprec/deeprec-dev:deeprec-cpu-py38-ubuntu22.04 |

**GPU Dev Docker**

| GCC Version | Python Version | CUDA VERSION | IMAGE |
| ----------- | -------------- | ------------ | --------------------------------------------------------- |
| 7.5.0 | 3.6.9 | CUDA 11.6.1 | alideeprec/deeprec-dev:deeprec-gpu-py36-cu116-ubuntu18.04 |
| 11.2.0 | 3.8.6 | CUDA 11.7.1 | alideeprec/deeprec-dev:deeprec-gpu-py38-cu117-ubuntu22.04 |


**CPU Dev Docker (with bazel cache)**

```
alideeprec/deeprec-build:deeprec-dev-cpu-py36-ubuntu18.04
```
| GCC Version | Python Version | IMAGE |
| ----------- | -------------- | ---------------------------------------------------------------- |
| 7.5.0 | 3.6.9 | alideeprec/deeprec-cicd:deeprec-dev-cpu-cibuild-py36-ubuntu18.04 |
| 11.2.0 | 3.8.6 | alideeprec/deeprec-cicd:deeprec-dev-cpu-cibuild-py38-ubuntu22.04 |

**GPU(cuda11.6) Dev Docker (with bazel cache)**

```
alideeprec/deeprec-build:deeprec-dev-gpu-py36-cu116-ubuntu18.04
```
**GPU Dev Docker (with bazel cache)**

| GCC Version | Python Version | CUDA VERSION | IMAGE |
| ----------- | -------------- | ------------ | ---------------------------------------------------------------------- |
| 7.5.0 | 3.6.9 | CUDA 11.6.1 | alideeprec/deeprec-cicd:deeprec-dev-gpu-cibuild-py36-cu116-ubuntu18.04 |
| 11.2.0 | 3.8.6 | CUDA 11.7.1 | alideeprec/deeprec-cicd:deeprec-dev-gpu-cibuild-py38-cu117-ubuntu22.04 |


## 代码编译

Expand Down
Loading