Add aarch64 wheel build support #3414

odidev · 2021-04-09T14:51:17Z

Added aarch64 wheel build support.
Related to #3413. @postrational could you please review this PR?

CLAassistant · 2021-04-09T14:52:34Z

All committers have signed the CLA.

.github/workflows/release_linux_aarch64.yml

.github/workflows/manylinux/entrypoint.sh

odidev · 2021-05-05T11:45:34Z

@jcwchen, could you please merge this PR. Thanks in advance.

jcwchen · 2021-05-05T16:02:36Z

Could @onnx/sig-archinfra-approvers please review this PR? Thanks.

odidev · 2021-06-23T10:16:44Z

Could you please review this PR? Thanks

askhade · 2021-06-23T16:19:01Z

@odidev : Please fix pipeline failures:

askhade · 2021-06-23T16:21:36Z

.github/workflows/release_linux_aarch64.yml

+    branches: [master, rel-*]
+  pull_request:
+    branches: [rel-*, master]  # TODO remove master before merge
+


add workflow_dispatch: on line 11 to enable manual trigger similar to : https://github.com/onnx/onnx/blob/master/.github/workflows/release_linux_x86_64.yml#L11

askhade · 2021-06-23T16:23:59Z

.github/workflows/release_linux_aarch64.yml

+
+jobs:
+  build:
+    runs-on: ubuntu-latest


add an if here similar to https://github.com/onnx/onnx/blob/master/.github/workflows/release_linux_x86_64.yml#L15
to enable manual trigger

We enable manual triggers from master by setting a label "run release CIs" on the PR

askhade · 2021-06-23T16:25:12Z

.github/workflows/release_linux_aarch64.yml

+        auth_header="$(git config --local --get http.https://github.com/.extraheader)"
+        git submodule sync --recursive
+        git -c "http.extraheader=$auth_header" -c protocol.version=2 submodule update --init --force --recursive --depth=1
+    - name: Install python dependencies


nit: add a space after even task for better readability

this workflow is different from others since you are running inside a docker container because this is aarch64 platform. Please add detailed comments through out this yml to help understand this workflow.

askhade · 2021-06-23T16:29:51Z

.github/workflows/release_linux_aarch64.yml

+      run: |
+        docker run --rm -v ${{ github.workspace }}:/ws:rw --workdir=/ws \
+          ${{ env.img }} ${{ env.py }} -m pip install -U numpy protobuf==3.11.3
+    - name: Install dependencies


Can you club "Install python dependencies" and "Install dependencies" together?

askhade · 2021-06-23T16:32:35Z

.github/workflows/release_linux_aarch64.yml

+      run: |
+        docker run --rm -v ${{ github.workspace }}:/ws:rw --workdir=/ws \
+          ${{ env.img }} \
+          bash -exc '${{ env.py }} -m pip install virtualenv && ${{ env.py }} -m venv .env && \


why do you need virtualenv?

By using virtual environment we are making already installed dependencies available for different tasks.

git actions does let you do this across tasks... you dont need virtual env for that

askhade · 2021-06-23T16:33:21Z

.github/workflows/release_linux_aarch64.yml

+          bash -exc '${{ env.py }} -m pip install virtualenv && ${{ env.py }} -m venv .env && \
+          source .env/bin/activate && \
+          python -m pip install --upgrade pip && \
+          pip install dist/*-manylinux2014_aarch64.whl && \


I think dist/*-manylinux2014_aarch64.whl should be changes to dist/*manylinux2014_aarch64.whl

askhade · 2021-06-23T16:35:34Z

.github/workflows/release_linux_aarch64.yml

+      if: (github.event_name == 'schedule') # Only triggered by weekly event
+      run: |
+        python -m pip install -q twine
+        twine upload --verbose dist/*.whl --repository-url https://test.pypi.org/legacy/ -u ${{ secrets.TESTPYPI_USERNAME }} -p ${{ secrets.TESTPYPI_PASSWORD }}


we added a few more verification steps to the pipelines to cut the manual testing effort. Please add them for this pipeline too. You can refer to: https://github.com/onnx/onnx/blob/master/.github/workflows/release_linux_x86_64.yml#L75

jcwchen · 2021-07-09T17:17:16Z

.github/workflows/release_linux_aarch64.yml

+  push:
+    branches: [master, rel-*]
+  pull_request:
+    branches: [rel-*, master]  # TODO remove master before merge


nit: this #TODO comment can be removed now

jcwchen · 2021-07-09T17:22:18Z

.github/workflows/release_linux_aarch64.yml

+    strategy:
+      matrix:
+        # the different python versions for building wheels
+        python-version: [cp37-cp37m, cp38-cp38, cp39-cp39]


How about Python 3.6?

jcwchen · 2021-07-09T17:42:10Z

.github/workflows/release_linux_aarch64.yml

+          bash -exc '${{ env.py }} -m pip install virtualenv && ${{ env.py }} -m venv .env && \
+          source .env/bin/activate && \
+          ${{ env.py }} -m pip install -U numpy protobuf==3.11.3 && \
+          yum install -y protobuf-compiler protobuf-devel


I assume the protobuf version from yum package is not 3.11.3, right? I think a better way is to install protobuf from source like this:

onnx/.github/workflows/manylinux/entrypoint.sh

Line 18 in 31f0fd6

# Build protobuf

BTW, I think anyway protobuf will be installed by entrypoint.sh? Is the installation here required?

Yes, I checked it, it is not required as it is already installed.

askhade · 2021-07-12T23:20:02Z

@odidev : can you please resolve conflicts?

askhade · 2021-07-13T04:31:26Z

@odidev : we can check this PR in once the conflicts are resolved

askhade · 2021-07-15T22:53:30Z

@odidev : FYI - The code freeze date for ONNX 1.10 is July 15th (today) since this PR is reviewed and ready we can wait until July 20th. Otherwise this PR will miss ONNX 1.10 release.

odidev · 2021-07-16T09:05:00Z

@odidev : can you please resolve conflicts?

Done.

Signed-off-by: odidev <odidev@puresoftware.com>

jcwchen

Thanks for updates! Overall looks good to me. Let me reopen this PR to test this new CI.

odidev requested a review from a team as a code owner April 9, 2021 14:51

odidev force-pushed the odidev_onnx branch from bdd946a to 803c348 Compare April 9, 2021 14:54

postrational requested a review from jcwchen April 9, 2021 20:56

jcwchen reviewed Apr 9, 2021

View reviewed changes

.github/workflows/release_linux_aarch64.yml Outdated Show resolved Hide resolved

jcwchen reviewed Apr 9, 2021

View reviewed changes

.github/workflows/manylinux/entrypoint.sh Outdated Show resolved Hide resolved

.github/workflows/manylinux/entrypoint.sh Outdated Show resolved Hide resolved

odidev force-pushed the odidev_onnx branch 2 times, most recently from 9f1e558 to 6a32cbf Compare April 13, 2021 04:24

askhade reviewed Jun 23, 2021

View reviewed changes

askhade added this to the 1.10 milestone Jun 29, 2021

odidev force-pushed the odidev_onnx branch from 6060424 to dc13b8f Compare July 9, 2021 05:34

jcwchen reviewed Jul 9, 2021

View reviewed changes

odidev force-pushed the odidev_onnx branch from 4bddaec to e5b123e Compare July 16, 2021 09:04

odidev force-pushed the odidev_onnx branch from e5b123e to 1dcefa3 Compare July 16, 2021 09:07

Add aarch64 wheel build support

bc0a96a

Signed-off-by: odidev <odidev@puresoftware.com>

odidev force-pushed the odidev_onnx branch from 1dcefa3 to bc0a96a Compare July 16, 2021 09:10

jcwchen added the run release CIs Use this label to trigger release tests in CI label Jul 16, 2021

jcwchen approved these changes Jul 16, 2021

View reviewed changes

jcwchen closed this Jul 16, 2021

jcwchen reopened this Jul 16, 2021

Merge branch 'master' into odidev_onnx

3d75454

askhade closed this Jul 19, 2021

askhade reopened this Jul 19, 2021

askhade approved these changes Jul 19, 2021

View reviewed changes

askhade enabled auto-merge (squash) July 19, 2021 23:52

askhade merged commit a71135c into onnx:master Jul 20, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add aarch64 wheel build support #3414

Add aarch64 wheel build support #3414

odidev commented Apr 9, 2021

CLAassistant commented Apr 9, 2021 •

edited

Loading

odidev commented May 5, 2021

jcwchen commented May 5, 2021

odidev commented Jun 23, 2021

askhade commented Jun 23, 2021

askhade Jun 23, 2021

odidev Jul 9, 2021

askhade Jun 23, 2021

odidev Jul 9, 2021

askhade Jun 23, 2021

askhade Jun 23, 2021

odidev Jul 9, 2021

askhade Jun 23, 2021

odidev Jul 9, 2021

askhade Jun 23, 2021

odidev Jul 9, 2021

askhade Jul 9, 2021

askhade Jun 23, 2021

odidev Jul 9, 2021

askhade Jun 23, 2021

odidev Jul 9, 2021

jcwchen Jul 9, 2021

odidev Jul 16, 2021

jcwchen Jul 9, 2021

odidev Jul 16, 2021

jcwchen Jul 9, 2021 •

edited

Loading

odidev Jul 16, 2021

askhade commented Jul 12, 2021

askhade commented Jul 13, 2021

askhade commented Jul 15, 2021

odidev commented Jul 16, 2021

jcwchen left a comment

Add aarch64 wheel build support #3414

Add aarch64 wheel build support #3414

Conversation

odidev commented Apr 9, 2021

CLAassistant commented Apr 9, 2021 • edited Loading

odidev commented May 5, 2021

jcwchen commented May 5, 2021

odidev commented Jun 23, 2021

askhade commented Jun 23, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jcwchen Jul 9, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

askhade commented Jul 12, 2021

askhade commented Jul 13, 2021

askhade commented Jul 15, 2021

odidev commented Jul 16, 2021

jcwchen left a comment

Choose a reason for hiding this comment

CLAassistant commented Apr 9, 2021 •

edited

Loading

jcwchen Jul 9, 2021 •

edited

Loading