[RFC] Add Intel GPU support into PyTorch CI/CD

[chuanqi129]

Motivation

As the [RFC] Intel GPU Upstreaming mentioned, to integrate the new Intel GPU device and its associated features into PyTorch, we need to implement PyTorch CI/CD tests specifically designed for Intel GPU devices. These tests will ensure the quality of incoming pull requests and gate their acceptance accordingly.

Design Philosophy

For the new part CI/CD enabling, we aim to leverage the existing PyTorch CI/CD infrastructure wherever possible. Intel GPU related test will be dispatched to Intel Develop Cloud (IDC) Instances, which provides Intel GPU hardware as self-hosted runners. Based on our understanding of current PyTorch CI/CD tests, we will divide Intel GPU related tests into several categories: pull test, inductor tests and other tests. For inductor tests, we will extend the existing inductor test workflow to accommodate Intel GPU inductor testing. And a new workflow will serve as the entry point for other tests for Intel GPU, mirroring the approach used for other devices. Overall, all Intel GPU tests follow the rules below.

• Docker based builds & tests
• Multiple steps both for inductor and other tests
  • Base Docker image build on AWS instance runners which provides PyTorch build and tests environment.
  • Wheel build on AWS instance runners directly.
  • Tests can be sharded and dispatched on IDC instance runners.

Detail

Entrance of pull test

For the basic build test for pull requests, we will add a new part for Intel GPU specific build in .github/workflows/pull.yml and triggered by each pull request.

Entrance of Inductor tests

For Inductor tests, we will add a new part for Intel GPU specific tests in .github/workflows/inductor.yml and triggered by ciflow/inductor. To avoid breaking other inductor related PRs at the first stage, we plan to add a new ciflow label ciflow/xpu in .github/pytorch-probot.yml, and limit the Intel GPU inductor tests only for PR which has both ciflow/inductor and ciflow/xpu.

Entrance of other tests

The Intel GPU device related remain tests will be triggered by PR with label ciflow/xpu or regular triggered by timer. To achieve it, we will add a new entrance workflow .github/workflows/xpu.yml, which like below content.

name: xpu

on:
  push:
    branches:
      - main
      - release/*
    tags:
      - ciflow/xpu/*
  workflow_dispatch:
  schedule:
    - cron: 0 0 * * *  

concurrency:
  group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref_name }}-${{ github.ref_type == 'branch' && github.sha }}-${{ github.event_name == 'workflow_dispatch' }}-${{ github.event_name == 'schedule' }}
  cancel-in-progress: true

jobs:
  linux-jammy-xpu-py3_8-build:
    name: linux-jammy-xpu-py3.8
    uses: ./.github/workflows/_linux-build.yml
    with:
      build-environment: linux-jammy-xpu-py3.8
      docker-image-name: pytorch-linux-jammy-xpu-n-py3
      sync-tag: xpu-build
      test-matrix: |
        { include: [
          { config: "default", shard: 1, num_shards: 2, runner: "linux.idc.xpu" },
          { config: "default", shard: 2, num_shards: 2, runner: "linux.idc.xpu" },
        ]}

  linux-jammy-xpu-py3_8-test:
    name: linux-jammy-xpu-py3.8
    uses: ./.github/workflows/_xpu-test.yml
    needs: linux-jammy-xpu-py3_8-build
    with:
      build-environment: linux-jammy-xpu-py3.8
      docker-image: ${{ needs.linux-jammy-xpu-py3_8-build.outputs.docker-image }}
      test-matrix: ${{ needs.linux-jammy-xpu-py3_8-build.outputs.test-matrix }} 

Build & Test

Will add Intel GPU specific base image Dockerfile .ci/docker/ubuntu-xpu/Dockerfile and Intel GPU part into image build script .ci/docker/build.sh to support Intel GPU based image build on linux.2xlarge runners.

For Pytorch wheel build, different with other devices, currently we need dispatch it to IDC instance runners. We will reuse .github/workflows/_linux-build.yml with Intel GPU specific build-environment and add Intel GPU part into Pytorch build script .ci/pytorch/build.sh.

For the tests part, we will add a new Intel GPU test workflow .github/workflows/_xpu-test.yml and some necessary GitHub action such as setup-xpu, teardown-xpu, etc. We also will add a new part in test script .ci/pytorch/test.sh and a series utils scripts for Intel GPU.

cc @seemethere @malfet @pytorch/pytorch-dev-infra @frank-wei @jgong5 @mingfeima @XiaobingSuper @sanchitintel @ashokei @jingxu10

[EikanWang]

@malfet , any comments?

[jbschlosser]

Marking triage review for visibility.

[malfet]

What is described in this RFC feels like a NorthStart, which will take a while to achieve, as implementing full CI/CD pipeline from the very beginning imply that all PyTorch developer needs to learn how to develop/debug/fix regressions for XPU backend, which does not feel like a reasonable expectation until backend reaches some level of feature maturity.

I think goal of enabling full CI should be broken into multiple stages, for example:

• Stage 0: XPU workflows are only triggered by label, which can be auto-applied when certain files are modified (see how its done for ciflow/mps for example)
• Stage 1: Conditional on build time/error messages readability/regression frequency: XPU build jobs are running on every PR.
  • Conditional that XPU builds are designed in a way that one rarely need to make XPU specific changes for their PRs.
  • If XPU builds are designed in in a way that for every other CPU/CUDA change dev would require a SYNC code modification, it should stay disabled by default and should be forward fixable by XPU codebase maintainers
  • Alternatively, SYCL becomes a new parallel computing programming paradigm, where there are performant backend into mobile/desktop/server GPUs, in which case we should throw away a bunch of custom code and all switch to SYCL
• Stage 2: XPU CI is running on label/periodically and yields reliable signal (i.e. XPU runners are available, XPU tests are roughly as reliable as CPU or CUDA tests)
  • If XPU runners are often unavailable, frequently fail to download build artifacts or fail during XPU initialization step then XPU CI remains limited to an opt-in runs (i.e. stages 0+1)
• Stage 3: XPU CI is running on every commit and regressions to it will result in PR reverts
  • Moving to this stage should be conditional to the infra stability requirement (outlined at stage 2) as well as some vaguely defined adoption metric (i.e. XPU HW is easily available to PyTorch user/developer community

CD requirements are much easier:

• One should be able to enable it on nightlies conditional on some sort of reasonable build time/binary artifacts expectations (i.e. if build requires 3+ hours or produces 5+Gb binary then it's a no go on a standard PyTorch runners)

Additional CD/UX points to consider:

• Is XPU support mutually exclusive from say CUDA?
• How big is the XPU binary footprint?
  If say XPU can co-exist with CUDA and binary footprint is small (say 30Mb) then why not include XPU support in all Linux x86_64 wheels?

[EikanWang]

Thanks for the practical comments. It is great that we are aligned in our thinking. In terms of execution, the staging plan aligns directly with our current approach. We are currently focused on stage 0 and developing a more detailed design.