[FX][docs] Document gotcha about training flag #68190 #68913

jubinchheda · 2021-11-25T11:20:08Z

Summary: [FX][docs] Document gotcha about training flag #68190

Test Plan: make html and view differences

Reviewers: jamesr66a

Subscribers: jamesr66a

Tasks:

Tags:

Committer: jubinchheda jubinchheda@fb.com

Fixes #{issue number}

Summary: [FX][docs] Document gotcha about training flag pytorch#68190 Test Plan: make html and view differences Reviewers: jamesr66a Subscribers: jamesr66a Tasks: Tags: Committer: jubinchheda <jubinchheda@fb.com>

pytorch-probot · 2021-11-25T11:20:13Z

CI Flow Status

⚛️ CI Flow

Ruleset - Version: v1
Ruleset - File: https://github.com/jubinchheda/pytorch/blob/35e3282227582f2348dd3a646550244bb0a1232e/.github/generated-ciflow-ruleset.json
PR ciflow labels: ciflow/default

Workflows	Labels (bold enabled)	Status
Triggered Workflows
linux-bionic-py3.6-clang9	`ciflow/all`, `ciflow/cpu`, `ciflow/default`, `ciflow/linux`, `ciflow/noarch`, `ciflow/xla`	✅ triggered
linux-vulkan-bionic-py3.6-clang9	`ciflow/all`, `ciflow/cpu`, `ciflow/default`, `ciflow/linux`, `ciflow/vulkan`	✅ triggered
linux-xenial-cuda11.3-py3.6-gcc7	`ciflow/all`, `ciflow/cuda`, `ciflow/default`, `ciflow/linux`	✅ triggered
linux-xenial-py3-clang5-mobile-build	`ciflow/all`, `ciflow/default`, `ciflow/linux`, `ciflow/mobile`	✅ triggered
linux-xenial-py3-clang5-mobile-custom-build-static	`ciflow/all`, `ciflow/default`, `ciflow/linux`, `ciflow/mobile`	✅ triggered
linux-xenial-py3.6-clang7-asan	`ciflow/all`, `ciflow/cpu`, `ciflow/default`, `ciflow/linux`, `ciflow/sanitizers`	✅ triggered
linux-xenial-py3.6-clang7-onnx	`ciflow/all`, `ciflow/cpu`, `ciflow/default`, `ciflow/linux`, `ciflow/onnx`	✅ triggered
linux-xenial-py3.6-gcc5.4	`ciflow/all`, `ciflow/cpu`, `ciflow/default`, `ciflow/linux`	✅ triggered
linux-xenial-py3.6-gcc7	`ciflow/all`, `ciflow/cpu`, `ciflow/default`, `ciflow/linux`	✅ triggered
linux-xenial-py3.6-gcc7-bazel-test	`ciflow/all`, `ciflow/bazel`, `ciflow/cpu`, `ciflow/default`, `ciflow/linux`	✅ triggered
pytorch-linux-xenial-py3-clang5-android-ndk-r19c-gradle-custom-build-single	`ciflow/all`, `ciflow/android`, `ciflow/cpu`, `ciflow/default`, `ciflow/linux`	✅ triggered
pytorch-linux-xenial-py3-clang5-android-ndk-r19c-gradle-custom-build-single-full-jit	`ciflow/all`, `ciflow/android`, `ciflow/cpu`, `ciflow/default`, `ciflow/linux`	✅ triggered
win-vs2019-cpu-py3	`ciflow/all`, `ciflow/cpu`, `ciflow/default`, `ciflow/win`	✅ triggered
win-vs2019-cuda11.3-py3	`ciflow/all`, `ciflow/cuda`, `ciflow/default`, `ciflow/win`	✅ triggered
Skipped Workflows
caffe2-linux-xenial-py3.6-gcc5.4	`ciflow/all`, `ciflow/cpu`, `ciflow/linux`	🚫 skipped
docker-builds	`ciflow/all`	🚫 skipped
ios-12-5-1-arm64	`ciflow/all`, `ciflow/ios`, `ciflow/macos`	🚫 skipped
ios-12-5-1-arm64-coreml	`ciflow/all`, `ciflow/ios`, `ciflow/macos`	🚫 skipped
ios-12-5-1-arm64-custom-ops	`ciflow/all`, `ciflow/ios`, `ciflow/macos`	🚫 skipped
ios-12-5-1-arm64-full-jit	`ciflow/all`, `ciflow/ios`, `ciflow/macos`	🚫 skipped
ios-12-5-1-arm64-metal	`ciflow/all`, `ciflow/ios`, `ciflow/macos`	🚫 skipped
ios-12-5-1-x86-64	`ciflow/all`, `ciflow/ios`, `ciflow/macos`	🚫 skipped
ios-12-5-1-x86-64-coreml	`ciflow/all`, `ciflow/ios`, `ciflow/macos`	🚫 skipped
ios-12-5-1-x86-64-full-jit	`ciflow/all`, `ciflow/ios`, `ciflow/macos`	🚫 skipped
libtorch-linux-xenial-cuda10.2-py3.6-gcc7	`ciflow/all`, `ciflow/cuda`, `ciflow/libtorch`, `ciflow/linux`	🚫 skipped
libtorch-linux-xenial-cuda11.3-py3.6-gcc7	`ciflow/all`, `ciflow/cuda`, `ciflow/libtorch`, `ciflow/linux`	🚫 skipped
linux-bionic-cuda10.2-py3.9-gcc7	`ciflow/all`, `ciflow/cuda`, `ciflow/linux`, `ciflow/slow`	🚫 skipped
macos-10-15-py3-arm64	`ciflow/all`, `ciflow/macos`	🚫 skipped
macos-10-15-py3-lite-interpreter-x86-64	`ciflow/all`, `ciflow/macos`	🚫 skipped
macos-10-15-py3-x86-64	`ciflow/all`, `ciflow/macos`	🚫 skipped
parallelnative-linux-xenial-py3.6-gcc5.4	`ciflow/all`, `ciflow/cpu`, `ciflow/linux`	🚫 skipped
periodic-libtorch-linux-xenial-cuda11.1-py3.6-gcc7	`ciflow/all`, `ciflow/cuda`, `ciflow/libtorch`, `ciflow/linux`, `ciflow/scheduled`	🚫 skipped
periodic-linux-xenial-cuda10.2-py3-gcc7-slow-gradcheck	`ciflow/all`, `ciflow/cuda`, `ciflow/linux`, `ciflow/scheduled`, `ciflow/slow`, `ciflow/slow-gradcheck`	🚫 skipped
periodic-linux-xenial-cuda11.1-py3.6-gcc7-debug	`ciflow/all`, `ciflow/cuda`, `ciflow/linux`, `ciflow/scheduled`	🚫 skipped
periodic-win-vs2019-cuda11.1-py3	`ciflow/all`, `ciflow/cuda`, `ciflow/scheduled`, `ciflow/win`	🚫 skipped

You can add a comment to the PR and tag @pytorchbot with the following commands:

# ciflow rerun, "ciflow/default" will always be added automatically
@pytorchbot ciflow rerun

# ciflow rerun with additional labels "-l <ciflow/label_name>", which is equivalent to adding these labels manually and trigger the rerun
@pytorchbot ciflow rerun -l ciflow/scheduled -l ciflow/slow

For more information, please take a look at the CI Flow Wiki.

facebook-github-bot · 2021-11-25T11:20:15Z

🔗 Helpful links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/68913
📄 Preview docs built from this PR
📄 Preview C++ docs built from this PR
↩️ [fb-only] Re-run with SSH instructions
🔧 Opt-in to CIFlow to control what jobs run on your PRs

💊 CI failures summary and remediations

As of commit 35e3282 (more details on the Dr. CI page):

15/16 failures introduced in this PR
1/16 tentatively recognized as flaky ❄️
- Click here to rerun these jobs

🕵️ 12 new failures recognized by patterns

The following CI failures do not appear to be due to upstream breakages:

pytorch-linux-xenial-py3-clang5-android-ndk-r19c-gradle-custom-build-single-full-jit / build-and-test (1/12)

Step: "Unknown" (full log | diagnosis details | 🔁 rerun)

2021-11-25T13:09:05.3857531Z �[36;1m echo "ERR...t available for the merge-base of your branch"�[0m

2021-11-25T13:09:05.3851608Z �[36;1mfi�[0m
2021-11-25T13:09:05.3852059Z �[36;1m# Covers the case where a previous tag doesn't exist for the tree�[0m
2021-11-25T13:09:05.3852784Z �[36;1m# this is only really applicable on trees that don't have `.circleci/docker` at its merge base, i.e. nightly�[0m
2021-11-25T13:09:05.3853460Z �[36;1mif ! git rev-parse "$MERGE_BASE:.circleci/docker"; then�[0m
2021-11-25T13:09:05.3854203Z �[36;1m  echo "Directory '.circleci/docker' not found in commit $MERGE_BASE, you should probably rebase onto a more recent commit"�[0m
2021-11-25T13:09:05.3854803Z �[36;1m  exit 1�[0m
2021-11-25T13:09:05.3855074Z �[36;1mfi�[0m
2021-11-25T13:09:05.3855533Z �[36;1mPREVIOUS_DOCKER_TAG=$(git rev-parse "$MERGE_BASE:.circleci/docker")�[0m
2021-11-25T13:09:05.3856212Z �[36;1m# If no image exists but the hash is the same as the previous hash then we should error out here�[0m
2021-11-25T13:09:05.3856832Z �[36;1mif [[ "${PREVIOUS_DOCKER_TAG}" = "${DOCKER_TAG}" ]]; then�[0m
2021-11-25T13:09:05.3857531Z �[36;1m  echo "ERROR: Something has gone wrong and the previous image isn't available for the merge-base of your branch"�[0m
2021-11-25T13:09:05.3858268Z �[36;1m  echo "       contact the PyTorch team to restore the original images"�[0m
2021-11-25T13:09:05.3858726Z �[36;1m  exit 1�[0m
2021-11-25T13:09:05.3858997Z �[36;1mfi�[0m
2021-11-25T13:09:05.3859367Z �[36;1mecho ::set-output name=rebuild::yes�[0m
2021-11-25T13:09:05.3870081Z shell: /usr/bin/bash -e {0}
2021-11-25T13:09:05.3870411Z env:
2021-11-25T13:09:05.3871997Z   BUILD_ENVIRONMENT: pytorch-linux-xenial-py3-clang5-android-ndk-r19c-gradle-custom-build-single-full-jit
2021-11-25T13:09:05.3874294Z   DOCKER_IMAGE_BASE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-xenial-py3-clang5-android-ndk-r19c
2021-11-25T13:09:05.3875583Z   SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2
2021-11-25T13:09:05.3876501Z   XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla

pytorch-linux-xenial-py3-clang5-android-ndk-r19c-gradle-custom-build-single / build-and-test (2/12)

Step: "Unknown" (full log | diagnosis details | 🔁 rerun)

2021-11-25T13:08:19.1551082Z �[36;1m echo "ERR...t available for the merge-base of your branch"�[0m

2021-11-25T13:08:19.1545142Z �[36;1mfi�[0m
2021-11-25T13:08:19.1545603Z �[36;1m# Covers the case where a previous tag doesn't exist for the tree�[0m
2021-11-25T13:08:19.1546332Z �[36;1m# this is only really applicable on trees that don't have `.circleci/docker` at its merge base, i.e. nightly�[0m
2021-11-25T13:08:19.1547030Z �[36;1mif ! git rev-parse "$MERGE_BASE:.circleci/docker"; then�[0m
2021-11-25T13:08:19.1547756Z �[36;1m  echo "Directory '.circleci/docker' not found in commit $MERGE_BASE, you should probably rebase onto a more recent commit"�[0m
2021-11-25T13:08:19.1548349Z �[36;1m  exit 1�[0m
2021-11-25T13:08:19.1548621Z �[36;1mfi�[0m
2021-11-25T13:08:19.1549087Z �[36;1mPREVIOUS_DOCKER_TAG=$(git rev-parse "$MERGE_BASE:.circleci/docker")�[0m
2021-11-25T13:08:19.1549783Z �[36;1m# If no image exists but the hash is the same as the previous hash then we should error out here�[0m
2021-11-25T13:08:19.1550392Z �[36;1mif [[ "${PREVIOUS_DOCKER_TAG}" = "${DOCKER_TAG}" ]]; then�[0m
2021-11-25T13:08:19.1551082Z �[36;1m  echo "ERROR: Something has gone wrong and the previous image isn't available for the merge-base of your branch"�[0m
2021-11-25T13:08:19.1551836Z �[36;1m  echo "       contact the PyTorch team to restore the original images"�[0m
2021-11-25T13:08:19.1552280Z �[36;1m  exit 1�[0m
2021-11-25T13:08:19.1552561Z �[36;1mfi�[0m
2021-11-25T13:08:19.1552923Z �[36;1mecho ::set-output name=rebuild::yes�[0m
2021-11-25T13:08:19.1563010Z shell: /usr/bin/bash -e {0}
2021-11-25T13:08:19.1563325Z env:
2021-11-25T13:08:19.1564688Z   BUILD_ENVIRONMENT: pytorch-linux-xenial-py3-clang5-android-ndk-r19c-gradle-custom-build-single
2021-11-25T13:08:19.1566787Z   DOCKER_IMAGE_BASE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-xenial-py3-clang5-android-ndk-r19c
2021-11-25T13:08:19.1568072Z   SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2
2021-11-25T13:08:19.1569022Z   XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla

linux-bionic-py3.6-clang9 / build (3/12)

Step: "Unknown" (full log | diagnosis details | 🔁 rerun)

2021-11-25T13:09:11.1412247Z �[36;1m echo "ERR...t available for the merge-base of your branch"�[0m

2021-11-25T13:09:11.1406466Z �[36;1mfi�[0m
2021-11-25T13:09:11.1406913Z �[36;1m# Covers the case where a previous tag doesn't exist for the tree�[0m
2021-11-25T13:09:11.1407607Z �[36;1m# this is only really applicable on trees that don't have `.circleci/docker` at its merge base, i.e. nightly�[0m
2021-11-25T13:09:11.1408286Z �[36;1mif ! git rev-parse "$MERGE_BASE:.circleci/docker"; then�[0m
2021-11-25T13:09:11.1409010Z �[36;1m  echo "Directory '.circleci/docker' not found in commit $MERGE_BASE, you should probably rebase onto a more recent commit"�[0m
2021-11-25T13:09:11.1409576Z �[36;1m  exit 1�[0m
2021-11-25T13:09:11.1409857Z �[36;1mfi�[0m
2021-11-25T13:09:11.1410296Z �[36;1mPREVIOUS_DOCKER_TAG=$(git rev-parse "$MERGE_BASE:.circleci/docker")�[0m
2021-11-25T13:09:11.1410974Z �[36;1m# If no image exists but the hash is the same as the previous hash then we should error out here�[0m
2021-11-25T13:09:11.1411582Z �[36;1mif [[ "${PREVIOUS_DOCKER_TAG}" = "${DOCKER_TAG}" ]]; then�[0m
2021-11-25T13:09:11.1412247Z �[36;1m  echo "ERROR: Something has gone wrong and the previous image isn't available for the merge-base of your branch"�[0m
2021-11-25T13:09:11.1412977Z �[36;1m  echo "       contact the PyTorch team to restore the original images"�[0m
2021-11-25T13:09:11.1413497Z �[36;1m  exit 1�[0m
2021-11-25T13:09:11.1413777Z �[36;1mfi�[0m
2021-11-25T13:09:11.1414148Z �[36;1mecho ::set-output name=rebuild::yes�[0m
2021-11-25T13:09:11.1424796Z shell: /usr/bin/bash -e {0}
2021-11-25T13:09:11.1425123Z env:
2021-11-25T13:09:11.1425579Z   BUILD_ENVIRONMENT: linux-bionic-py3.6-clang9
2021-11-25T13:09:11.1426547Z   DOCKER_IMAGE_BASE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-py3.6-clang9
2021-11-25T13:09:11.1427561Z   SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2
2021-11-25T13:09:11.1428466Z   XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla

linux-xenial-py3.6-gcc5.4 / build (4/12)

Step: "Unknown" (full log | diagnosis details | 🔁 rerun)

2021-11-25T13:08:52.4026221Z �[36;1m echo "ERR...t available for the merge-base of your branch"�[0m

2021-11-25T13:08:52.4020121Z �[36;1mfi�[0m
2021-11-25T13:08:52.4020592Z �[36;1m# Covers the case where a previous tag doesn't exist for the tree�[0m
2021-11-25T13:08:52.4021345Z �[36;1m# this is only really applicable on trees that don't have `.circleci/docker` at its merge base, i.e. nightly�[0m
2021-11-25T13:08:52.4022060Z �[36;1mif ! git rev-parse "$MERGE_BASE:.circleci/docker"; then�[0m
2021-11-25T13:08:52.4022807Z �[36;1m  echo "Directory '.circleci/docker' not found in commit $MERGE_BASE, you should probably rebase onto a more recent commit"�[0m
2021-11-25T13:08:52.4023422Z �[36;1m  exit 1�[0m
2021-11-25T13:08:52.4023698Z �[36;1mfi�[0m
2021-11-25T13:08:52.4024168Z �[36;1mPREVIOUS_DOCKER_TAG=$(git rev-parse "$MERGE_BASE:.circleci/docker")�[0m
2021-11-25T13:08:52.4024882Z �[36;1m# If no image exists but the hash is the same as the previous hash then we should error out here�[0m
2021-11-25T13:08:52.4025506Z �[36;1mif [[ "${PREVIOUS_DOCKER_TAG}" = "${DOCKER_TAG}" ]]; then�[0m
2021-11-25T13:08:52.4026221Z �[36;1m  echo "ERROR: Something has gone wrong and the previous image isn't available for the merge-base of your branch"�[0m
2021-11-25T13:08:52.4026995Z �[36;1m  echo "       contact the PyTorch team to restore the original images"�[0m
2021-11-25T13:08:52.4027527Z �[36;1m  exit 1�[0m
2021-11-25T13:08:52.4027814Z �[36;1mfi�[0m
2021-11-25T13:08:52.4028186Z �[36;1mecho ::set-output name=rebuild::yes�[0m
2021-11-25T13:08:52.4038342Z shell: /usr/bin/bash -e {0}
2021-11-25T13:08:52.4038674Z env:
2021-11-25T13:08:52.4039150Z   BUILD_ENVIRONMENT: linux-xenial-py3.6-gcc5.4
2021-11-25T13:08:52.4040168Z   DOCKER_IMAGE_BASE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-xenial-py3.6-gcc5.4
2021-11-25T13:08:52.4041249Z   SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2
2021-11-25T13:08:52.4042210Z   XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla

linux-xenial-cuda11.3-py3.6-gcc7 / build (5/12)

Step: "Unknown" (full log | diagnosis details | 🔁 rerun)

2021-11-25T13:09:05.4482451Z �[36;1m echo "ERR...t available for the merge-base of your branch"�[0m

2021-11-25T13:09:05.4476575Z �[36;1mfi�[0m
2021-11-25T13:09:05.4477017Z �[36;1m# Covers the case where a previous tag doesn't exist for the tree�[0m
2021-11-25T13:09:05.4477724Z �[36;1m# this is only really applicable on trees that don't have `.circleci/docker` at its merge base, i.e. nightly�[0m
2021-11-25T13:09:05.4478390Z �[36;1mif ! git rev-parse "$MERGE_BASE:.circleci/docker"; then�[0m
2021-11-25T13:09:05.4479114Z �[36;1m  echo "Directory '.circleci/docker' not found in commit $MERGE_BASE, you should probably rebase onto a more recent commit"�[0m
2021-11-25T13:09:05.4479697Z �[36;1m  exit 1�[0m
2021-11-25T13:09:05.4479960Z �[36;1mfi�[0m
2021-11-25T13:09:05.4480414Z �[36;1mPREVIOUS_DOCKER_TAG=$(git rev-parse "$MERGE_BASE:.circleci/docker")�[0m
2021-11-25T13:09:05.4481081Z �[36;1m# If no image exists but the hash is the same as the previous hash then we should error out here�[0m
2021-11-25T13:09:05.4481688Z �[36;1mif [[ "${PREVIOUS_DOCKER_TAG}" = "${DOCKER_TAG}" ]]; then�[0m
2021-11-25T13:09:05.4482451Z �[36;1m  echo "ERROR: Something has gone wrong and the previous image isn't available for the merge-base of your branch"�[0m
2021-11-25T13:09:05.4483171Z �[36;1m  echo "       contact the PyTorch team to restore the original images"�[0m
2021-11-25T13:09:05.4483621Z �[36;1m  exit 1�[0m
2021-11-25T13:09:05.4483900Z �[36;1mfi�[0m
2021-11-25T13:09:05.4484258Z �[36;1mecho ::set-output name=rebuild::yes�[0m
2021-11-25T13:09:05.4493850Z shell: /usr/bin/bash -e {0}
2021-11-25T13:09:05.4494156Z env:
2021-11-25T13:09:05.4494691Z   BUILD_ENVIRONMENT: linux-xenial-cuda11.3-py3.6-gcc7
2021-11-25T13:09:05.4495880Z   DOCKER_IMAGE_BASE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-xenial-cuda11.3-cudnn8-py3-gcc7
2021-11-25T13:09:05.4497074Z   SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2
2021-11-25T13:09:05.4497990Z   XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla

linux-xenial-py3-clang5-mobile-build / build (6/12)

Step: "Unknown" (full log | diagnosis details | 🔁 rerun)

2021-11-25T13:08:55.3230027Z �[36;1m echo "ERR...t available for the merge-base of your branch"�[0m

2021-11-25T13:08:55.3224115Z �[36;1mfi�[0m
2021-11-25T13:08:55.3224554Z �[36;1m# Covers the case where a previous tag doesn't exist for the tree�[0m
2021-11-25T13:08:55.3225255Z �[36;1m# this is only really applicable on trees that don't have `.circleci/docker` at its merge base, i.e. nightly�[0m
2021-11-25T13:08:55.3225934Z �[36;1mif ! git rev-parse "$MERGE_BASE:.circleci/docker"; then�[0m
2021-11-25T13:08:55.3226656Z �[36;1m  echo "Directory '.circleci/docker' not found in commit $MERGE_BASE, you should probably rebase onto a more recent commit"�[0m
2021-11-25T13:08:55.3227358Z �[36;1m  exit 1�[0m
2021-11-25T13:08:55.3227628Z �[36;1mfi�[0m
2021-11-25T13:08:55.3228066Z �[36;1mPREVIOUS_DOCKER_TAG=$(git rev-parse "$MERGE_BASE:.circleci/docker")�[0m
2021-11-25T13:08:55.3228745Z �[36;1m# If no image exists but the hash is the same as the previous hash then we should error out here�[0m
2021-11-25T13:08:55.3229355Z �[36;1mif [[ "${PREVIOUS_DOCKER_TAG}" = "${DOCKER_TAG}" ]]; then�[0m
2021-11-25T13:08:55.3230027Z �[36;1m  echo "ERROR: Something has gone wrong and the previous image isn't available for the merge-base of your branch"�[0m
2021-11-25T13:08:55.3230765Z �[36;1m  echo "       contact the PyTorch team to restore the original images"�[0m
2021-11-25T13:08:55.3231208Z �[36;1m  exit 1�[0m
2021-11-25T13:08:55.3231543Z �[36;1mfi�[0m
2021-11-25T13:08:55.3231903Z �[36;1mecho ::set-output name=rebuild::yes�[0m
2021-11-25T13:08:55.3241966Z shell: /usr/bin/bash -e {0}
2021-11-25T13:08:55.3242279Z env:
2021-11-25T13:08:55.3242846Z   BUILD_ENVIRONMENT: linux-xenial-py3-clang5-mobile-build
2021-11-25T13:08:55.3243957Z   DOCKER_IMAGE_BASE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-xenial-py3-clang5-asan
2021-11-25T13:08:55.3245018Z   SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2
2021-11-25T13:08:55.3245905Z   XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla

linux-xenial-py3-clang5-mobile-custom-build-static / build (7/12)

Step: "Unknown" (full log | diagnosis details | 🔁 rerun)

2021-11-25T13:08:23.6644307Z �[36;1m echo "ERR...t available for the merge-base of your branch"�[0m

2021-11-25T13:08:23.6637877Z �[36;1mfi�[0m
2021-11-25T13:08:23.6638869Z �[36;1m# Covers the case where a previous tag doesn't exist for the tree�[0m
2021-11-25T13:08:23.6639754Z �[36;1m# this is only really applicable on trees that don't have `.circleci/docker` at its merge base, i.e. nightly�[0m
2021-11-25T13:08:23.6640405Z �[36;1mif ! git rev-parse "$MERGE_BASE:.circleci/docker"; then�[0m
2021-11-25T13:08:23.6641100Z �[36;1m  echo "Directory '.circleci/docker' not found in commit $MERGE_BASE, you should probably rebase onto a more recent commit"�[0m
2021-11-25T13:08:23.6641649Z �[36;1m  exit 1�[0m
2021-11-25T13:08:23.6641918Z �[36;1mfi�[0m
2021-11-25T13:08:23.6642345Z �[36;1mPREVIOUS_DOCKER_TAG=$(git rev-parse "$MERGE_BASE:.circleci/docker")�[0m
2021-11-25T13:08:23.6642999Z �[36;1m# If no image exists but the hash is the same as the previous hash then we should error out here�[0m
2021-11-25T13:08:23.6643580Z �[36;1mif [[ "${PREVIOUS_DOCKER_TAG}" = "${DOCKER_TAG}" ]]; then�[0m
2021-11-25T13:08:23.6644307Z �[36;1m  echo "ERROR: Something has gone wrong and the previous image isn't available for the merge-base of your branch"�[0m
2021-11-25T13:08:23.6645012Z �[36;1m  echo "       contact the PyTorch team to restore the original images"�[0m
2021-11-25T13:08:23.6645429Z �[36;1m  exit 1�[0m
2021-11-25T13:08:23.6645697Z �[36;1mfi�[0m
2021-11-25T13:08:23.6646053Z �[36;1mecho ::set-output name=rebuild::yes�[0m
2021-11-25T13:08:23.6655459Z shell: /usr/bin/bash -e {0}
2021-11-25T13:08:23.6655767Z env:
2021-11-25T13:08:23.6656527Z   BUILD_ENVIRONMENT: linux-xenial-py3-clang5-mobile-custom-build-static
2021-11-25T13:08:23.6657998Z   DOCKER_IMAGE_BASE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-xenial-py3-clang5-android-ndk-r19c
2021-11-25T13:08:23.6659204Z   SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2
2021-11-25T13:08:23.6660089Z   XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla

linux-xenial-py3.6-clang7-onnx / build (8/12)

Step: "Unknown" (full log | diagnosis details | 🔁 rerun)

2021-11-25T13:09:03.6131035Z �[36;1m echo "ERR...t available for the merge-base of your branch"�[0m

2021-11-25T13:09:03.6124976Z �[36;1mfi�[0m
2021-11-25T13:09:03.6125437Z �[36;1m# Covers the case where a previous tag doesn't exist for the tree�[0m
2021-11-25T13:09:03.6126183Z �[36;1m# this is only really applicable on trees that don't have `.circleci/docker` at its merge base, i.e. nightly�[0m
2021-11-25T13:09:03.6126880Z �[36;1mif ! git rev-parse "$MERGE_BASE:.circleci/docker"; then�[0m
2021-11-25T13:09:03.6127637Z �[36;1m  echo "Directory '.circleci/docker' not found in commit $MERGE_BASE, you should probably rebase onto a more recent commit"�[0m
2021-11-25T13:09:03.6128243Z �[36;1m  exit 1�[0m
2021-11-25T13:09:03.6128516Z �[36;1mfi�[0m
2021-11-25T13:09:03.6128989Z �[36;1mPREVIOUS_DOCKER_TAG=$(git rev-parse "$MERGE_BASE:.circleci/docker")�[0m
2021-11-25T13:09:03.6129691Z �[36;1m# If no image exists but the hash is the same as the previous hash then we should error out here�[0m
2021-11-25T13:09:03.6130327Z �[36;1mif [[ "${PREVIOUS_DOCKER_TAG}" = "${DOCKER_TAG}" ]]; then�[0m
2021-11-25T13:09:03.6131035Z �[36;1m  echo "ERROR: Something has gone wrong and the previous image isn't available for the merge-base of your branch"�[0m
2021-11-25T13:09:03.6131787Z �[36;1m  echo "       contact the PyTorch team to restore the original images"�[0m
2021-11-25T13:09:03.6132331Z �[36;1m  exit 1�[0m
2021-11-25T13:09:03.6132618Z �[36;1mfi�[0m
2021-11-25T13:09:03.6132986Z �[36;1mecho ::set-output name=rebuild::yes�[0m
2021-11-25T13:09:03.6143621Z shell: /usr/bin/bash -e {0}
2021-11-25T13:09:03.6143935Z env:
2021-11-25T13:09:03.6144471Z   BUILD_ENVIRONMENT: linux-xenial-py3.6-clang7-onnx
2021-11-25T13:09:03.6145566Z   DOCKER_IMAGE_BASE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-xenial-py3-clang7-onnx
2021-11-25T13:09:03.6146697Z   SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2
2021-11-25T13:09:03.6147660Z   XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla

linux-xenial-py3.6-gcc7-bazel-test / build-and-test (9/12)

Step: "Unknown" (full log | diagnosis details | 🔁 rerun)

2021-11-25T13:08:50.9854135Z �[36;1m echo "ERR...t available for the merge-base of your branch"�[0m

2021-11-25T13:08:50.9848511Z �[36;1mfi�[0m
2021-11-25T13:08:50.9848934Z �[36;1m# Covers the case where a previous tag doesn't exist for the tree�[0m
2021-11-25T13:08:50.9849616Z �[36;1m# this is only really applicable on trees that don't have `.circleci/docker` at its merge base, i.e. nightly�[0m
2021-11-25T13:08:50.9850345Z �[36;1mif ! git rev-parse "$MERGE_BASE:.circleci/docker"; then�[0m
2021-11-25T13:08:50.9851020Z �[36;1m  echo "Directory '.circleci/docker' not found in commit $MERGE_BASE, you should probably rebase onto a more recent commit"�[0m
2021-11-25T13:08:50.9851577Z �[36;1m  exit 1�[0m
2021-11-25T13:08:50.9851833Z �[36;1mfi�[0m
2021-11-25T13:08:50.9852265Z �[36;1mPREVIOUS_DOCKER_TAG=$(git rev-parse "$MERGE_BASE:.circleci/docker")�[0m
2021-11-25T13:08:50.9852918Z �[36;1m# If no image exists but the hash is the same as the previous hash then we should error out here�[0m
2021-11-25T13:08:50.9853488Z �[36;1mif [[ "${PREVIOUS_DOCKER_TAG}" = "${DOCKER_TAG}" ]]; then�[0m
2021-11-25T13:08:50.9854135Z �[36;1m  echo "ERROR: Something has gone wrong and the previous image isn't available for the merge-base of your branch"�[0m
2021-11-25T13:08:50.9854842Z �[36;1m  echo "       contact the PyTorch team to restore the original images"�[0m
2021-11-25T13:08:50.9855262Z �[36;1m  exit 1�[0m
2021-11-25T13:08:50.9855529Z �[36;1mfi�[0m
2021-11-25T13:08:50.9855867Z �[36;1mecho ::set-output name=rebuild::yes�[0m
2021-11-25T13:08:50.9865629Z shell: /usr/bin/bash -e {0}
2021-11-25T13:08:50.9865932Z env:
2021-11-25T13:08:50.9866468Z   BUILD_ENVIRONMENT: linux-xenial-py3.6-gcc7-bazel-test
2021-11-25T13:08:50.9867651Z   DOCKER_IMAGE_BASE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda10.2-cudnn7-py3.9-gcc7
2021-11-25T13:08:50.9868808Z   SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2
2021-11-25T13:08:50.9869677Z   XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla

linux-xenial-py3.6-clang7-asan / build (10/12)

Step: "Unknown" (full log | diagnosis details | 🔁 rerun)

2021-11-25T13:09:17.7450681Z �[36;1m echo "ERR...t available for the merge-base of your branch"�[0m

2021-11-25T13:09:17.7444793Z �[36;1mfi�[0m
2021-11-25T13:09:17.7445247Z �[36;1m# Covers the case where a previous tag doesn't exist for the tree�[0m
2021-11-25T13:09:17.7445961Z �[36;1m# this is only really applicable on trees that don't have `.circleci/docker` at its merge base, i.e. nightly�[0m
2021-11-25T13:09:17.7446649Z �[36;1mif ! git rev-parse "$MERGE_BASE:.circleci/docker"; then�[0m
2021-11-25T13:09:17.7447385Z �[36;1m  echo "Directory '.circleci/docker' not found in commit $MERGE_BASE, you should probably rebase onto a more recent commit"�[0m
2021-11-25T13:09:17.7447964Z �[36;1m  exit 1�[0m
2021-11-25T13:09:17.7448253Z �[36;1mfi�[0m
2021-11-25T13:09:17.7448700Z �[36;1mPREVIOUS_DOCKER_TAG=$(git rev-parse "$MERGE_BASE:.circleci/docker")�[0m
2021-11-25T13:09:17.7449388Z �[36;1m# If no image exists but the hash is the same as the previous hash then we should error out here�[0m
2021-11-25T13:09:17.7450005Z �[36;1mif [[ "${PREVIOUS_DOCKER_TAG}" = "${DOCKER_TAG}" ]]; then�[0m
2021-11-25T13:09:17.7450681Z �[36;1m  echo "ERROR: Something has gone wrong and the previous image isn't available for the merge-base of your branch"�[0m
2021-11-25T13:09:17.7451517Z �[36;1m  echo "       contact the PyTorch team to restore the original images"�[0m
2021-11-25T13:09:17.7451962Z �[36;1m  exit 1�[0m
2021-11-25T13:09:17.7452249Z �[36;1mfi�[0m
2021-11-25T13:09:17.7452628Z �[36;1mecho ::set-output name=rebuild::yes�[0m
2021-11-25T13:09:17.7463606Z shell: /usr/bin/bash -e {0}
2021-11-25T13:09:17.7463940Z env:
2021-11-25T13:09:17.7464457Z   BUILD_ENVIRONMENT: linux-xenial-py3.6-clang7-asan
2021-11-25T13:09:17.7465528Z   DOCKER_IMAGE_BASE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-xenial-py3-clang7-asan
2021-11-25T13:09:17.7466616Z   SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2
2021-11-25T13:09:17.7467547Z   XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla

linux-vulkan-bionic-py3.6-clang9 / build (11/12)

Step: "Unknown" (full log | diagnosis details | 🔁 rerun)

2021-11-25T13:09:14.3706908Z �[36;1m echo "ERR...t available for the merge-base of your branch"�[0m

2021-11-25T13:09:14.3700975Z �[36;1mfi�[0m
2021-11-25T13:09:14.3701424Z �[36;1m# Covers the case where a previous tag doesn't exist for the tree�[0m
2021-11-25T13:09:14.3702152Z �[36;1m# this is only really applicable on trees that don't have `.circleci/docker` at its merge base, i.e. nightly�[0m
2021-11-25T13:09:14.3702844Z �[36;1mif ! git rev-parse "$MERGE_BASE:.circleci/docker"; then�[0m
2021-11-25T13:09:14.3703573Z �[36;1m  echo "Directory '.circleci/docker' not found in commit $MERGE_BASE, you should probably rebase onto a more recent commit"�[0m
2021-11-25T13:09:14.3704173Z �[36;1m  exit 1�[0m
2021-11-25T13:09:14.3704439Z �[36;1mfi�[0m
2021-11-25T13:09:14.3704899Z �[36;1mPREVIOUS_DOCKER_TAG=$(git rev-parse "$MERGE_BASE:.circleci/docker")�[0m
2021-11-25T13:09:14.3705599Z �[36;1m# If no image exists but the hash is the same as the previous hash then we should error out here�[0m
2021-11-25T13:09:14.3706214Z �[36;1mif [[ "${PREVIOUS_DOCKER_TAG}" = "${DOCKER_TAG}" ]]; then�[0m
2021-11-25T13:09:14.3706908Z �[36;1m  echo "ERROR: Something has gone wrong and the previous image isn't available for the merge-base of your branch"�[0m
2021-11-25T13:09:14.3707758Z �[36;1m  echo "       contact the PyTorch team to restore the original images"�[0m
2021-11-25T13:09:14.3708205Z �[36;1m  exit 1�[0m
2021-11-25T13:09:14.3708488Z �[36;1mfi�[0m
2021-11-25T13:09:14.3708850Z �[36;1mecho ::set-output name=rebuild::yes�[0m
2021-11-25T13:09:14.3718550Z shell: /usr/bin/bash -e {0}
2021-11-25T13:09:14.3718861Z env:
2021-11-25T13:09:14.3719417Z   BUILD_ENVIRONMENT: linux-vulkan-bionic-py3.6-clang9
2021-11-25T13:09:14.3720474Z   DOCKER_IMAGE_BASE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-py3.6-clang9
2021-11-25T13:09:14.3721541Z   SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2
2021-11-25T13:09:14.3722482Z   XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla

Lint / quick-checks (12/12)

Step: "Ensure no trailing spaces" (full log | diagnosis details | 🔁 rerun)

2021-11-25T13:06:24.6556937Z python: can't open..._launches.py': [Errno 2] No such file or directory

2021-11-25T13:06:24.6233628Z ##[group]Run set -eux
2021-11-25T13:06:24.6234071Z �[36;1mset -eux�[0m
2021-11-25T13:06:24.6234787Z �[36;1mpython torch/testing/_check_kernel_launches.py |& tee "${GITHUB_WORKSPACE}"/cuda_kernel_launch_checks.txt�[0m
2021-11-25T13:06:24.6270891Z shell: /bin/bash -e {0}
2021-11-25T13:06:24.6271272Z env:
2021-11-25T13:06:24.6271824Z   pythonLocation: /opt/hostedtoolcache/Python/3.10.0/x64
2021-11-25T13:06:24.6272533Z   LD_LIBRARY_PATH: /opt/hostedtoolcache/Python/3.10.0/x64/lib
2021-11-25T13:06:24.6273065Z ##[endgroup]
2021-11-25T13:06:24.6377120Z + python torch/testing/_check_kernel_launches.py
2021-11-25T13:06:24.6381379Z + tee /home/runner/work/pytorch/pytorch/cuda_kernel_launch_checks.txt
2021-11-25T13:06:24.6556937Z python: can't open file '/home/runner/work/pytorch/pytorch/torch/testing/_check_kernel_launches.py': [Errno 2] No such file or directory
2021-11-25T13:06:24.6621132Z ##[group]Run (! git --no-pager grep -I -no $'#include <cub/' --  ./aten  ':(exclude)aten/src/ATen/cuda/cub*.cuh' || (echo "The above files have direct cub include; please include ATen/cuda/cub.cuh instead and wrap your cub calls in at::native namespace if necessary"; false))
2021-11-25T13:06:24.6623039Z �[36;1m(! git --no-pager grep -I -no $'#include <cub/' --  ./aten  ':(exclude)aten/src/ATen/cuda/cub*.cuh' || (echo "The above files have direct cub include; please include ATen/cuda/cub.cuh instead and wrap your cub calls in at::native namespace if necessary"; false))�[0m
2021-11-25T13:06:24.6656197Z shell: /bin/bash -e {0}
2021-11-25T13:06:24.6656552Z env:
2021-11-25T13:06:24.6657337Z   pythonLocation: /opt/hostedtoolcache/Python/3.10.0/x64
2021-11-25T13:06:24.6658064Z   LD_LIBRARY_PATH: /opt/hostedtoolcache/Python/3.10.0/x64/lib
2021-11-25T13:06:24.6658591Z ##[endgroup]
2021-11-25T13:06:24.6984372Z ##[group]Run (! git --no-pager grep -I -no $'cudaStreamSynchronize' --  ./aten ./c10 ':(exclude)aten/src/ATen/test' ':(exclude)c10/cuda/CUDAFunctions.h' || (echo "The above files call raw cuda APIs directly; please use at::cuda wrappers instead"; false))
2021-11-25T13:06:24.6986390Z �[36;1m(! git --no-pager grep -I -no $'cudaStreamSynchronize' --  ./aten ./c10 ':(exclude)aten/src/ATen/test' ':(exclude)c10/cuda/CUDAFunctions.h' || (echo "The above files call raw cuda APIs directly; please use at::cuda wrappers instead"; false))�[0m
2021-11-25T13:06:24.7019619Z shell: /bin/bash -e {0}

3 failures not recognized by patterns:

Job	Step	Action
^{win-vs2019-cuda11.3-py3 / build}	^{Cleanup build-results and workspaces}	🔁 rerun
^{Lint / clang-tidy}	^Unknown	🔁 rerun
^{win-vs2019-cpu-py3 / build}	^Unknown	🔁 rerun

❄️ 1 failure tentatively classified as flaky

but reruns have not yet been triggered to confirm:

pytorch-linux-xenial-py3-clang5-android-ndk-r19c-gradle-build-x86_32 (1/1)

Step: "Checkout code" (full log | diagnosis details | 🔁 rerun) ❄️

fatal: Could not read from remote repository.

Using SSH Config Dir '/home/circleci/.ssh'
git version 2.31.1
Cloning git repository
Cloning into '.'...
ssh: connect to host github.com port 22: Connection timed out

fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.


exit status 128

This comment was automatically generated by Dr. CI (expand for details).

Please report bugs/suggestions to the (internal) Dr. CI Users group.

Click here to manually regenerate this comment.

Update fx.rst

Summary: [FX][docs] Document gotcha about training flag Test Plan: make html Reviewers: jamesr66a Subscribers: jamesr66a Tasks: Tags: Committer: jubinchheda <jubinchheda@fb.com>

Test Plan: make html and view differences Reviewers: jamesr66a Subscribers: jamesr66a Tasks: Tags: Committer: jubinchheda jubinchheda@fb.com

Summary: Fixes #68913 Pull Request resolved: #68915 Reviewed By: jamesr66a Differential Revision: D32705410 Pulled By: jubinchheda fbshipit-source-id: a44c17ab0e62465823ceb0ef983ae330b50fb073

[FX][docs] Document gotcha about training flag pytorch#68190

e43b7d2

Summary: [FX][docs] Document gotcha about training flag pytorch#68190 Test Plan: make html and view differences Reviewers: jamesr66a Subscribers: jamesr66a Tasks: Tags: Committer: jubinchheda <jubinchheda@fb.com>

pytorch-probot bot added the ciflow/default label Nov 25, 2021

facebook-github-bot added the cla signed label Nov 25, 2021

jubinchheda and others added 5 commits November 25, 2021 07:33

Update fx.rst

8f1acd5

Merge pull request #1 from jubinchheda/jubinchheda-patch-1

510dae1

Update fx.rst

[FX][docs] Document gotcha about training flag

a28fdd6

Summary: [FX][docs] Document gotcha about training flag Test Plan: make html Reviewers: jamesr66a Subscribers: jamesr66a Tasks: Tags: Committer: jubinchheda <jubinchheda@fb.com>

[FX][docs] Document gotcha about training flag

55b6b57

Summary: [FX][docs] Document gotcha about training flag Test Plan: make html Reviewers: jamesr66a Subscribers: jamesr66a Tasks: Tags: Committer: jubinchheda <jubinchheda@fb.com>

Summary: [FX][docs] Document gotcha about training flag pytorch#68190

35e3282

Test Plan: make html and view differences Reviewers: jamesr66a Subscribers: jamesr66a Tasks: Tags: Committer: jubinchheda jubinchheda@fb.com

jubinchheda closed this Nov 25, 2021

jubinchheda mentioned this pull request Nov 25, 2021

[FX][docs] Document gotcha about training flag #68915

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[FX][docs] Document gotcha about training flag #68190 #68913

[FX][docs] Document gotcha about training flag #68190 #68913

Uh oh!

jubinchheda commented Nov 25, 2021

Uh oh!

pytorch-probot bot commented Nov 25, 2021 •

edited

Loading

⚛️ CI Flow

Uh oh!

facebook-github-bot commented Nov 25, 2021 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

[FX][docs] Document gotcha about training flag #68190 #68913

[FX][docs] Document gotcha about training flag #68190 #68913

Uh oh!

Conversation

jubinchheda commented Nov 25, 2021

Uh oh!

pytorch-probot bot commented Nov 25, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

⚛️ CI Flow

Uh oh!

facebook-github-bot commented Nov 25, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful links

💊 CI failures summary and remediations

🕵️ 12 new failures recognized by patterns

pytorch-linux-xenial-py3-clang5-android-ndk-r19c-gradle-custom-build-single-full-jit / build-and-test (1/12)

pytorch-linux-xenial-py3-clang5-android-ndk-r19c-gradle-custom-build-single / build-and-test (2/12)

linux-bionic-py3.6-clang9 / build (3/12)

linux-xenial-py3.6-gcc5.4 / build (4/12)

linux-xenial-cuda11.3-py3.6-gcc7 / build (5/12)

linux-xenial-py3-clang5-mobile-build / build (6/12)

linux-xenial-py3-clang5-mobile-custom-build-static / build (7/12)

linux-xenial-py3.6-clang7-onnx / build (8/12)

linux-xenial-py3.6-gcc7-bazel-test / build-and-test (9/12)

linux-xenial-py3.6-clang7-asan / build (10/12)

linux-vulkan-bionic-py3.6-clang9 / build (11/12)

Lint / quick-checks (12/12)

3 failures not recognized by patterns:

❄️ 1 failure tentatively classified as flaky

pytorch-linux-xenial-py3-clang5-android-ndk-r19c-gradle-build-x86_32 (1/1)

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

pytorch-probot bot commented Nov 25, 2021 •

edited

Loading

facebook-github-bot commented Nov 25, 2021 •

edited

Loading