[BE][CI] migrate .jenkins to .ci #92569

janeyx99 · 2023-01-18T19:39:32Z

PLEASE LOOK AT #92621 INSTEAD

Completes what I started in #82753

Most the changes here are:

Copying .jenkins files to .ci
Replacing .jenkins references with .ci

Once this lands, we'll likely have to tell people to rebase past this change.

Next steps:

Remove .jenkins
Remove old things in .ci

Why don't I remove .jenkins already?

Removing it makes the bc test not pass and starts a rabbit hole of problems as can be seen in the first PR attempt.
There may be other dependencies from other repos to .jenkins and we may want a buffer time/not have this PR grow in scope unnecessarily.

pytorch-bot · 2023-01-18T19:39:35Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/92569

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

❌ 5 Failures

As of commit d0c28b7:

NEW FAILURES - The following jobs have failed:

BROKEN TRUNK - The following jobs failed but were present on the merge base 0bc875a:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

janeyx99 · 2023-01-18T20:30:00Z

.ci/pytorch/test.sh

+  COMMON_BUILD_SCRIPT="$(dirname "${BASH_SOURCE[0]}")/common-build.sh"
+  if [[ -f "$COMMON_BUILD_SCRIPT" ]]; then
+    # shellcheck source=./common-build.sh
+    bash "$COMMON_BUILD_SCRIPT"
+  else
+    # shellcheck source=./common-build.sh
+    bash .jenkins/pytorch/common-build.sh
+  fi


i changed only these manually.

all other changes are mostly a .jenkins -> .ci

huydhn

LGTM! Should we look around for a solution to keep .jenkins and .ci in sync in the interim before the former can be removed?

janeyx99 · 2023-01-18T21:55:51Z

LGTM! Should we look around for a solution to keep .jenkins and .ci in sync in the interim before the former can be removed?

Maybe? But it may not be very worth it since I intend for that interim to be small (a week at most).

Another thing we could do to completely eliminate this gap is to include the .jenkins removal here, and force merge the bc test breaking as an intentional break.

huydhn · 2023-01-18T22:05:45Z

Maybe? But it may not be very worth it since I intend for that interim to be small (a week at most).

Another thing we could do to completely eliminate this gap is to include the .jenkins removal here, and force merge the bc test breaking as an intentional break.

That sounds good to me. I like your approach here better with .jenkins removal done in a separate PR.

ZainRizvi

There's a bunch of jenkins references still existing in the copied files. Are those supposed to still be there?

ZainRizvi · 2023-01-19T16:36:06Z

.ci/caffe2/README.md

@@ -0,0 +1,14 @@
+# Jenkins


Want to rename the jenkins references in this file?

ZainRizvi · 2023-01-19T16:37:04Z

.ci/caffe2/test.sh

+  # This test has been flaky in ROCm CI (but note the tests are
+  # cpu-only so should be unrelated to ROCm)
+  rocm_ignore_test+=("--ignore $caffe2_pypath/python/operator_test/blobs_queue_db_test.py")
+  # This test is skipped on Jenkins(compiled without MKL) and otherwise known flaky


more jenkins refs

ZainRizvi · 2023-01-19T16:37:28Z

.ci/caffe2/test.sh

+
+if [[ ${BUILD_ENVIRONMENT} == *onnx* ]]; then
+  pip install click mock tabulate networkx==2.0
+  pip -q install --user "file:///var/lib/jenkins/workspace/third_party/onnx#egg=onnx"


This prob needs to go too...

ZainRizvi · 2023-01-19T16:37:39Z

.ci/onnx/README.md

@@ -0,0 +1,14 @@
+# Jenkins


ZainRizvi · 2023-01-19T16:38:01Z

.ci/onnx/test.sh

+  # We don't actually need it for our tests, but it's imported if it's present, so uninstall.
+  pip uninstall -q --yes numba
+  # JIT C++ extensions require ninja, so put it into PATH.
+  export PATH="/var/lib/jenkins/.local/bin:$PATH"


And this I'm guessing

malfet

Please consider creating symlinks and deleting them later to make transition process smoother.

janeyx99 · 2023-01-19T18:09:02Z

Closing in favor of #92621.

@ZainRizvi I think it'd be better as a followup to replace the Jenkins refs (since those are mostly doc changes)

Follow up after #92569

Follow up after #92569 Pull Request resolved: #93104 Approved by: https://github.com/huydhn, https://github.com/seemethere, https://github.com/ZainRizvi

pytorch-bot bot added the release notes: onnx torch.onnx related changes that should show up in the release notes label Jan 18, 2023

janeyx99 force-pushed the copy-.jenkins branch 2 times, most recently from e509bf6 to 30b92f6 Compare January 18, 2023 20:27

janeyx99 commented Jan 18, 2023

View reviewed changes

janeyx99 marked this pull request as ready for review January 18, 2023 21:14

janeyx99 requested a review from a team as a code owner January 18, 2023 21:14

janeyx99 added topic: not user facing topic category and removed release notes: onnx torch.onnx related changes that should show up in the release notes labels Jan 18, 2023

huydhn added the skip-pr-sanity-checks label Jan 18, 2023

huydhn approved these changes Jan 18, 2023

View reviewed changes

janeyx99 added ciflow/trunk Trigger trunk jobs on your pull request ciflow/mps Run MPS tests (subset of trunk) and removed skip-pr-sanity-checks labels Jan 18, 2023

janeyx99 mentioned this pull request Jan 18, 2023

[WIP][BE][CI] Remove .jenkins #92591

Closed

janeyx99 force-pushed the copy-.jenkins branch from 999ec47 to d47ab50 Compare January 19, 2023 16:12

janeyx99 added 5 commits January 19, 2023 16:26

[BE][CI] migrate .jenkins to .ci

dfa0949

typo

89e01eb

add if to use .jenkins

edce0c6

remove untracked file

bdf60f5

update according to latest master

d0c28b7

janeyx99 force-pushed the copy-.jenkins branch from d47ab50 to d0c28b7 Compare January 19, 2023 16:27

ZainRizvi reviewed Jan 19, 2023

View reviewed changes

malfet approved these changes Jan 19, 2023

View reviewed changes

janeyx99 closed this Jan 19, 2023

malfet mentioned this pull request Jan 26, 2023

[CI][BE] Move docker forlder to .ci #93104

Closed

malfet added a commit that referenced this pull request Jan 26, 2023

[CI][BE] Move docker forlder to .ci

2a6f714

Follow up after #92569

malfet added a commit that referenced this pull request Jan 27, 2023

[CI][BE] Move docker forlder to .ci

733eda8

Follow up after #92569

malfet added a commit that referenced this pull request Jan 31, 2023

[CI][BE] Move docker forlder to .ci

8ede83a

Follow up after #92569

malfet added a commit that referenced this pull request Feb 3, 2023

[CI][BE] Move docker forlder to .ci

ff4c887

Follow up after #92569

pytorchmergebot pushed a commit that referenced this pull request Feb 3, 2023

[CI][BE] Move docker forlder to .ci (#93104)

6c4dc98

Follow up after #92569 Pull Request resolved: #93104 Approved by: https://github.com/huydhn, https://github.com/seemethere, https://github.com/ZainRizvi

github-actions bot deleted the copy-.jenkins branch July 21, 2024 01:59

Uh oh!

[BE][CI] migrate .jenkins to .ci #92569

[BE][CI] migrate .jenkins to .ci #92569

Uh oh!

Conversation

janeyx99 commented Jan 18, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PLEASE LOOK AT #92621 INSTEAD

Uh oh!

pytorch-bot bot commented Jan 18, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/92569

❌ 5 Failures

Uh oh!

janeyx99 Jan 18, 2023

Choose a reason for hiding this comment

Uh oh!

huydhn left a comment

Choose a reason for hiding this comment

Uh oh!

janeyx99 commented Jan 18, 2023

Uh oh!

huydhn commented Jan 18, 2023

Uh oh!

ZainRizvi left a comment

Choose a reason for hiding this comment

Uh oh!

ZainRizvi Jan 19, 2023

Choose a reason for hiding this comment

Uh oh!

ZainRizvi Jan 19, 2023

Choose a reason for hiding this comment

Uh oh!

ZainRizvi Jan 19, 2023

Choose a reason for hiding this comment

Uh oh!

ZainRizvi Jan 19, 2023

Choose a reason for hiding this comment

Uh oh!

ZainRizvi Jan 19, 2023

Choose a reason for hiding this comment

Uh oh!

malfet left a comment

Choose a reason for hiding this comment

Uh oh!

janeyx99 commented Jan 19, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

janeyx99 commented Jan 18, 2023 •

edited

Loading

pytorch-bot bot commented Jan 18, 2023 •

edited

Loading