-
Notifications
You must be signed in to change notification settings - Fork 25.6k
Merge job-spec env variables of Pytorch/Caffe2 CI jobs #16649
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
aa3ad7a
to
37956ab
Compare
I think what we want is for https://github.com/pytorch/pytorch/blob/master/.jenkins/pytorch/enabled-configs.txt#L8-L52 to exactly correspond with (the corresponding BUILD_ENVIRONMENTs of) https://github.com/pytorch/pytorch/blob/master/.circleci/config.yml#L2940-L3023 and https://github.com/pytorch/ossci-job-dsl/blob/master/src/main/groovy/ossci/caffe2/Images.groovy#L239-L298 (the remaining jenkins builds). |
…iables of Pytorch/Caffe2 CI jobs). The idea is to unify the environment variables JOB_BASE_NAME and BUILD_ENVIRONMENT which controlled the Pytorch and Caffe2 jobs respectively. In this commit, We have converted all the JOB_BASE_NAME references in .jenkins/pytorch/* files to BUILD_ENVIRONMENT. Then, did the same in .circleci/config.yml. One thing that we needed to be careful was when both BUILD_NAME and JOB_BASE_NAME were present under same declaration in config.yml file (e.g. for "caffe2-" stuffs). To ensure that all "==" checks work as expected, we also had to add "*" in some if conditions in .jenkins/caffe2/build.sh file. Finally, removed "-build", "-test", etc. suffixes from COMPACT_JOB_NAME variable assignment in the bash scriupt files in .jenkins/pytorch folder, e.g., modify COMPACT_JOB_NAME="${BUILD_ENVIRONMENT}-build" to COMPACT_JOB_NAME="${BUILD_ENVIRONMENT}".
Incorporated the requested changes into enabled-config.txt file. Now the job names enabled in the text file matches with the corresponding BUILD_ENVIRONMENTs and remaining jenkins builds. |
test_custom_script_ops | ||
else | ||
if [[ "${JOB_BASE_NAME}" == *-test1 ]]; then | ||
if [[ "${BUILD_ENVIRONMENT}" == *-test1 ]]; then |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This looks like dead code to me. @yf225 did there used to be more mac test jobs?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We used to split the test job in half for parallel testing which is what this was for. We can probably keep them around in case we are doing it in the future (which would be a pretty nice CI speed improvement)
fi | ||
else | ||
if [[ "${JOB_BASE_NAME}" == *-test1 ]]; then | ||
if [[ "${BUILD_ENVIRONMENT}" == *-test1 ]]; then |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This looks dead as well
$TMP_DIR/ci_scripts/test_libtorch.bat | ||
else | ||
if [[ "${JOB_BASE_NAME}" == *-test1 ]]; then | ||
if [[ "${BUILD_ENVIRONMENT}" == *-test1 ]]; then |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Here as well
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I believe the Windows tests are no longer running at all after this change.
- Here's an example of
pytorch-win-ws2016-cuda9-cudnn7-py3-test1
that ran fully: https://ci.pytorch.org/jenkins/job/pytorch-builds/job/pytorch-win-ws2016-cuda9-cudnn7-py3-test1/22629/console - Here's the latest
pytorch-win-ws2016-cuda9-cudnn7-py3-test1
run on master: https://ci.pytorch.org/jenkins/job/pytorch-builds/job/pytorch-win-ws2016-cuda9-cudnn7-py3-test1/22869//console
Note that the value of ${JOB_BASE_NAME}
used to be pytorch-win-ws2016-cuda9-cudnn7-py3-test1
, but now ${BUILD_ENVIRONMENT}
is just pytorch-win-ws2016-cuda9-cudnn7-py3
(without the -test1
suffix), so none of the conditions in the run_tests()
function are matched.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Filed Issue #17101
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@mmh683 has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@mmh683 has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@mmh683 is landing this pull request. If you are a Facebook employee, you can view this diff on Phabricator.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@mmh683 has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@mmh683 is landing this pull request. If you are a Facebook employee, you can view this diff on Phabricator.
The idea is to unify the environment variables
JOB_BASE_NAME
andBUILD_ENVIRONMENT
which controlled the Pytorch and Caffe2 jobs respectively. In this commit, we have converted all theJOB_BASE_NAME
references in .jenkins/pytorch/* files toBUILD_ENVIRONMENT
. Then, did the same thing in .circleci/config.yml. One thing that we needed to be careful was when bothBUILD_ENVIRONMENT
andJOB_BASE_NAME
were present under same declaration in config.yml file (e.g., for "caffe2-" stuffs). To ensure that all "==" checks work as expected, we also had to add "*" in some if conditions in .jenkins/caffe2/build.sh file. Finally, removed "-build", "-test", etc. suffixes fromCOMPACT_JOB_NAME
variable assignment in the bash script files in .jenkins/pytorch folder, e.g., modifyCOMPACT_JOB_NAME="${BUILD_ENVIRONMENT}-build"
toCOMPACT_JOB_NAME="${BUILD_ENVIRONMENT}"
.