Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Try distributed run for 3 times to prevent failure #5305

Merged
merged 7 commits into from
Jun 24, 2021
Merged

Try distributed run for 3 times to prevent failure #5305

Changes from 5 commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
6cd01ab
Don't build test by default
jackalcooper Jun 24, 2021
ba36e11
refine
jackalcooper Jun 24, 2021
f0f371a
Retry distributed run for 3 times to prevent failure
jackalcooper Jun 24, 2021
f4734b9
refine
jackalcooper Jun 24, 2021
45de1b1
revert changes unwanted
jackalcooper Jun 24, 2021
4e609f9
address review
jackalcooper Jun 24, 2021
ed82ca9
Merge branch 'master' into run_distributed_test_3_times
oneflow-ci-bot Jun 24, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 25 additions & 7 deletions .github/workflows/test.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -283,7 +283,7 @@ jobs:
"xla",
"cuda_op_eager",
"cpu_op_eager",
"xla_cpu"
"xla_cpu",
]
include:
- test_suite: "cuda"
Expand Down Expand Up @@ -394,14 +394,32 @@ jobs:
docker run $extra_docker_args \
${image_name} \
bash -c "bash ci/test/try_install.sh && bash ci/test/build_docs.sh"
- name: Op test (distributed)
- name: Op test (distributed, 1st try)
if: matrix.test_suite == 'cuda'
continue-on-error: true
id: distributed_try_1
run: |
python3 ci/test/distributed_run.py --bash_script=ci/test/2node_op_test.sh --custom_img_tag=${{ env.image_name }} --oneflow_wheel_path=${{ env.wheelhouse_dir }}
- name: Op test (distributed, 2nd try)
if: matrix.test_suite == 'cuda' && steps.distributed_try_1.outcome=='failure'
continue-on-error: true
id: distributed_try_2
run: |
python3 ci/test/distributed_run.py --bash_script=ci/test/2node_op_test.sh --custom_img_tag=${{ env.image_name }} --oneflow_wheel_path=${{ env.wheelhouse_dir }}
- name: Op test (distributed, 3rd try)
if: matrix.test_suite == 'cuda' && steps.distributed_try_2.outcome=='failure'
continue-on-error: true
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这个设置成 false 是不是就可以省掉下面的判断

id: distributed_try_3
run: |
python3 ci/test/distributed_run.py --bash_script=ci/test/2node_op_test.sh --custom_img_tag=${{ env.image_name }} --oneflow_wheel_path=${{ env.wheelhouse_dir }}
- name: Op test (distributed, check success)
if: always()
run: |
set -x
python3 ci/test/distributed_run.py \
--bash_script=ci/test/2node_op_test.sh \
--custom_img_tag=${image_name} \
--oneflow_wheel_path=${wheelhouse_dir}
if ${{ steps.distributed_try_1.outcome=='success' || steps.distributed_try_2.outcome=='success' || steps.distributed_try_3.outcome=='success' }}; then
echo success
else
exit 1
fi
- name: Upload log (distributed test)
if: failure() && matrix.test_suite == 'cuda'
uses: ./.github/actions/upload_oss
Expand Down