Skip to content
This repository has been archived by the owner on Nov 17, 2023. It is now read-only.

Flaky Test test_operator:test_custom_op_fork hanging #17467

Open
rondogency opened this issue Jan 29, 2020 · 3 comments
Open

Flaky Test test_operator:test_custom_op_fork hanging #17467

rondogency opened this issue Jan 29, 2020 · 3 comments
Labels

Comments

@rondogency
Copy link
Contributor

Description

test_operator.py - test_custom_op_fork

Python custom operator test hanging causing some CI to timeout

Occurrences

http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/mxnet-validation%2Funix-cpu/detail/PR-17270/16/pipeline/298

http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/mxnet-validation%2Funix-cpu/detail/PR-17241/19/pipeline/294

Note that the above PR are C++ custom operator and has no connection with Python custom op

What have you tried to solve it?

  1. Tried "python tools/flakiness_checker.py test_operator.test_custom_op_fork -n 100" on a EC2 and the test will intermittently hang

  2. It may be related to the deadlock issue fixed last year mx.nd.Custom not working in subprocess #14396

@anirudh2290
Copy link
Member

@rondogency I was also able to reproduce the hang. Can you disable the test, to unblock CI for now.

@rondogency
Copy link
Contributor Author

@anirudh2290 PR is here #17481

@samskalicky
Copy link
Contributor

@Zha0q1 FYI

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Projects
None yet
Development

No branches or pull requests

3 participants