Skip to content
This repository has been archived by the owner on Nov 17, 2023. It is now read-only.

[CI][NightlyTestsForBinaries] nightly test failed on new AMP tutorial #15028

Closed
roywei opened this issue May 21, 2019 · 5 comments
Closed

[CI][NightlyTestsForBinaries] nightly test failed on new AMP tutorial #15028

roywei opened this issue May 21, 2019 · 5 comments

Comments

@roywei
Copy link
Member

roywei commented May 21, 2019

After AMP PR #14173 added an AMP tutorial, but the nightly test on it is failing:
http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/NightlyTestsForBinaries/detail/master/320/pipeline

Seems the tutorial depends on gluoncv, we need to add it in CI environment here:
https://github.com/apache/incubator-mxnet/blob/master/ci/docker/install/ubuntu_tutorials.sh

@mxnet-label-bot
Copy link
Contributor

Hey, this is the MXNet Label Bot.
Thank you for submitting the issue! I will try and suggest some labels so that the appropriate MXNet community members can help resolve it.
Here are my recommended labels: Test

@roywei
Copy link
Member Author

roywei commented May 21, 2019

@mxnet-label-bot add[Test, CI]

@roywei roywei changed the title [nightly test] nightly test failed on new AMP tutorial [CI][NightlyTestsForBinaries] nightly test failed on new AMP tutorial May 21, 2019
@Chancebair
Copy link
Contributor

Taking a look

@roywei
Copy link
Member Author

roywei commented Jun 4, 2019

Now no more warnings, but nightly still fails with CUDA malloc OOM.
http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/NightlyTestsForBinaries/detail/master/333/pipeline/148

This is not observed in AWS P3 instances.

@roywei
Copy link
Member Author

roywei commented Jun 6, 2019

fixed in #15141

@roywei roywei closed this as completed Jun 6, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

4 participants