Skip to content
This repository has been archived by the owner on Nov 17, 2023. It is now read-only.

test_gluon_data.test_dataloader_context broken on g4 #17961

Closed
leezu opened this issue Apr 2, 2020 · 0 comments
Closed

test_gluon_data.test_dataloader_context broken on g4 #17961

leezu opened this issue Apr 2, 2020 · 0 comments
Assignees
Labels

Comments

@leezu
Copy link
Contributor

leezu commented Apr 2, 2020

Description

test_gluon_data.test_dataloader_context broken on g4

Error Message

======================================================================
ERROR: test suite for <module 'test_gluon_data' from 'C:\\jenkins_slave\\workspace\\ut-python-gpu\\tests\\python\\unittest\\test_gluon_data.py'>
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Python37\lib\site-packages\nose\suite.py", line 229, in run
    self.tearDown()
  File "C:\Python37\lib\site-packages\nose\suite.py", line 352, in tearDown
    self.teardownContext(ancestor)
  File "C:\Python37\lib\site-packages\nose\suite.py", line 368, in teardownContext
    try_run(context, names)
  File "C:\Python37\lib\site-packages\nose\util.py", line 471, in try_run
    return func()
  File "C:\jenkins_slave\workspace\ut-python-gpu\tests\python\unittest\common.py", line 310, in teardown
    mx.nd.waitall()
  File "c:\jenkins_slave\workspace\ut-python-gpu\python\mxnet\ndarray\ndarray.py", line 211, in waitall
    check_call(_LIB.MXNDArrayWaitAll())
  File "c:\jenkins_slave\workspace\ut-python-gpu\python\mxnet\base.py", line 246, in check_call
    raise get_last_ffi_error()
mxnet.base.MXNetError: Traceback (most recent call last):
  File "C:\jenkins_slave\workspace\build-gpu\src\common\../common/cuda_utils.h", line 414
CUDA: Check failed: e == cudaSuccess || e == cudaErrorCudartUnloading: invalid device ordinal

----------------------------------------------------------------------
Ran 7 tests in 1

http://jenkins.mxnet-ci.amazon-ml.com/blue/rest/organizations/jenkins/pipelines/mxnet-validation/pipelines/windows-gpu/branches/PR-17808/runs/59/nodes/108/steps/154/log/?start=0

To Reproduce

Steps to reproduce

(Paste the commands you ran that produced the error.)

  1. On g4 instance with Windows: python -m nose --verbose .\tests\python\unittest\test_gluon_data.py

What have you tried to solve it?

Run all other tests but the broken tests works:

python -m nose --verbose .\tests\python\unittest\test_gluon_data.py -m "test_array_dataset|test_recordimage_dataset|test_recordimage_dataset_with_data_loader_multiworker|test_sampler|test_datasets|test_image_folder_dataset|test_list_dataset|test_multi_worker|test_multi_worker_shape|test_multi_worker_forked_data_loader|test_multi_worker_dataloader_release_pool|test_dataset_filter|test_dataset_shard|test_dataset_take"

@leezu leezu added the Bug label Apr 2, 2020
leezu added a commit to vexilligera/incubator-mxnet that referenced this issue Apr 2, 2020
@leezu leezu closed this as completed Apr 4, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Projects
None yet
Development

No branches or pull requests

2 participants