Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

run validate.py has a error: failed to allocate xxx from device: CUDA_ERROR_OUT_OF_MEMORY #25

Open
Depblu opened this issue Dec 3, 2019 · 1 comment
Open

run validate.py has a error: failed to allocate xxx from device: CUDA_ERROR_OUT_OF_MEMORY #25

Depblu opened this issue Dec 3, 2019 · 1 comment

Comments

@Depblu
Copy link

Depblu commented Dec 3, 2019

log info:
python validate.py --n_proc 2 --exp_dir ../../data/experiments/NGSIM-gail/ --params_filename itr_1000.npz --random_seed 42
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcublas.so.8.0 locally
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcudnn.so.5 locally
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcufft.so.8.0 locally
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcuda.so.1 locally
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcurand.so.8.0 locally
50 vehicles with H = 200

W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE3 instructions, but these are available on your machine and could speed up CPU computations.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE3 instructions, but these are available on your machine and could speed up CPU computations.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations.
I tensorflow/core/common_runtime/gpu/gpu_device.cc:885] Found device 0 with properties:
name: GeForce GTX 1080
major: 6 minor: 1 memoryClockRate (GHz) 1.8095
pciBusID 0000:83:00.0
Total memory: 7.93GiB
Free memory: 7.70GiB
W tensorflow/stream_executor/cuda/cuda_driver.cc:590] creating context when one is currently active; existing: 0x641f8c0
I tensorflow/core/common_runtime/gpu/gpu_device.cc:885] Found device 0 with properties:
name: GeForce GTX 1080
major: 6 minor: 1 memoryClockRate (GHz) 1.8095
pciBusID 0000:83:00.0
Total memory: 7.93GiB
Free memory: 7.70GiB
W tensorflow/stream_executor/cuda/cuda_driver.cc:590] creating context when one is currently active; existing: 0x730cd40
I tensorflow/core/common_runtime/gpu/gpu_device.cc:885] Found device 1 with properties:
name: GeForce GTX 1080
major: 6 minor: 1 memoryClockRate (GHz) 1.8095
pciBusID 0000:84:00.0
Total memory: 7.93GiB
Free memory: 7.70GiB
I tensorflow/core/common_runtime/gpu/gpu_device.cc:885] Found device 1 with properties:
name: GeForce GTX 1080
major: 6 minor: 1 memoryClockRate (GHz) 1.8095
pciBusID 0000:84:00.0
Total memory: 7.93GiB
Free memory: 7.70GiB
I tensorflow/core/common_runtime/gpu/gpu_device.cc:906] DMA: 0 1
I tensorflow/core/common_runtime/gpu/gpu_device.cc:916] 0: Y Y
I tensorflow/core/common_runtime/gpu/gpu_device.cc:906] DMA: 0 1
I tensorflow/core/common_runtime/gpu/gpu_device.cc:916] 1: Y Y
I tensorflow/core/common_runtime/gpu/gpu_device.cc:916] 0: Y Y
I tensorflow/core/common_runtime/gpu/gpu_device.cc:916] 1: Y Y
I tensorflow/core/common_runtime/gpu/gpu_device.cc:975] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX 1080, pci bus id: 0000:83:00.0)
I tensorflow/core/common_runtime/gpu/gpu_device.cc:975] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX 1080, pci bus id: 0000:83:00.0)
I tensorflow/core/common_runtime/gpu/gpu_device.cc:975] Creating TensorFlow device (/gpu:1) -> (device: 1, name: GeForce GTX 1080, pci bus id: 0000:84:00.0)
I tensorflow/core/common_runtime/gpu/gpu_device.cc:975] Creating TensorFlow device (/gpu:1) -> (device: 1, name: GeForce GTX 1080, pci bus id: 0000:84:00.0)
E tensorflow/stream_executor/cuda/cuda_driver.cc:1002] failed to allocate 7.32G (7857173248 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY
E tensorflow/stream_executor/cuda/cuda_driver.cc:1002] failed to allocate 6.58G (7071455744 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY
E tensorflow/stream_executor/cuda/cuda_driver.cc:1002] failed to allocate 5.93G (6364310016 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY
E tensorflow/stream_executor/cuda/cuda_driver.cc:1002] failed to allocate 5.33G (5727878656 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY
E tensorflow/stream_executor/cuda/cuda_driver.cc:1002] failed to allocate 4.80G (5155090432 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY
E tensorflow/stream_executor/cuda/cuda_driver.cc:1002] failed to allocate 4.32G (4639581184 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY
E tensorflow/stream_executor/cuda/cuda_driver.cc:1002] failed to allocate 3.89G (4175622912 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY
E tensorflow/stream_executor/cuda/cuda_driver.cc:1002] failed to allocate 3.50G (3758060544 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY
E tensorflow/stream_executor/cuda/cuda_driver.cc:1002] failed to allocate 3.15G (3382254336 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY
E tensorflow/stream_executor/cuda/cuda_driver.cc:1002] failed to allocate 2.83G (3044028928 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY
E tensorflow/stream_executor/cuda/cuda_driver.cc:1002] failed to allocate 2.55G (2739625984 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY
E tensorflow/stream_executor/cuda/cuda_driver.cc:1002] failed to allocate 2.30G (2465663232 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY
E tensorflow/stream_executor/cuda/cuda_driver.cc:1002] failed to allocate 2.07G (2219096832 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY
E tensorflow/stream_executor/cuda/cuda_driver.cc:1002] failed to allocate 1.86G (1997187072 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY
E tensorflow/stream_executor/cuda/cuda_driver.cc:1002] failed to allocate 1.67G (1797468416 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY
E tensorflow/stream_executor/cuda/cuda_driver.cc:1002] failed to allocate 1.51G (1617721600 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY
E tensorflow/stream_executor/cuda/cuda_driver.cc:1002] failed to allocate 1.36G (1455949568 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY
E tensorflow/stream_executor/cuda/cuda_driver.cc:1002] failed to allocate 1.22G (1310354688 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY
E tensorflow/stream_executor/cuda/cuda_driver.cc:1002] failed to allocate 1.10G (1179319296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY
E tensorflow/stream_executor/cuda/cuda_driver.cc:1002] failed to allocate 1012.22M (1061387520 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY
E tensorflow/stream_executor/cuda/cuda_driver.cc:1002] failed to allocate 911.00M (955248896 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY
E tensorflow/stream_executor/cuda/cuda_driver.cc:1002] failed to allocate 819.90M (859724032 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY
E tensorflow/stream_executor/cuda/cuda_driver.cc:1002] failed to allocate 737.91M (773751808 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY
E tensorflow/stream_executor/cuda/cuda_driver.cc:1002] failed to allocate 664.12M (696376832 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY
E tensorflow/stream_executor/cuda/cuda_driver.cc:1002] failed to allocate 597.71M (626739200 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY
E tensorflow/stream_executor/cuda/cuda_driver.cc:1002] failed to allocate 537.93M (564065280 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY
E tensorflow/stream_executor/cuda/cuda_driver.cc:1002] failed to allocate 484.14M (507658752 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY
E tensorflow/stream_executor/cuda/cuda_driver.cc:1002] failed to allocate 435.73M (456892928 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY
E tensorflow/stream_executor/cuda/cuda_driver.cc:1002] failed to allocate 392.15M (411203840 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY
pid: 0 traj: 0 / 1031E tensorflow/stream_executor/cuda/cuda_blas.cc:372] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED
W tensorflow/stream_executor/stream.cc:1390] attempting to perform BLAS operation using StreamExecutor without BLAS support
E tensorflow/stream_executor/cuda/cuda_blas.cc:372] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED
W tensorflow/stream_executor/stream.cc:1390] attempting to perform BLAS operation using StreamExecutor without BLAS support
Traceback (most recent call last):
File "validate.py", line 381, in
random_seed=run_args.random_seed
File "validate.py", line 266, in collect
random_seed=random_seed
File "validate.py", line 188, in parallel_collect_trajectories
[res.get() for res in results]
File "validate.py", line 188, in
[res.get() for res in results]
File "/opt/conda/envs/rllab3/lib/python3.5/multiprocessing/pool.py", line 644, in get
raise self._value
multiprocessing.pool.MaybeEncodingError: Error sending result: '<multiprocessing.pool.ExceptionWithTraceback object at 0x7f23f5742710>'. Reason: 'PicklingError("Can't pickle <class 'module'>: attribute lookup module on builtins failed",)'

signal (15): Terminated
in expression starting at no file:0
do_futex_wait.constprop.3 at /lib/x86_64-linux-gnu/libpthread.so.0 (unknown line)
__new_sem_wait_slow.constprop.0 at /lib/x86_64-linux-gnu/libpthread.so.0 (unknown line)
sem_wait at /lib/x86_64-linux-gnu/libpthread.so.0 (unknown line)
semlock_acquire at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Modules/_multiprocessing/semaphore.c:310

signal (15): Terminated
in expression starting at no file:0
PyCFunction_Call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/methodobject.c:98
call_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4758 [inlined]
PyEval_EvalFrameEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:3289
_PyEval_EvalCodeWithName at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4071
PyEval_EvalCodeEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4092
function_call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/funcobject.c:627
PyObject_Call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/abstract.c:2166
method_call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/classobject.c:330
PyObject_Call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/abstract.c:2166
PyObject_CallFunctionObjArgs at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/abstract.c:2446
PyEval_EvalFrameEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:3160
fast_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4856 [inlined]
call_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4783 [inlined]
PyEval_EvalFrameEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:3289
_PyEval_EvalCodeWithName at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4071
PyEval_EvalCodeEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4092
function_call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/funcobject.c:627
PyObject_Call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/abstract.c:2166
ext_do_call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:5087 [inlined]
PyEval_EvalFrameEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:3328
fast_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4856 [inlined]
call_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4783 [inlined]
PyEval_EvalFrameEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:3289
fast_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4856 [inlined]
call_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4783 [inlined]
PyEval_EvalFrameEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:3289
fast_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4856 [inlined]
call_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4783 [inlined]
PyEval_EvalFrameEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:3289
_PyEval_EvalCodeWithName at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4071
PyEval_EvalCodeEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4092
at /root/code/julia-1.1.0/bin/../lib/julia/libLLVM-6.0.so (unknown line)
ZN4llvm15ScalarEvolution16isKnownPredicateENS_7CmpInst9PredicateEPKNS_4SCEVES5 at /root/code/julia-1.1.0/bin/../lib/julia/libLLVM-6.0.so (unknown line)
function_call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/funcobject.c:627
PyObject_Call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/abstract.c:2166
method_call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/classobject.c:330
PyObject_Call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/abstract.c:2166
ZN4llvm15ScalarEvolution25isKnownViaSimpleReasoningENS_7CmpInst9PredicateEPKNS_4SCEVES5 at /root/code/julia-1.1.0/bin/../lib/julia/libLLVM-6.0.so (unknown line)
ZN4llvm15ScalarEvolution27isImpliedCondOperandsHelperENS_7CmpInst9PredicateEPKNS_4SCEVES5_S5_S5 at /root/code/julia-1.1.0/bin/../lib/julia/libLLVM-6.0.so (unknown line)
ZN4llvm15ScalarEvolution21isImpliedCondOperandsENS_7CmpInst9PredicateEPKNS_4SCEVES5_S5_S5 at /root/code/julia-1.1.0/bin/../lib/julia/libLLVM-6.0.so (unknown line)
slot_tp_init at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/typeobject.c:6314
type_call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/typeobject.c:905
PyObject_Call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/abstract.c:2166
do_call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4989 [inlined]
call_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4785 [inlined]
PyEval_EvalFrameEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:3289
fast_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4856 [inlined]
call_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4783 [inlined]
PyEval_EvalFrameEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:3289
fast_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4856 [inlined]
call_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4783 [inlined]
PyEval_EvalFrameEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:3289
fast_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4856 [inlined]
call_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4783 [inlined]
PyEval_EvalFrameEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:3289
_PyEval_EvalCodeWithName at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4071
PyEval_EvalCodeEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4092
function_call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/funcobject.c:627
PyObject_Call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/abstract.c:2166
method_call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/classobject.c:330
PyObject_Call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/abstract.c:2166
ZN4llvm15ScalarEvolution13isImpliedCondENS_7CmpInst9PredicateEPKNS_4SCEVES5_S2_S5_S5 at /root/code/julia-1.1.0/bin/../lib/julia/libLLVM-6.0.so (unknown line)
ZN4llvm15ScalarEvolution27isLoopBackedgeGuardedByCondEPKNS_4LoopENS_7CmpInst9PredicateEPKNS_4SCEVES8 at /root/code/julia-1.1.0/bin/../lib/julia/libLLVM-6.0.so (unknown line)
_ZN4llvm15ScalarEvolution17getSignExtendExprEPKNS_4SCEVEPNS_4TypeEj at /root/code/julia-1.1.0/bin/../lib/julia/libLLVM-6.0.so (unknown line)
slot_tp_init at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/typeobject.c:6314
_ZN4llvm17simplifyUsersOfIVEPNS_7PHINodeEPNS_15ScalarEvolutionEPNS_13DominatorTreeEPNS_8LoopInfoERNS_15SmallVectorImplINS_14WeakTrackingVHEEERNS_12SCEVExpanderEPNS_9IVVisitorE at /root/code/julia-1.1.0/bin/../lib/julia/libLLVM-6.0.so (unknown line)
type_call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/typeobject.c:905
PyObject_Call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/abstract.c:2166
do_call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4989 [inlined]
call_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4785 [inlined]
PyEval_EvalFrameEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:3289
_PyEval_EvalCodeWithName at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4071
fast_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4866 [inlined]
call_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4783 [inlined]
PyEval_EvalFrameEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:3289
_PyEval_EvalCodeWithName at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4071
fast_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4866 [inlined]
call_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4783 [inlined]
PyEval_EvalFrameEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:3289
_PyEval_EvalCodeWithName at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4071
fast_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4866 [inlined]
call_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4783 [inlined]
PyEval_EvalFrameEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:3289
_PyEval_EvalCodeWithName at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4071
PyEval_EvalCodeEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4092
PyEval_EvalCode at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:777
run_mod at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/pythonrun.c:991 [inlined]
PyRun_FileExFlags at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/pythonrun.c:944
PyRun_SimpleFileExFlags at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/pythonrun.c:411
run_file at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Modules/main.c:328 [inlined]
Py_Main at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Modules/main.c:779
main at python (unknown line)
_ZN12_GLOBAL__N_114IndVarSimplify17simplifyAndExtendEPN4llvm4LoopERNS1_12SCEVExpanderEPNS1_8LoopInfoE.constprop.426 at /root/code/julia-1.1.0/bin/../lib/julia/libLLVM-6.0.so (unknown line)
_ZN12_GLOBAL__N_114IndVarSimplify3runEPN4llvm4LoopE.part.412 at /root/code/julia-1.1.0/bin/../lib/julia/libLLVM-6.0.so (unknown line)
__libc_start_main at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
unknown function (ip: 0x400998)
unknown function (ip: 0xffffffffffffffff)
Allocations: 112352957 (Pool: 112338583; Big: 14374); GC: 238

signal (11): Segmentation fault
in expression starting at no file:0
PyErr_Fetch at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/errors.c:328
PyObject_ClearWeakRefs at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/weakrefobject.c:909
_ZN12_GLOBAL__N_124IndVarSimplifyLegacyPass9runOnLoopEPN4llvm4LoopERNS1_13LPPassManagerE.part.413 at /root/code/julia-1.1.0/bin/../lib/julia/libLLVM-6.0.so (unknown line)
_ZN4llvm13LPPassManager13runOnFunctionERNS_8FunctionE at /root/code/julia-1.1.0/bin/../lib/julia/libLLVM-6.0.so (unknown line)
_ZN4llvm13FPPassManager13runOnFunctionERNS_8FunctionE at /root/code/julia-1.1.0/bin/../lib/julia/libLLVM-6.0.so (unknown line)
_ZN4llvm13FPPassManager11runOnModuleERNS_6ModuleE at /root/code/julia-1.1.0/bin/../lib/julia/libLLVM-6.0.so (unknown line)
ZN4llvm6legacy15PassManagerImpl3runERNS_6ModuleE at /root/code/julia-1.1.0/bin/../lib/julia/libLLVM-6.0.so (unknown line)
array_dealloc at /home/ilan/minonda/conda-bld/numpy-1.12_1486268766587/work/numpy-1.12.0/numpy/core/src/multiarray/arrayobject.c:384
pydecref at /root/.julia/packages/PyCall/ttONZ/src/PyCall.jl:121
pydecref at /root/.julia/packages/PyCall/ttONZ/src/PyCall.jl:126
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2219
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1571 [inlined]
run_finalizer at /buildworker/worker/package_linux64/build/src/gc.c:232
jl_gc_run_finalizers_in_list at /buildworker/worker/package_linux64/build/src/gc.c:318
jl_gc_run_all_finalizers at /buildworker/worker/package_linux64/build/src/gc.c:346
jl_atexit_hook at /buildworker/worker/package_linux64/build/src/init.c:249
jl_exit at /buildworker/worker/package_linux64/build/src/jl_uv.c:592
jl_exit_thread0_cb at /buildworker/worker/package_linux64/build/src/signals-unix.c:344
unknown function (ip: 0xffffffffffffffff)
Allocations: 112353419 (Pool: 112339045; Big: 14374); GC: 238
operator() at /buildworker/worker/package_linux64/build/src/jitlayers.cpp:335
E tensorflow/stream_executor/cuda/cuda_blas.cc:372] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED
W tensorflow/stream_executor/stream.cc:1390] attempting to perform BLAS operation using StreamExecutor without BLAS support
addModule at /buildworker/worker/package_linux64/build/usr/include/llvm/ExecutionEngine/Orc/IRCompileLayer.h:57 [inlined]
addModule at /buildworker/worker/package_linux64/build/src/jitlayers.cpp:454
jl_add_to_ee at /buildworker/worker/package_linux64/build/src/jitlayers.cpp:679 [inlined]
jl_finalize_function at /buildworker/worker/package_linux64/build/src/jitlayers.cpp:687
E tensorflow/stream_executor/cuda/cuda_blas.cc:372] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED
W tensorflow/stream_executor/stream.cc:1390] attempting to perform BLAS operation using StreamExecutor without BLAS support
getAddressForFunction at /buildworker/worker/package_linux64/build/src/codegen.cpp:1281
jl_generate_fptr at /buildworker/worker/package_linux64/build/src/codegen.cpp:1388
jl_compile_method_internal at /buildworker/worker/package_linux64/build/src/gf.c:1856 [inlined]
jl_fptr_trampoline at /buildworker/worker/package_linux64/build/src/gf.c:1863
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2219
deepcopy_internal at ./deepcopy.jl:78 [inlined]
deepcopy at ./deepcopy.jl:28 [inlined]
get_features at /home/lius/tmp/github/ngsim_env/julia/src/ngsim_env.jl:218
#reset#22 at /home/lius/tmp/github/ngsim_env/julia/src/ngsim_env.jl:125
unknown function (ip: 0x7f23cbe7e1a4)
jl_fptr_trampoline at /buildworker/worker/package_linux64/build/src/gf.c:1864
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2219
#reset at ./none:0 [inlined]
#reset#2 at /home/lius/tmp/github/ngsim_env/julia/src/env.jl:19 [inlined]
#reset at ./none:0
unknown function (ip: 0x7f23cbe7dcb9)
jl_fptr_trampoline at /buildworker/worker/package_linux64/build/src/gf.c:1864
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2219
f_kw_closure at /root/.julia/packages/PyCall/ttONZ/src/callback.jl:36
jl_fptr_trampoline at /buildworker/worker/package_linux64/build/src/gf.c:1864
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2219
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1571 [inlined]
jl_f__apply at /buildworker/worker/package_linux64/build/src/builtins.c:556
jl_f__apply_latest at /buildworker/worker/package_linux64/build/src/builtins.c:594
_pyjlwrap_call at /root/.julia/packages/PyCall/ttONZ/src/callback.jl:37
unknown function (ip: 0x7f23cbe1c443)
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2219
pyjlwrap_call at /root/.julia/packages/PyCall/ttONZ/src/callback.jl:49
unknown function (ip: 0x7f23cbe1bd76)
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2219
unknown function (ip: 0x7f23cbddbafc)
unknown function (ip: 0x7f23cbddb9ce)
PyObject_Call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/abstract.c:2166
ext_do_call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:5087 [inlined]
PyEval_EvalFrameEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:3328
_PyEval_EvalCodeWithName at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4071
PyEval_EvalCodeEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4092
function_call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/funcobject.c:627
PyObject_Call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/abstract.c:2166
ext_do_call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:5087 [inlined]
PyEval_EvalFrameEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:3328
_PyEval_EvalCodeWithName at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4071
PyEval_EvalCodeEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4092
function_call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/funcobject.c:627
PyObject_Call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/abstract.c:2166
ext_do_call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:5087 [inlined]
PyEval_EvalFrameEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:3328
_PyEval_EvalCodeWithName at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4071
PyEval_EvalCodeEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4092
function_call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/funcobject.c:627
PyObject_Call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/abstract.c:2166
ext_do_call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:5087 [inlined]
PyEval_EvalFrameEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:3328
_PyEval_EvalCodeWithName at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4071
fast_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4866 [inlined]
call_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4783 [inlined]
PyEval_EvalFrameEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:3289
_PyEval_EvalCodeWithName at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4071
PyEval_EvalCodeEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4092
function_call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/funcobject.c:627
PyObject_Call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/abstract.c:2166
ext_do_call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:5087 [inlined]
PyEval_EvalFrameEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:3328
_PyEval_EvalCodeWithName at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4071
PyEval_EvalCodeEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4092
function_call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/funcobject.c:627
PyObject_Call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/abstract.c:2166
ext_do_call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:5087 [inlined]
PyEval_EvalFrameEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:3328
fast_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4856 [inlined]
call_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4783 [inlined]
PyEval_EvalFrameEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:3289
fast_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4856 [inlined]
call_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4783 [inlined]
PyEval_EvalFrameEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:3289
fast_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4856 [inlined]
call_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4783 [inlined]
PyEval_EvalFrameEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:3289
_PyEval_EvalCodeWithName at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4071
PyEval_EvalCodeEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4092
function_call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/funcobject.c:627
PyObject_Call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/abstract.c:2166
method_call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/classobject.c:330
PyObject_Call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/abstract.c:2166
slot_tp_init at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/typeobject.c:6314
type_call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/typeobject.c:905
PyObject_Call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/abstract.c:2166
do_call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4989 [inlined]
call_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4785 [inlined]
PyEval_EvalFrameEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:3289
fast_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4856 [inlined]
call_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4783 [inlined]
PyEval_EvalFrameEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:3289
fast_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4856 [inlined]
call_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4783 [inlined]
PyEval_EvalFrameEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:3289
fast_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4856 [inlined]
call_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4783 [inlined]
PyEval_EvalFrameEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:3289
_PyEval_EvalCodeWithName at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4071
PyEval_EvalCodeEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4092
function_call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/funcobject.c:627
PyObject_Call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/abstract.c:2166
method_call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/classobject.c:330
PyObject_Call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/abstract.c:2166
slot_tp_init at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/typeobject.c:6314
type_call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/typeobject.c:905
PyObject_Call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/abstract.c:2166
do_call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4989 [inlined]
call_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4785 [inlined]
PyEval_EvalFrameEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:3289
_PyEval_EvalCodeWithName at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4071
fast_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4866 [inlined]
call_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4783 [inlined]
PyEval_EvalFrameEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:3289
_PyEval_EvalCodeWithName at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4071
fast_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4866 [inlined]
call_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4783 [inlined]
PyEval_EvalFrameEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:3289
_PyEval_EvalCodeWithName at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4071
fast_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4866 [inlined]
call_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4783 [inlined]
PyEval_EvalFrameEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:3289
_PyEval_EvalCodeWithName at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4071
PyEval_EvalCodeEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4092
PyEval_EvalCode at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:777
run_mod at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/pythonrun.c:991 [inlined]
PyRun_FileExFlags at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/pythonrun.c:944
PyRun_SimpleFileExFlags at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/pythonrun.c:411
run_file at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Modules/main.c:328 [inlined]
Py_Main at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Modules/main.c:779
main at python (unknown line)
__libc_start_main at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
unknown function (ip: 0x400998)
unknown function (ip: 0xffffffffffffffff)
Allocations: 112471597 (Pool: 112457178; Big: 14419); GC: 238

signal (11): Segmentation fault
in expression starting at no file:0
PyErr_Fetch at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/errors.c:328
PyObject_ClearWeakRefs at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/weakrefobject.c:909
array_dealloc at /home/ilan/minonda/conda-bld/numpy-1.12_1486268766587/work/numpy-1.12.0/numpy/core/src/multiarray/arrayobject.c:384
pydecref_ at /root/.julia/packages/PyCall/ttONZ/src/PyCall.jl:121
pydecref at /root/.julia/packages/PyCall/ttONZ/src/PyCall.jl:126
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2219
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1571 [inlined]
run_finalizer at /buildworker/worker/package_linux64/build/src/gc.c:232
jl_gc_run_finalizers_in_list at /buildworker/worker/package_linux64/build/src/gc.c:318
jl_gc_run_all_finalizers at /buildworker/worker/package_linux64/build/src/gc.c:346
jl_atexit_hook at /buildworker/worker/package_linux64/build/src/init.c:249
jl_exit at /buildworker/worker/package_linux64/build/src/jl_uv.c:592
jl_exit_thread0_cb at /buildworker/worker/package_linux64/build/src/signals-unix.c:344
unknown function (ip: 0xffffffffffffffff)
Allocations: 112472059 (Pool: 112457640; Big: 14419); GC: 238
@mengyuest
Copy link

I encountered a similar issue (CUDA OUT OF MEMORY). One simple solution is just to uninstalled the tensorflow-gpu and reinstalled the tensorflow (so they will not use CUDA). There should be a more elegant way to handle this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@Depblu @mengyuest