fresh pip install error: requires nvidia-tensorboard==1.15.0+nv20.11, but you have 1.15.0+nv20.12

[bayesrule]

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
nvidia-tensorflow 1.15.4+nv20.11 requires nvidia-tensorboard==1.15.0+nv20.11, but you have nvidia-tensorboard 1.15.0+nv20.12 which is incompatible.

[DEKHTIARJonathan]

The error is explicit. Please uninstall first any trace of nvidia-tensorboard

[bayesrule]

Thanks for answering, but it was a fresh install in a new env with the pip command instructed in the README...

[DEKHTIARJonathan]

You must have done something incorrectly otherwise you would not get an error saying that you already have nvidia-tensorboard 1.15.0+nv20.12

[rahman-mdatiqur]

Hello @DEKHTIARJonathan ,
exact same issue here as mentioned by @bayesrule. I tried to install nvidia-tensorflow==1.15.4+nv20.11 on a freshly created virtual environment which had no trace of tensorboard. But the installation had the below error message displayed at the end

Can you please advise?

[rahman-mdatiqur]

Just to provide an update, I am now training using nvidia-tensorflow==1.15.4+nv20.11. But, my training aborts abnormally after few hundred steps spitting out the following error message. Just wondering, if this abnormal termination is related to the error being discussed on this post.

In that case, can anyone please suggest a solution/workaround?

Thanks.

`
....................
step: 1664 train-loss: 3.7866263389587402 train-acc: 0.05000000074505806
step: 1665 train-loss: 3.7862656116485596 train-acc: 0.10000000149011612

Fatal Python error: Segmentation fault

Thread 0x2021-05-31 09:10:27.487499: E tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_LAUNCH_FAILED: unspecified launch failure
00007f4c2e7fc700 (most recent call first):
File "/usr/lib2021-05-31 09:10:27.487531: F tensorflow/core/common_runtime/gpu/gpu_event_mgr.cc:273] Unexpected Event status: 1
/python3.6/threading.py", line 295 in wait
File "/Fatal Python error: uAborteds

r/lib/python3.6/queue.py", line 164 in get
File "/home/atiqur/nvidia-tf-1.15.4-nv-20.11/lib/python3.2021-05-31 09:10:27.487552: F ./tensorflow/core/kernels/conv_2d_gpu.h:1015] Non-OK-status: GpuLaunchKernel( SwapDimension1And2InTensor3UsingTiles<T, kNumThreads, kTileSize, kTileSize, conjugate>, total_tiles_count, kNumThreads, 0, d.stream(), input, input_dims, output) status: Internal: unspecified launch failure
6/site-packages/tensorflow_core/python/summary/writer/event_file_writer.py", line 159 in run
File "/usr/lib/python3.6/threading.py", line 916 in _bootstrap_inner
File "/us
`