RuntimeError: Function "cuMemAllocAsync" not found

[vincentxiaotang]

When I ran the example file TensorFlowToTensorRT-NHWC.py, it occurs：Traceback (most recent call last):
File "TensorFlowToTensorRT-NHWC.py", line 161, in
_, inputD0 = cudart.cudaMallocAsync(inputH0.nbytes, stream)
File "cuda/cudart.pyx", line 16938, in cuda.cudart.cudaMallocAsync
File "cuda/ccudart.pyx", line 1210, in cuda.ccudart.cudaMallocAsync
File "cuda/_cuda/ccuda.pyx", line 4970, in cuda._cuda.ccuda._cuMemAllocAsync
RuntimeError: Function "cuMemAllocAsync" not found.

---

I'm using the NVIDIA NGC nvcr.io/nvidia/tensorflow:21.12-tf1-py3, the detailed environment are as follows:
GeForce RTX 2080 Ti，Driver Version: 455.23.05，nvcr.io/nvidia/tensorflow:21.12-tf1-py3，
Package Version

---

absl-py 1.0.0
appdirs 1.4.4
argon2-cffi 21.1.0
asgiref 3.4.1
astor 0.8.1
astunparse 1.6.3
attrs 21.2.0
audioread 2.1.9
backcall 0.2.0
bleach 4.1.0
cachetools 4.2.4
certifi 2021.10.8
cffi 1.15.0
charset-normalizer 2.0.8
click 8.0.3
cloudpickle 2.0.0
cmake-setuptools 0.1.3
cuda-python 11.7.0
cudf 21.10.0a0+345.ge05bd4bf3c
cugraph 21.10.0a0+102.gab401cad
cuml 21.10.0a0+116.gdc14361ba
cupy-cuda114 9.3.0
cupy-cuda115 9.6.0
cycler 0.11.0
Cython 0.29.24
dask 2021.9.1
dask-cuda 21.10.0
dask-cudf 21.10.0a0+345.ge05bd4bf3c
dask-glm 0.2.0
dask-ml 1.9.0
debugpy 1.5.1
decorator 5.1.0
defusedxml 0.7.1
distributed 2021.9.1
Django 3.2.6
entrypoints 0.3
fastavro 1.4.4
fastrlock 0.8
filelock 3.4.0
flatbuffers 1.12
fsspec 2021.7.0
future 0.18.2
gast 0.3.3
google-pasta 0.2.0
graphsurgeon 0.4.5
grpcio 1.42.0
gunicorn 20.1.0
h11 0.12.0
h5py 2.10.0
HeapDict 1.0.1
horovod 0.22.1
httptools 0.2.0
huggingface-hub 0.0.12
idna 3.3
importlib-metadata 4.8.2
importlib-resources 5.4.0
iniconfig 1.1.1
ipykernel 6.6.0
ipython 7.30.0
ipython-genutils 0.2.0
jedi 0.18.1
Jinja2 3.0.3
joblib 1.1.0
json5 0.9.6
jsonschema 4.2.1
jupyter-client 7.1.0
jupyter-core 4.9.1
jupyter-tensorboard 0.2.0
jupyterlab 2.3.2
jupyterlab-pygments 0.1.2
jupyterlab-server 1.2.0
jupytext 1.13.2
Keras-Applications 1.0.8
Keras-Preprocessing 1.0.5
kiwisolver 1.3.2
librosa 0.9.1
llvmlite 0.36.0
locket 0.2.1
Markdown 3.3.6
markdown-it-py 1.1.0
MarkupSafe 2.0.1
matplotlib 3.4.3
matplotlib-inline 0.1.3
mdit-py-plugins 0.2.8
mistune 0.8.4
mock 3.0.5
msgpack 1.0.3
multipledispatch 0.6.0
nbclient 0.5.9
nbconvert 6.3.0
nbformat 5.1.3
nest-asyncio 1.5.4
networkx 2.6.3
nltk 3.6.4
notebook 6.4.3
numba 0.53.1
numpy 1.22.4
nvidia-dali-cuda110 1.8.0
nvidia-dali-tf-plugin-cuda110 1.8.0
nvidia-dlprofviewer 1.8.0
nvidia-pyindex 1.0.9
nvtx 0.2.3
onnx 1.11.0
onnxruntime-gpu 1.11.1
opencv-python 4.5.5.64
opt-einsum 3.3.0
packaging 21.3
pandas 1.2.5
pandocfilters 1.5.0
parso 0.8.3
partd 1.2.0
pexpect 4.7.0
pickleshare 0.7.5
Pillow 8.4.0
pip 21.3.1
pluggy 1.0.0
polygraphy 0.33.0
pooch 1.6.0
portpicker 1.3.1
prometheus-client 0.12.0
prompt-toolkit 3.0.23
protobuf 3.19.1
psutil 5.7.0
ptyprocess 0.7.0
py 1.11.0
pyarrow 5.0.0
pycparser 2.21
Pygments 2.10.0
pynvml 11.4.1
pyparsing 3.0.6
pypi-kenlm 0.1.20210121
pyrsistent 0.18.0
pytest 6.2.5
python-dateutil 2.8.2
python-dotenv 0.19.2
pytz 2021.3
PyYAML 6.0
pyzmq 22.3.0
regex 2021.11.10
requests 2.26.0
resampy 0.2.2
rmm 21.10.0a0+42.gae27a57
sacremoses 0.0.46
scikit-learn 0.24.0
scipy 1.4.1
Send2Trash 1.8.0
setuptools 59.4.0
six 1.16.0
sortedcontainers 2.4.0
SoundFile 0.10.3.post1
sqlparse 0.4.2
tblib 1.7.0
tensorboard 1.15.0
tensorflow 1.15.5+nv
tensorflow-estimator 1.15.1
tensorrt 8.2.1.8
termcolor 1.1.0
terminado 0.12.1
testpath 0.5.0
tf2onnx 1.10.1
threadpoolctl 3.0.0
tokenizers 0.10.3
toml 0.10.2
toolz 0.11.2
tornado 6.1
tqdm 4.62.3
traitlets 5.1.1
transformers 4.9.1
treelite 2.1.0
treelite-runtime 2.1.0
typing_extensions 4.0.1
ucx-py 0.21.0a0+37.gbfa0450
uff 0.6.9
urllib3 1.26.7
uvicorn 0.15.0
uvloop 0.16.0
watchgod 0.7
wcwidth 0.2.5
webencodings 0.5.1
websockets 10.1
Werkzeug 2.0.2
wheel 0.37.0
whitenoise 5.3.0
wrapt 1.13.3
xgboost 1.4.2
zict 2.0.0
zipp 3.6.0

[vzhurba01]

Driver Version: 455.23.05

Driver 455 is associated with CUDA Toolkit 11.1 (CTK 11.1). However cudaMallocAsync/cuMemAllocAsync was introduced in CUDA Toolkit 11.2.

Getting the latest (or at minimum 11.2) Toolkit + driver should resolve this error.

[vincentxiaotang]

Driver Version: 455.23.05

Driver 455 is associated with CUDA Toolkit 11.1 (CTK 11.1). However cudaMallocAsync/cuMemAllocAsync was introduced in CUDA Toolkit 11.2.

Getting the latest (or at minimum 11.2) Toolkit + driver should resolve this error.

thank you, i'll have a try.