Fail to run the 3d-unet-99 with error "No matching distribution found for nvidia-nvcomp-cu12" after my mlc upgrade. #2187
Description
Fail to run the 3d-unet-99 with error "No matching distribution found for nvidia-nvcomp-cu12" after my mlc upgrade.
mlcr run-mlperf,inference,_find-performance,_full,_r5.0-dev
--model=3d-unet-99
--implementation=nvidia
--framework=tensorrt
--category=edge
--scenario=Offline
--execution_mode=test
--device=cuda
--docker --quiet
--test_query_count=50
I re-try some other benchmark and got the same error.
237.0 [2025-05-08 03:34:34,729 module.py:5121 INFO] - ! call /home/mlcuser/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/install.sh from tmp-run.sh
237.0
237.0 /usr/bin/python3 -m pip install "nvidia-dali-cuda120" --upgrade --default-timeout=900 --extra-index-url https://developer.download.nvidia.com/compute/redist
237.8 Looking in indexes: https://pypi.org/simple, https://developer.download.nvidia.com/compute/redist
241.8 Collecting nvidia-dali-cuda120
242.0 Downloading https://developer.download.nvidia.cn/compute/redist/nvidia-dali-cuda120/nvidia_dali_cuda120-1.49.0-py3-none-manylinux2014_x86_64.whl (435.7 MB)
268.1 Collecting astunparse<=1.6.3,>=1.6.0
268.5 Downloading astunparse-1.6.3-py2.py3-none-any.whl (12 kB)
270.5 Collecting nvidia-nvimgcodec-cu12[all]<0.6.0,>=0.5.0
270.7 Downloading nvidia_nvimgcodec_cu12-0.5.0.13-py3-none-manylinux2014_x86_64.whl (23.3 MB)
279.9 Collecting dm-tree<=0.1.8; python_version < "3.10"
280.1 Downloading dm_tree-0.1.8-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (152 kB)
281.9 Collecting packaging<=24.2
282.1 Downloading packaging-24.2-py3-none-any.whl (65 kB)
282.2 Requirement already satisfied, skipping upgrade: six<=1.17,>=1.16 in /home/mlcuser/.local/lib/python3.8/site-packages (from nvidia-dali-cuda120) (1.17.0)
283.8 Collecting gast<=0.6.0,>=0.3.3
284.0 Downloading gast-0.6.0-py3-none-any.whl (21 kB)
284.0 Requirement already satisfied, skipping upgrade: wheel<1.0,>=0.23.0 in /usr/lib/python3/dist-packages (from astunparse<=1.6.3,>=1.6.0->nvidia-dali-cuda120) (0.34.2)
285.9 Collecting nvidia-nvjpeg2k-cu12; extra == "all"
286.1 Downloading nvidia_nvjpeg2k_cu12-0.8.1.40-py3-none-manylinux2014_x86_64.whl (3.8 MB)
290.1 ERROR: Could not find a version that satisfies the requirement nvidia-nvcomp-cu12; extra == "all" (from nvidia-nvimgcodec-cu12[all]<0.6.0,>=0.5.0->nvidia-dali-cuda120) (from versions: none)
290.3 ERROR: No matching distribution found for nvidia-nvcomp-cu12; extra == "all" (from nvidia-nvimgcodec-cu12[all]<0.6.0,>=0.5.0->nvidia-dali-cuda120)
290.4 Traceback (most recent call last):
290.4 File "/home/mlcuser/.local/bin/mlcr", line 8, in <module>
290.4 sys.exit(mlcr())
290.4 File "/home/mlcuser/.local/lib/python3.8/site-packages/mlc/main.py", line 86, in mlcr
290.4 main()
290.4 File "/home/mlcuser/.local/lib/python3.8/site-packages/mlc/main.py", line 242, in main
290.4 res = method(run_args)
290.4 File "/home/mlcuser/.local/lib/python3.8/site-packages/mlc/script_action.py", line 305, in run
290.4 return self.call_script_module_function("run", run_args)
290.4 File "/home/mlcuser/.local/lib/python3.8/site-packages/mlc/script_action.py", line 231, in call_script_module_function
290.4 result = automation_instance.run(run_args) # Pass args to the run method
290.4 File "/home/mlcuser/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 226, in run
290.4 r = self._run(i)
290.4 File "/home/mlcuser/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 1856, in _run
290.4 r = self._call_run_deps(prehook_deps, self.local_env_keys, local_env_keys_from_meta, env, state, const, const_state, add_deps_recursive,
290.4 File "/home/mlcuser/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 3317, in _call_run_deps
290.4 r = script._run_deps(deps, local_env_keys, env, state, const, const_state, add_deps_recursive, recursion_spaces,
290.4 File "/home/mlcuser/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 3485, in _run_deps
290.4 r = self.action_object.access(ii)
290.4 File "/home/mlcuser/.local/lib/python3.8/site-packages/mlc/action.py", line 56, in access
290.4 result = method(options)
290.4 File "/home/mlcuser/.local/lib/python3.8/site-packages/mlc/script_action.py", line 305, in run
290.4 return self.call_script_module_function("run", run_args)
290.4 File "/home/mlcuser/.local/lib/python3.8/site-packages/mlc/script_action.py", line 231, in call_script_module_function
290.4 result = automation_instance.run(run_args) # Pass args to the run method
290.4 File "/home/mlcuser/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 226, in run
290.4 r = self._run(i)
290.4 File "/home/mlcuser/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 1641, in _run
290.4 r = self._call_run_deps(deps, self.local_env_keys, local_env_keys_from_meta, env, state, const, const_state, add_deps_recursive,
290.4 File "/home/mlcuser/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 3317, in _call_run_deps
290.4 r = script._run_deps(deps, local_env_keys, env, state, const, const_state, add_deps_recursive, recursion_spaces,
290.4 File "/home/mlcuser/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 3485, in _run_deps
290.4 r = self.action_object.access(ii)
290.4 File "/home/mlcuser/.local/lib/python3.8/site-packages/mlc/action.py", line 56, in access
290.4 result = method(options)
290.4 File "/home/mlcuser/.local/lib/python3.8/site-packages/mlc/script_action.py", line 305, in run
290.4 return self.call_script_module_function("run", run_args)
290.4 File "/home/mlcuser/.local/lib/python3.8/site-packages/mlc/script_action.py", line 231, in call_script_module_function
290.4 result = automation_instance.run(run_args) # Pass args to the run method
290.4 File "/home/mlcuser/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 226, in run
290.4 r = self._run(i)
290.4 File "/home/mlcuser/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 1641, in _run
290.4 r = self._call_run_deps(deps, self.local_env_keys, local_env_keys_from_meta, env, state, const, const_state, add_deps_recursive,
290.4 File "/home/mlcuser/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 3317, in _call_run_deps
290.4 r = script._run_deps(deps, local_env_keys, env, state, const, const_state, add_deps_recursive, recursion_spaces,
290.4 File "/home/mlcuser/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 3485, in _run_deps
290.4 r = self.action_object.access(ii)
290.4 File "/home/mlcuser/.local/lib/python3.8/site-packages/mlc/action.py", line 56, in access
290.4 result = method(options)
290.4 File "/home/mlcuser/.local/lib/python3.8/site-packages/mlc/script_action.py", line 305, in run
290.4 return self.call_script_module_function("run", run_args)
290.4 File "/home/mlcuser/.local/lib/python3.8/site-packages/mlc/script_action.py", line 241, in call_script_module_function
290.4 raise ScriptExecutionError(f"Script {function_name} execution failed. Error : {error}")
290.4 mlc.script_action.ScriptExecutionError: Script run execution failed. Error : MLC script failed (name = get-generic-python-lib, return code = 256)
290.4
290.4
290.4 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
290.4 Please file an issue at https://github.com/mlcommons/mlperf-automations/issues along with the full MLC command being run and the relevant
290.4 or full console log.
290.4
------
1 warning found (use docker --debug to expand):
- SecretsUsedInArgOrEnv: Do not use ARG or ENV instructions for sensitive data (ARG "MLC_GH_TOKEN") (line 12)
nvcr.io-nvidia-mlperf-mlperf-inference-mlpinf-v4.0-cuda12.2-cudnn8.9-x86_64-ubuntu20.04-public.Dockerfile:44
--------------------
42 |
43 | # Run commands
44 | >>> RUN mlc pull repo && mlcr --tags=app,mlperf,inference,generic,_nvidia,_3d-unet-99,_tensorrt,_cuda,_test,_r5.0-dev_default,_offline --quiet=true --env.MLC_QUIET=yes --env.MLC_MLPERF_IMPLEMENTATION=nvidia --env.MLC_MLPERF_MODEL=3d-unet-99 --env.MLC_MLPERF_RUN_STYLE=test --env.MLC_MLPERF_SKIP_SUBMISSION_GENERATION=False --env.MLC_DOCKER_PRIVILEGED_MODE=True --env.MLC_MLPERF_SUBMISSION_DIVISION=open --env.MLC_MLPERF_INFERENCE_TP_SIZE=1 --env.MLC_MLPERF_SUBMISSION_SYSTEM_TYPE=edge --env.MLC_MLPERF_DEVICE=cuda --env.MLC_MLPERF_USE_DOCKER=True --env.MLC_MLPERF_BACKEND=tensorrt --env.MLC_MLPERF_LOADGEN_SCENARIO=Offline --env.MLC_TEST_QUERY_COUNT=50 --env.MLC_MLPERF_FIND_PERFORMANCE_MODE=yes --env.MLC_MLPERF_LOADGEN_ALL_MODES=no --env.MLC_MLPERF_LOADGEN_MODE=performance --env.MLC_MLPERF_RESULT_PUSH_TO_GITHUB=False --env.MLC_MLPERF_SUBMISSION_GENERATION_STYLE=full --env.MLC_MLPERF_INFERENCE_VERSION=5.0-dev --env.MLC_RUN_MLPERF_INFERENCE_APP_DEFAULTS=r5.0-dev_default --env.MLC_MLPERF_SUBMISSION_CHECKER_VERSION=v5.0 --env.MLC_MLPERF_INFERENCE_SOURCE_VERSION=5.0.16 --env.MLC_MLPERF_LAST_RELEASE=v5.0 --env.MLC_MLPERF_INFERENCE_RESULTS_VERSION=r5.0-dev --env.MLC_MODEL=3d-unet-99 --env.MLC_MLPERF_LOADGEN_COMPLIANCE=no --env.MLC_MLPERF_LOADGEN_EXTRA_OPTIONS= --env.MLC_MLPERF_LOADGEN_SCENARIOS,=Offline --env.MLC_MLPERF_LOADGEN_MODES,=performance --env.MLC_OUTPUT_FOLDER_NAME=test_results --add_deps_recursive.coco2014-original.tags=_full --add_deps_recursive.coco2014-preprocessed.tags=_full --add_deps_recursive.imagenet-original.tags=_full --add_deps_recursive.imagenet-preprocessed.tags=_full --add_deps_recursive.openimages-original.tags=_full --add_deps_recursive.openimages-preprocessed.tags=_full --add_deps_recursive.openorca-original.tags=_full --add_deps_recursive.openorca-preprocessed.tags=_full --add_deps_recursive.coco2014-dataset.tags=_full --add_deps_recursive.igbh-dataset.tags=_full --add_deps_recursive.get-mlperf-inference-results-dir.tags=_version.r5.0-dev --add_deps_recursive.get-mlperf-inference-submission-dir.tags=_version.r5.0-dev --add_deps_recursive.mlperf-inference-nvidia-scratch-space.tags=_version.r5.0-dev --print_env=False --print_deps=False --dump_version_info=True --quiet --fake_run --env.MLC_RUN_STATE_DOCKER=True
45 | RUN mlc pull repo && mlcr --tags=app,mlperf,inference,generic,_nvidia,_3d-unet-99,_tensorrt,_cuda,_test,_r5.0-dev_default,_offline --quiet=true --env.MLC_QUIET=yes --env.MLC_MLPERF_IMPLEMENTATION=nvidia --env.MLC_MLPERF_MODEL=3d-unet-99 --env.MLC_MLPERF_RUN_STYLE=test --env.MLC_MLPERF_SKIP_SUBMISSION_GENERATION=False --env.MLC_DOCKER_PRIVILEGED_MODE=True --env.MLC_MLPERF_SUBMISSION_DIVISION=open --env.MLC_MLPERF_INFERENCE_TP_SIZE=1 --env.MLC_MLPERF_SUBMISSION_SYSTEM_TYPE=edge --env.MLC_MLPERF_DEVICE=cuda --env.MLC_MLPERF_USE_DOCKER=True --env.MLC_MLPERF_BACKEND=tensorrt --env.MLC_MLPERF_LOADGEN_SCENARIO=Offline --env.MLC_TEST_QUERY_COUNT=50 --env.MLC_MLPERF_FIND_PERFORMANCE_MODE=yes --env.MLC_MLPERF_LOADGEN_ALL_MODES=no --env.MLC_MLPERF_LOADGEN_MODE=performance --env.MLC_MLPERF_RESULT_PUSH_TO_GITHUB=False --env.MLC_MLPERF_SUBMISSION_GENERATION_STYLE=full --env.MLC_MLPERF_INFERENCE_VERSION=5.0-dev --env.MLC_RUN_MLPERF_INFERENCE_APP_DEFAULTS=r5.0-dev_default --env.MLC_MLPERF_SUBMISSION_CHECKER_VERSION=v5.0 --env.MLC_MLPERF_INFERENCE_SOURCE_VERSION=5.0.16 --env.MLC_MLPERF_LAST_RELEASE=v5.0 --env.MLC_MLPERF_INFERENCE_RESULTS_VERSION=r5.0-dev --env.MLC_MODEL=3d-unet-99 --env.MLC_MLPERF_LOADGEN_COMPLIANCE=no --env.MLC_MLPERF_LOADGEN_EXTRA_OPTIONS= --env.MLC_MLPERF_LOADGEN_SCENARIOS,=Offline --env.MLC_MLPERF_LOADGEN_MODES,=performance --env.MLC_OUTPUT_FOLDER_NAME=test_results --add_deps_recursive.coco2014-original.tags=_full --add_deps_recursive.coco2014-preprocessed.tags=_full --add_deps_recursive.imagenet-original.tags=_full --add_deps_recursive.imagenet-preprocessed.tags=_full --add_deps_recursive.openimages-original.tags=_full --add_deps_recursive.openimages-preprocessed.tags=_full --add_deps_recursive.openorca-original.tags=_full --add_deps_recursive.openorca-preprocessed.tags=_full --add_deps_recursive.coco2014-dataset.tags=_full --add_deps_recursive.igbh-dataset.tags=_full --add_deps_recursive.get-mlperf-inference-results-dir.tags=_version.r5.0-dev --add_deps_recursive.get-mlperf-inference-submission-dir.tags=_version.r5.0-dev --add_deps_recursive.mlperf-inference-nvidia-scratch-space.tags=_version.r5.0-dev --print_env=False --print_deps=False --dump_version_info=True --quiet --env.MLC_RUN_STATE_DOCKER=True
46 |
--------------------
ERROR: failed to solve: process "/bin/sh -c mlc pull repo && mlcr --tags=app,mlperf,inference,generic,_nvidia,_3d-unet-99,_tensorrt,_cuda,_test,_r5.0-dev_default,_offline --quiet=true --env.MLC_QUIET=yes --env.MLC_MLPERF_IMPLEMENTATION=nvidia --env.MLC_MLPERF_MODEL=3d-unet-99 --env.MLC_MLPERF_RUN_STYLE=test --env.MLC_MLPERF_SKIP_SUBMISSION_GENERATION=False --env.MLC_DOCKER_PRIVILEGED_MODE=True --env.MLC_MLPERF_SUBMISSION_DIVISION=open --env.MLC_MLPERF_INFERENCE_TP_SIZE=1 --env.MLC_MLPERF_SUBMISSION_SYSTEM_TYPE=edge --env.MLC_MLPERF_DEVICE=cuda --env.MLC_MLPERF_USE_DOCKER=True --env.MLC_MLPERF_BACKEND=tensorrt --env.MLC_MLPERF_LOADGEN_SCENARIO=Offline --env.MLC_TEST_QUERY_COUNT=50 --env.MLC_MLPERF_FIND_PERFORMANCE_MODE=yes --env.MLC_MLPERF_LOADGEN_ALL_MODES=no --env.MLC_MLPERF_LOADGEN_MODE=performance --env.MLC_MLPERF_RESULT_PUSH_TO_GITHUB=False --env.MLC_MLPERF_SUBMISSION_GENERATION_STYLE=full --env.MLC_MLPERF_INFERENCE_VERSION=5.0-dev --env.MLC_RUN_MLPERF_INFERENCE_APP_DEFAULTS=r5.0-dev_default --env.MLC_MLPERF_SUBMISSION_CHECKER_VERSION=v5.0 --env.MLC_MLPERF_INFERENCE_SOURCE_VERSION=5.0.16 --env.MLC_MLPERF_LAST_RELEASE=v5.0 --env.MLC_MLPERF_INFERENCE_RESULTS_VERSION=r5.0-dev --env.MLC_MODEL=3d-unet-99 --env.MLC_MLPERF_LOADGEN_COMPLIANCE=no --env.MLC_MLPERF_LOADGEN_EXTRA_OPTIONS= --env.MLC_MLPERF_LOADGEN_SCENARIOS,=Offline --env.MLC_MLPERF_LOADGEN_MODES,=performance --env.MLC_OUTPUT_FOLDER_NAME=test_results --add_deps_recursive.coco2014-original.tags=_full --add_deps_recursive.coco2014-preprocessed.tags=_full --add_deps_recursive.imagenet-original.tags=_full --add_deps_recursive.imagenet-preprocessed.tags=_full --add_deps_recursive.openimages-original.tags=_full --add_deps_recursive.openimages-preprocessed.tags=_full --add_deps_recursive.openorca-original.tags=_full --add_deps_recursive.openorca-preprocessed.tags=_full --add_deps_recursive.coco2014-dataset.tags=_full --add_deps_recursive.igbh-dataset.tags=_full --add_deps_recursive.get-mlperf-inference-results-dir.tags=_version.r5.0-dev --add_deps_recursive.get-mlperf-inference-submission-dir.tags=_version.r5.0-dev --add_deps_recursive.mlperf-inference-nvidia-scratch-space.tags=_version.r5.0-dev --print_env=False --print_deps=False --dump_version_info=True --quiet --fake_run --env.MLC_RUN_STATE_DOCKER=True" did not complete successfully: exit code: 1
Traceback (most recent call last):
File "/home/test/mlc/bin/mlcr", line 8, in <module>
sys.exit(mlcr())
File "/home/test/mlc/lib/python3.10/site-packages/mlc/main.py", line 86, in mlcr
main()
File "/home/test/mlc/lib/python3.10/site-packages/mlc/main.py", line 242, in main
res = method(run_args)
File "/home/test/mlc/lib/python3.10/site-packages/mlc/script_action.py", line 305, in run
return self.call_script_module_function("run", run_args)
File "/home/test/mlc/lib/python3.10/site-packages/mlc/script_action.py", line 231, in call_script_module_function
result = automation_instance.run(run_args) # Pass args to the run method
File "/home/test/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 226, in run
r = self._run(i)
File "/home/test/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 1782, in _run
r = customize_code.preprocess(ii)
File "/home/test/MLC/repos/mlcommons@mlperf-automations/script/run-mlperf-inference-app/customize.py", line 285, in preprocess
r = mlc.access(ii)
File "/home/test/mlc/lib/python3.10/site-packages/mlc/action.py", line 56, in access
result = method(options)
File "/home/test/mlc/lib/python3.10/site-packages/mlc/script_action.py", line 283, in docker
return self.call_script_module_function("docker", run_args)
File "/home/test/mlc/lib/python3.10/site-packages/mlc/script_action.py", line 233, in call_script_module_function
result = automation_instance.docker(run_args) # Pass args to the run method
File "/home/test/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 4470, in docker
return docker_run(self, i)
File "/home/test/MLC/repos/mlcommons@mlperf-automations/automation/script/docker.py", line 415, in docker_run
r = self_module.action_object.access(mlc_docker_input)
File "/home/test/mlc/lib/python3.10/site-packages/mlc/action.py", line 56, in access
result = method(options)
File "/home/test/mlc/lib/python3.10/site-packages/mlc/script_action.py", line 305, in run
return self.call_script_module_function("run", run_args)
File "/home/test/mlc/lib/python3.10/site-packages/mlc/script_action.py", line 231, in call_script_module_function
result = automation_instance.run(run_args) # Pass args to the run method
File "/home/test/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 226, in run
r = self._run(i)
File "/home/test/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 1856, in _run
r = self._call_run_deps(prehook_deps, self.local_env_keys, local_env_keys_from_meta, env, state, const, const_state, add_deps_recursive,
File "/home/test/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 3317, in _call_run_deps
r = script._run_deps(deps, local_env_keys, env, state, const, const_state, add_deps_recursive, recursion_spaces,
File "/home/test/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 3485, in _run_deps
r = self.action_object.access(ii)
File "/home/test/mlc/lib/python3.10/site-packages/mlc/action.py", line 56, in access
result = method(options)
File "/home/test/mlc/lib/python3.10/site-packages/mlc/script_action.py", line 305, in run
return self.call_script_module_function("run", run_args)
File "/home/test/mlc/lib/python3.10/site-packages/mlc/script_action.py", line 241, in call_script_module_function
raise ScriptExecutionError(f"Script {function_name} execution failed. Error : {error}")
mlc.script_action.ScriptExecutionError: Script run execution failed. Error : MLC script failed (name = build-docker-image, return code = 256)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Please file an issue at https://github.com/mlcommons/mlperf-automations/issues along with the full MLC command being run and the relevant
or full console log.