The test result for the inference on 3d-unet-99 model seems abnormal. #2160
Description
Hi @arjunsuresh,
it's successful to build the docker container by running the be below command but still report the errors pasted at the end.
mlcr run-mlperf,inference,_find-performance,_full,_r5.0-dev
--model=3d-unet-99
--implementation=nvidia
--framework=tensorrt
--category=edge
--scenario=Offline
--execution_mode=test
--device=cuda
--docker --quiet
--test_query_count=50
In the new-created docker container, run the below command successfully and get the final result.
_mlcr run-mlperf,inference,_full,r5.0-dev
--model=3d-unet-99
--implementation=nvidia
--framework=tensorrt
--category=edge
--scenario=Offline
--execution_mode=valid
--device=cuda
--quiet
+------------+----------+----------+------------+-----------------+ | Model | Scenario | Accuracy | Throughput | Latency (in ms) | +------------+----------+----------+------------+-----------------+ | 3d-unet-99 | Offline | 0.86234 | 2.049 | - | +------------+----------+----------+------------+-----------------+ The MLPerf inference results are stored at /home/mlcuser/MLC/repos/local/cache/get-mlperf-inference-results-dir_c369e3b3/valid_results [2025-03-27 20:50:41,091 module.py:5256 INFO] - ! call "postprocess" from /home/mlcuser/MLC/repos/mlcommons@mlperf-automations/script/run-mlperf-inference-app/customize.py mlcuser@3e124278351a:~$
I have two questions:
- I just want to figure out what the error message occurred during the docker building process is and why this error will not impact the following inference testing.
- why the final result is so bad? Accuracy < 1% and Throughput = 2, that seems abnormal and how to improve it?
"the below log is for the docker building process"
......
......
......
0, GPU 2247 (MiB)
[I] Creating batcher thread: 0 EnableBatcherThreadPerDevice: true
Finished setting up SUT.
Starting warmup. Running for a minimum of 5 seconds.
Finished warmup. Ran for 5.83001s.
Starting running actual test.
================================================
MLPerf Results Summary
================================================
SUT name : Server_3DUNet
Scenario : Offline
Mode : PerformanceOnly
Samples per second: 2.0352
Result is : VALID
Min duration satisfied : Yes
Min queries satisfied : Yes
Early stopping satisfied: Yes
================================================
Additional Stats
================================================
Min latency (ns) : 473988432
Max latency (ns) : 24567628500
Mean latency (ns) : 12511875273
50.00 percentile latency (ns) : 12985068001
90.00 percentile latency (ns) : 22595759637
95.00 percentile latency (ns) : 23949121811
97.00 percentile latency (ns) : 24327248185
99.00 percentile latency (ns) : 24567628500
99.90 percentile latency (ns) : 24567628500
================================================
Test Parameters Used
================================================
samples_per_query : 50
target_qps : 0.01
target_latency (ns): 0
max_async_queries : 1
min_duration (ms): 0
max_duration (ms): 0
min_query_count : 1
max_query_count : 50
qsl_rng_seed : 6023615788873153749
sample_index_rng_seed : 15036839855038426416
schedule_rng_seed : 9933818062894767841
accuracy_log_rng_seed : 0
accuracy_log_probability : 0
accuracy_log_sampling_target : 0
print_timestamps : 0
performance_issue_unique : 0
performance_issue_same : 0
performance_issue_same_index : 0
performance_sample_count : 43
No warnings encountered during test.
No errors encountered during test.
Finished running actual test.
Device Device:0 processed:
9 batches of size 2
2 batches of size 3
8 batches of size 4
4 batches of size 5
400 batches of size 8
Memcpy Calls: 0
PerSampleCudaMemcpy Calls: 50
BatchedCudaMemcpy Calls: 0
&&&& PASSED MLPerf_Inference_3DUNet_Harness # ./build/bin/harness_3dunet
[2025-03-27 20:00:08,401 run_harness.py:166 INFO] Result: result_samples_per_second: 2.0352, Result is VALID
======================== Result summaries: ========================
[2025-03-27 20:00:08,831 module.py:5256 INFO] - ! call "postprocess" from /home/mlcuser/MLC/repos/mlcommons@mlperf-automations/script/benchmark-program/customize.py
[2025-03-27 20:00:08,832 module.py:5319 DEBUG] - - Running postprocess ...
[2025-03-27 20:00:08,835 module.py:2200 INFO] - - running time of script "program,benchmark,benchmark
[2025-03-27 20:27:08,646 index.py:177 DEBUG] - Shared index for script saved to /home/mlcuser/MLC/repos/index_script.json.
[2025-03-27 20:27:08,647 index.py:177 DEBUG] - Shared index for cache saved to /home/mlcuser/MLC/repos/index_cache.json.
[2025-03-27 20:27:08,647 index.py:177 DEBUG] - Shared index for experiment saved to /home/mlcuser/MLC/repos/index_experiment.json.
[2025-03-27 20:27:08,651 index.py:177 DEBUG] - Shared index for script saved to /home/mlcuser/MLC/repos/index_script.json.
[2025-03-27 20:27:08,652 index.py:177 DEBUG] - Shared index for cache saved to /home/mlcuser/MLC/repos/index_cache.json.
[2025-03-27 20:27:08,652 index.py:177 DEBUG] - Shared index for experiment saved to /home/mlcuser/MLC/repos/index_experiment.json.
[2025-03-27 20:27:08,652 module.py:1402 DEBUG] - - Changing to /home/mlcuser/MLC/repos/local/cache/get-cache-dir_e3afe6f0
[2025-03-27 20:27:08,653 module.py:1762 DEBUG] - - Running preprocess ...
[2025-03-27 20:27:08,659 module.py:5256 INFO] - ! call "postprocess" from /home/mlcuser/MLC/repos/mlcommons@mlperf-automations/script/get-cache-dir/customize.py
[2025-03-27 20:27:08,660 module.py:5319 DEBUG] - - Running postprocess ...
[2025-03-27 20:27:08,673 module.py:2014 DEBUG] - - Removing tmp tag in the script cached output e3afe6f077774eed ...
[2025-03-27 20:27:08,678 index.py:177 DEBUG] - Shared index for script saved to /home/mlcuser/MLC/repos/index_script.json.
[2025-03-27 20:27:08,679 index.py:177 DEBUG] - Shared index for cache saved to /home/mlcuser/MLC/repos/index_cache.json.
[2025-03-27 20:27:08,679 index.py:177 DEBUG] - Shared index for experiment saved to /home/mlcuser/MLC/repos/index_experiment.json.
[2025-03-27 20:27:08,679 module.py:2158 INFO] - - cache UID: e3afe6f077774eed
[2025-03-27 20:27:08,679 module.py:2200 INFO] - - running time of script "get,cache,dir,directory": 0.05 sec.
[2025-03-27 20:27:08,680 module.py:1640 DEBUG] - - Processing env after dependencies ...
[2025-03-27 20:27:08,680 module.py:1762 DEBUG] - - Running preprocess ...
Generating SUT description file for 3e124278351a-tensorrt
HW description file for 3e124278351a not found. Copying from default!!!
[2025-03-27 20:27:08,687 module.py:5256 INFO] - ! call "postprocess" from /home/mlcuser/MLC/repos/mlcommons@mlperf-automations/script/get-mlperf-inference-sut-description/customize.py
[2025-03-27 20:27:08,687 module.py:5319 DEBUG] - - Running postprocess ...
[2025-03-27 20:27:08,692 module.py:2200 INFO] - - running time of script "get,mlperf,sut,description,system-under-test,system-description": 1619.82 sec.
[2025-03-27 20:27:08,693 module.py:5319 DEBUG] - - Running postprocess ...
......
......
......
[2025-03-27 20:27:18,705 module.py:1640 DEBUG] - - Processing env after dependencies ...
[2025-03-27 20:27:18,705 module.py:1762 DEBUG] - - Running preprocess ...
[2025-03-27 20:27:18,712 module.py:5102 DEBUG] - - Running native script "/home/mlcuser/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/run.sh" from temporal script "tmp-run.sh" in "/home/mlcuser/MLC/repos/local/cache/get-generic-python-lib_c7805511" ...
[2025-03-27 20:27:18,713 module.py:5109 INFO] - ! cd /home/mlcuser/MLC/repos/local/cache/get-generic-python-lib_c7805511
[2025-03-27 20:27:18,713 module.py:5110 INFO] - ! call /home/mlcuser/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/run.sh from tmp-run.sh
[2025-03-27 20:27:18,810 module.py:5256 INFO] - ! call "detect_version" from /home/mlcuser/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/customize.py
Extra PIP CMD:
[2025-03-27 20:27:18,820 module.py:5102 DEBUG] - - Running native script "/home/mlcuser/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/install.sh" from temporal script "tmp-run.sh" in "/home/mlcuser/MLC/repos/local/cache/get-generic-python-lib_c7805511" ...
[2025-03-27 20:27:18,820 module.py:5109 INFO] - ! cd /home/mlcuser/MLC/repos/local/cache/get-generic-python-lib_c7805511
[2025-03-27 20:27:18,820 module.py:5110 INFO] - ! call /home/mlcuser/MLC/repos/mlcommons@mlperf-automations/script/get-generic-python-lib/install.sh from tmp-run.sh
/usr/bin/python3 -m pip install "matplotlib"
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
/usr/share/python-wheels/urllib3-1.25.8-py2.py3-none-any.whl/urllib3/connectionpool.py:1004: InsecureRequestWarning: Unverified HTTPS request is being made to host 'pypi.ngc.nvidia.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
Collecting matplotlib
Downloading matplotlib-3.7.5-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (9.2 MB)
|████████████████████████████████| 9.2 MB 634 kB/s
Requirement already satisfied: packaging>=20.0 in /home/mlcuser/.local/lib/python3.8/site-packages (from matplotlib) (24.2)
/usr/share/python-wheels/urllib3-1.25.8-py2.py3-none-any.whl/urllib3/connectionpool.py:1004: InsecureRequestWarning: Unverified HTTPS request is being made to host 'pypi.ngc.nvidia.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
Collecting kiwisolver>=1.0.1
Downloading kiwisolver-1.4.7-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.whl (1.2 MB)
|████████████████████████████████| 1.2 MB 596 kB/s
Requirement already satisfied: importlib-resources>=3.2.0; python_version < "3.10" in /home/mlcuser/.local/lib/python3.8/site-packages (from matplotlib) (6.4.5)
Requirement already satisfied: numpy<2,>=1.20 in /home/mlcuser/.local/lib/python3.8/site-packages (from matplotlib) (1.23.5)
Requirement already satisfied: python-dateutil>=2.7 in /home/mlcuser/.local/lib/python3.8/site-packages (from matplotlib) (2.9.0.post0)
Requirement already satisfied: pillow>=6.2.0 in /home/mlcuser/.local/lib/python3.8/site-packages (from matplotlib) (10.4.0)
/usr/share/python-wheels/urllib3-1.25.8-py2.py3-none-any.whl/urllib3/connectionpool.py:1004: InsecureRequestWarning: Unverified HTTPS request is being made to host 'pypi.ngc.nvidia.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
Collecting fonttools>=4.22.0
Downloading fonttools-4.56.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.7 MB)
|████████████████████████████████| 4.7 MB 583 kB/s
/usr/share/python-wheels/urllib3-1.25.8-py2.py3-none-any.whl/urllib3/connectionpool.py:1004: InsecureRequestWarning: Unverified HTTPS request is being made to host 'pypi.ngc.nvidia.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
Collecting cycler>=0.10
Downloading cycler-0.12.1-py3-none-any.whl (8.3 kB)
/usr/share/python-wheels/urllib3-1.25.8-py2.py3-none-any.whl/urllib3/connectionpool.py:1004: InsecureRequestWarning: Unverified HTTPS request is being made to host 'pypi.ngc.nvidia.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
Collecting contourpy>=1.0.1
Downloading contourpy-1.1.1-cp38-cp38-manylinux_2_17_x[2025-03-27 20:27:08,189 module.py:1250 DEBUG] - - Checking prehook dependencies on other MLC scripts:
[2025-03-27 20:27:08,189 module.py:1263 DEBUG] - - Loading state from cached entry ...
[2025-03-27 20:27:08,189 module.py:1276 INFO] - ! load /home/mlcuser/MLC/repos/local/cache/get-python3_2ae8ade1/mlc-cached-state.json
[2025-03-27 20:27:08,189 module.py:1311 DEBUG] - - Checking posthook dependencies on other MLC scripts:
[2025-03-27 20:27:08,189 module.py:1324 DEBUG] - - Checking post dependencies on other MLC scripts:
......
......
......
automations/script/draw-graph-from-json-data/run.sh from tmp-run.sh
Running:
/usr/bin/python3 '/home/mlcuser/MLC/repos/mlcommons@mlperf-automations/script/draw-graph-from-json-data/process-mlc-deps.py' '/home/mlcuser/MLC/repos/local/cache/get-mlperf-inference-results-dir_c369e3b3/test_results/3e124278351a-nvidia_original-gpu-tensorrt-vdefault-default_config/3d-unet-99/offline/performance/run_1/mlc-version-info.json' --output_image '/home/mlcuser/MLC/repos/local/cache/get-mlperf-inference-results-dir_c369e3b3/test_results/3e124278351a-nvidia_original-gpu-tensorrt-vdefault-default_config/3d-unet-99/offline/performance/run_1/mlc-deps.png' --output_mermaid '/home/mlcuser/MLC/repos/local/cache/get-mlperf-inference-results-dir_c369e3b3/test_results/3e124278351a-nvidia_original-gpu-tensorrt-vdefault-default_config/3d-unet-99/offline/performance/run_1/mlc-deps.mmd'
Graph visualization saved as /home/mlcuser/MLC/repos/local/cache/get-mlperf-inference-results-dir_c369e3b3/test_results/3e124278351a-nvidia_original-gpu-tensorrt-vdefault-default_config/3d-unet-99/offline/performance/run_1/mlc-deps.png
Mermaid syntax saved to /home/mlcuser/MLC/repos/local/cache/get-mlperf-inference-results-dir_c369e3b3/test_results/3e124278351a-nvidia_original-gpu-tensorrt-vdefault-default_config/3d-unet-99/offline/performance/run_1/mlc-deps.mmd
Graph data saved as graph.graphml
[2025-03-27 20:28:06,065 module.py:5256 INFO] - ! call "postprocess" from /home/mlcuser/MLC/repos/mlcommons@mlperf-automations/script/draw-graph-from-json-data/customize.py
[2025-03-27 20:28:06,065 module.py:5319 DEBUG] - - Running postprocess ...
[2025-03-27 20:28:06,071 module.py:2200 INFO] - - running time of script "draw,graph,from-json,from-json-data": 56.27 sec.
[2025-03-27 20:28:06,072 module.py:3638 INFO] - mlcr detect,os
[2025-03-27 20:28:06,073 module.py:3638 INFO] - mlcr get,sys-utils-cm
[2025-03-27 20:28:06,073 module.py:3638 INFO] - mlcr get,python
[2025-03-27 20:28:06,073 module.py:3638 INFO] - mlcr get,mlcommons,inference,src
[2025-03-27 20:28:06,073 module.py:3638 INFO] - mlcr get,mlperf,inference,utils
[2025-03-27 20:28:06,073 module.py:3638 INFO] - mlcr get,cuda-devices,_with-pycuda
[2025-03-27 20:28:06,073 module.py:3638 INFO] - mlcr reproduce,mlperf,nvidia,inference,_run_harness,_cuda,_tensorrt,_offline,_3d-unet-99,_gpu_memory.48
[2025-03-27 20:28:06,073 module.py:3638 INFO] - mlcr get,mlperf,sut,description
[2025-03-27 20:28:06,073 module.py:3638 INFO] - mlcr draw,graph,from-json
[2025-03-27 20:28:06,076 module.py:2200 INFO] - - running time of script "app,vision,language,mlcommons,mlperf,inference,generic": 10887.38 sec.
mlcuser@3e124278351a:~$ mlcr run-mlperf,inference,_find-performance,_full,_r5.0-dev \
> --model=3d-unet-99 \
> --implementation=nvidia \
> --framework=tensorrt \
> --category=edge \
> --scenario=Offline \
> --execution_mode=test \
> --device=cuda \
> --docker --quiet \
> --test_query_count=50mlcr run-mlperf,inference,_find-performance,_full,_r5.0-dev \
> --model=3d-unet-99 \
> --implementation=nvidia \
> --framework=tensorrt \
> --category=edge \
> --scenario=Offline \
> --execution_mode=test \
> --device=cuda \
> --docker --quiet \
> --test_query_count=50mlcr run-mlperf,inference,_find-performance,_full,_r5.0-dev \
> --model=3d-unet-99 \
> --implementation=nvidia \
> --framework=tensorrt \
> --category=edge \
> --scenario=Offline \
> --execution_mode=test \
> --device=cuda \
> --docker --quiet \
> --test_query_count=50mlcr run-mlperf,inference,_find-performance,_full,_r5.0-dev \
> --model=3d-unet-99 \
> --implementation=nvidia \
> --framework=tensorrt \
> --category=edge \
> --scenario=Offline \
> --execution_mode=test \
> --device=cuda \
> --docker --quiet \
> --test_query_count=50mlcr run-mlperf,inference,_find-performance,_full,_r5.0-dev \
> --model=3d-unet-99 \
> --implementation=nvidia \
> --framework=tensorrt \
> --category=edge \
> --scenario=Offline \
> --execution_mode=test \
> --device=cuda \
> --docker --quiet \
> --test_query_count=50mlcr run-mlperf,inference,_find-performance,_full,_r5.0-dev \
> --model=3d-unet-99 \
> --implementation=nvidia \
> --framework=tensorrt \
> --category=edge \
> --scenario=Offline \
> --execution_mode=test \
> --device=cuda \
> --docker --quiet \
> --test_query_count=50mlcr run-mlperf,inference,_find-performance,_full,_r5.0-dev \
> --model=3d-unet-99 \
> --implementation=nvidia \
> --framework=tensorrt \
> --category=edge \
> --scenario=Offline \
> --execution_mode=test \
> --device=cuda \
> --docker --quiet \
> --test_query_count=50mlcr run-mlperf,inference,_find-performance,_full,_r5.0-dev \
> --model=3d-unet-99 \
> --implementation=nvidia \
> --framework=tensorrt \
> --category=edge \
> --scenario=Offline \
> --execution_mode=test \
> --device=cuda \
> --docker --quiet \
> --test_query_count=5013522989755
[2025-03-27 20:28:07,460 module.py:561 INFO] - * mlcr run-mlperf,inference,_find-performance,_full,_r5.0-dev
[2025-03-27 20:28:07,475 module.py:561 INFO] - * mlcr get,mlcommons,inference,src
[2025-03-27 20:28:07,475 module.py:1276 INFO] - ! load /home/mlcuser/MLC/repos/local/cache/get-mlperf-inference-src_9b6bb300/mlc-cached-state.json
[2025-03-27 20:28:07,484 module.py:561 INFO] - * mlcr get,mlperf,inference,results,dir,_version.r5.0-dev
[2025-03-27 20:28:07,485 module.py:1276 INFO] - ! load /home/mlcuser/MLC/repos/local/cache/get-mlperf-inference-results-dir_c369e3b3/mlc-cached-state.json
[2025-03-27 20:28:07,491 module.py:561 INFO] - * mlcr install,pip-package,for-mlc-python,_package.tabulate
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Requirement already satisfied: tabulate in /home/mlcuser/.local/lib/python3.8/site-packages (0.9.0)
[2025-03-27 20:28:09,145 module.py:5256 INFO] - ! call "postprocess" from /home/mlcuser/MLC/repos/mlcommons@mlperf-automations/script/install-pip-package-for-mlc-python/customize.py
[2025-03-27 20:28:09,165 module.py:561 INFO] - * mlcr get,mlperf,inference,utils
[2025-03-27 20:28:09,179 module.py:561 INFO] - * mlcr get,mlperf,inference,src
[2025-03-27 20:28:09,180 module.py:1276 INFO] - ! load /home/mlcuser/MLC/repos/local/cache/get-mlperf-inference-src_9b6bb300/mlc-cached-state.json
[2025-03-27 20:28:09,182 module.py:5256 INFO] - ! call "postprocess" from /home/mlcuser/MLC/repos/mlcommons@mlperf-automations/script/get-mlperf-inference-utils/customize.py
Using MLCommons Inference source from /home/mlcuser/MLC/repos/local/cache/get-git-repo_0a774c1a/inference
Running loadgen scenario: Offline and mode: performance
[2025-03-27 20:28:09,282 module.py:561 INFO] - * mlcr detect,os
[2025-03-27 20:28:09,284 module.py:5109 INFO] - ! cd /home/mlcuser
[2025-03-27 20:28:09,284 module.py:5110 INFO] - ! call /home/mlcuser/MLC/repos/mlcommons@mlperf-automations/script/detect-os/run.sh from tmp-run.sh
[2025-03-27 20:28:09,320 module.py:5256 INFO] - ! call "postprocess" from /home/mlcuser/MLC/repos/mlcommons@mlperf-automations/script/detect-os/customize.py
[2025-03-27 20:28:09,340 module.py:561 INFO] - * mlcr build,dockerfile
[2025-03-27 20:28:09,350 module.py:561 INFO] - * mlcr get,docker
[2025-03-27 20:28:09,366 module.py:561 INFO] - * mlcr detect,os
[2025-03-27 20:28:09,369 module.py:5109 INFO] - ! cd /home/mlcuser/MLC/repos/local/cache/get-docker_4530afcd
[2025-03-27 20:28:09,369 module.py:5110 INFO] - ! call /home/mlcuser/MLC/repos/mlcommons@mlperf-automations/script/detect-os/run.sh from tmp-run.sh
[2025-03-27 20:28:09,406 module.py:5256 INFO] - ! call "postprocess" from /home/mlcuser/MLC/repos/mlcommons@mlperf-automations/script/detect-os/customize.py
[2025-03-27 20:28:09,439 module.py:5109 INFO] - ! cd /home/mlcuser/MLC/repos/local/cache/get-docker_4530afcd
[2025-03-27 20:28:09,439 module.py:5110 INFO] - ! call /home/mlcuser/MLC/repos/mlcommons@mlperf-automations/script/get-docker/install-ubuntu.sh from tmp-run.sh
Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease
Get:2 http://security.ubuntu.com/ubuntu focal-security InRelease [128 kB]
Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [128 kB]
Get:5 http://security.ubuntu.com/ubuntu focal-security/ms already cached ...ultiverse amd64 Packages [33.2 kB]
Get:4 https://developer.download.nvidia.cn/compute/cuda/repos/ubuntu2004/x86_64 InRelease [1581 B]
Get:6 http://archive.ubuntu.com/ubuntu focal-backports InRelease [128 kB]
Get:7 https://developer.download.nvidia.cn/compute/cuda/repos/ubuntu2004/x86_64 Packages [2068 kB]
Get:8 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [4291 kB]
Get:9 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [36.8 kB]
Get:10 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [1597 kB]
Get:11 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [4688 kB]
Get:12 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [1304 kB]
Get:13 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [4491 kB]
Get:14 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [4777 kB]
Fetched 23.7 MB in 5s (4425 kB/s)
Reading package lists... Done
sudo apt-get install -y ca-certificates curl gnupg
Reading package lists... Done
Building dependency tree
Reading state information... Done
curl is already the newest version (7.68.0-1ubuntu2.25).
gnupg is already the newest version (2.2.19-3ubuntu2.2).
gnupg set to manually installed.
The following packages will be upgraded:
ca-certificates
1 upgraded, 0 newly installed, 0 to remove and 76 not upgraded.
Need to get 159 kB of archives.
After this operation, 9216 B of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 ca-certificates all 20240203~20.04.1 [159 kB]
Fetched 159 kB in 2s (90.4 kB/s)
debconf: unable to initialize frontend: Dialog
debconf: (No usable dialog-like program is installed, so the dialog based frontend cannot be used. at /usr/share/perl5/Debconf/FrontEnd/Dialog.pm line 76, <> line 1.)
debconf: falling back to frontend: Readline
Preconfiguring packages ...
(Reading database ... 47541 files and directories currently installed.)
Preparing to unpack .../ca-certificates_20240203~20.04.1_all.deb ...
Unpacking ca-certificates (20240203~20.04.1) over (20230311ubuntu0.20.04.1) ...
Setting up ca-certificates (20240203~20.04.1) ...
debconf: unable to initialize frontend: Dialog
debconf: (No usable dialog-like program is installed, so the dialog based frontend cannot be used. at /usr/share/perl5/Debconf/FrontEnd/Dialog.pm line 76.)
debconf: falling back to frontend: Readline
Updating certificates in /etc/ssl/certs...
rehash: warning: skipping ca-certificates.crt,it does not contain exactly one certificate or CRL
14 added, 5 removed; done.
Processing triggers for ca-certificates (20240203~20.04.1) ...
Updating certificates in /etc/ssl/certs...
0 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...
done.
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
Ign:1 https://download.docker.com/linux/ubuntu focal InRelease
Err:2 https://download.docker.com/linux/ubuntu focal Release
Could not handshake: Error in the pull function. [IP: 18.172.31.124 443]
Hit:3 http://archive.ubuntu.com/ubuntu focal InRelease
Hit:4 http://security.ubuntu.com/ubuntu focal-security InRelease
Hit:5 http://archive.ubuntu.com/ubuntu focal-updates InRelease
Hit:6 http://archive.ubuntu.com/ubuntu focal-backports InRelease
Hit:7 https://developer.download.nvidia.cn/compute/cuda/repos/ubuntu2004/x86_64 InRelease
Reading package lists... Done
E: The repository 'https://download.docker.com/linux/ubuntu focal Release' does not have a Release file.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.
sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
Reading package lists... Done
Building dependency tree
Reading state information... Done
Package docker-ce is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source
E: Package 'docker-ce' has no installation candidate
E: Unable to locate package docker-ce-cli
E: Unable to locate package containerd.io
E: Couldn't find any package by glob 'containerd.io'
E: Couldn't find any package by regex 'containerd.io'
E: Unable to locate package docker-buildx-plugin
E: Unable to locate package docker-compose-plugin
Traceback (most recent call last):
File "/home/mlcuser/.local/bin/mlcr", line 8, in <module>
sys.exit(mlcr())
File "/home/mlcuser/.local/lib/python3.8/site-packages/mlc/main.py", line 86, in mlcr
main()
File "/home/mlcuser/.local/lib/python3.8/site-packages/mlc/main.py", line 173, in main
res = method(run_args)
File "/home/mlcuser/.local/lib/python3.8/site-packages/mlc/script_action.py", line 141, in run
return self.call_script_module_function("run", run_args)
File "/home/mlcuser/.local/lib/python3.8/site-packages/mlc/script_action.py", line 121, in call_script_module_function
result = automation_instance.run(run_args) # Pass args to the run method
File "/home/mlcuser/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 225, in run
r = self._run(i)
File "/home/mlcuser/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 1776, in _run
r = customize_code.preprocess(ii)
File "/home/mlcuser/MLC/repos/mlcommons@mlperf-automations/script/run-mlperf-inference-app/customize.py", line 284, in preprocess
r = mlc.access(ii)
File "/home/mlcuser/.local/lib/python3.8/site-packages/mlc/action.py", line 56, in access
result = method(options)
File "/home/mlcuser/.local/lib/python3.8/site-packages/mlc/script_action.py", line 138, in docker
return self.call_script_module_function("docker", run_args)
File "/home/mlcuser/.local/lib/python3.8/site-packages/mlc/script_action.py", line 123, in call_script_module_function
result = automation_instance.docker(run_args) # Pass args to the run method
File "/home/mlcuser/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 4460, in docker
return docker_run(self, i)
File "/home/mlcuser/MLC/repos/mlcommons@mlperf-automations/automation/script/docker.py", line 226, in docker_run
r = dockerfile(self_module, i)
File "/home/mlcuser/MLC/repos/mlcommons@mlperf-automations/automation/script/docker.py", line 187, in dockerfile
dockerfile_result = self_module.action_object.access(mlc_docker_input)
File "/home/mlcuser/.local/lib/python3.8/site-packages/mlc/action.py", line 56, in access
result = method(options)
File "/home/mlcuser/.local/lib/python3.8/site-packages/mlc/script_action.py", line 141, in run
return self.call_script_module_function("run", run_args)
File "/home/mlcuser/.local/lib/python3.8/site-packages/mlc/script_action.py", line 121, in call_script_module_function
result = automation_instance.run(run_args) # Pass args to the run method
File "/home/mlcuser/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 225, in run
r = self._run(i)
File "/home/mlcuser/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 1634, in _run
r = self._call_run_deps(deps, self.local_env_keys, local_env_keys_from_meta, env, state, const, const_state, add_deps_recursive,
File "/home/mlcuser/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 3301, in _call_run_deps
r = script._run_deps(deps, local_env_keys, env, state, const, const_state, add_deps_recursive, recursion_spaces,
File "/home/mlcuser/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 3471, in _run_deps
r = self.action_object.access(ii)
File "/home/mlcuser/.local/lib/python3.8/site-packages/mlc/action.py", line 56, in access
result = method(options)So I just want to figure out what the error message occurred during the docker building process is and why this error will not impact the following valid inference testing.
File "/home/mlcuser/.local/lib/python3.8/site-packages/mlc/script_action.py", line 141, in run
return self.call_script_module_function("run", run_args)
File "/home/mlcuser/.local/lib/python3.8/site-packages/mlc/script_action.py", line 131, in call_script_module_function
raise ScriptExecutionError(f"Script {function_name} execution failed. Error : {error}")
mlc.script_action.ScriptExecutionError: Script run execution failed. Error : MLC script failed (name = get-docker, return code = 256)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Please file an issue at https://github.com/mlcommons/mlperf-automations/issues along witSo I just want to figure out what the error message occurred during the docker building process is and why this error will not impact the following valid inference testing.h the full MLC command being run and the relevant
or full console log.
mlcuser@3e124278351a:~$