Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ERROR: [RUN] /home/umair/Desktop/umair/mace/tools/device.py:290: Failed to run the model #760

Open
umairjavaid opened this issue Feb 10, 2022 · 1 comment
Open

ERROR: [RUN] /home/umair/Desktop/umair/mace/tools/device.py:290: Failed to run the model #760

umairjavaid opened this issue Feb 10, 2022 · 1 comment

Comments

@umairjavaid
Copy link

Getting the following error when trying to run mobilenet-v2 on GPU.

commands:

python tools/converter.py convert --config=mace-models/mobilenet-v2/mobilenet-v2-host.yml
python tools/converter.py run --config=mace-models/mobilenet-v2/mobilenet-v2-host.yml

mobilenet-v2-host.yml:

library_name: mobilenet-v2-host
target_abis: [host]
model_graph_format: file
model_data_format: file
models:
  mobilenet_v2_host:
    platform: tensorflow
    model_file_path: https://cnbj1.fds.api.xiaomi.com/mace/miai-models/mobilenet-v2/mobilenet-v2-1.0.pb
    model_sha256_checksum: 369f9a5f38f3c15b4311c1c84c032ce868da9f371b5f78c13d3ea3c537389bb4
    subgraphs:
      - input_tensors:
          - input
        input_shapes:
          - 1,224,224,3
        output_tensors:
          - MobilenetV2/Predictions/Reshape_1
        output_shapes:
          - 1,1001
    runtime: gpu
    limit_opencl_kernel_time: 0
    dsp_mode: 0
    obfuscate: 1
    winograd: 0

shell output:

root@umair:/home/umair/Desktop/umair/mace# python tools/converter.py run --config=mace-models/mobilenet-v2/mobilenet-v2-host.yml 
CMD> bazel build //mace/proto:mace_py
WARNING: --batch mode is deprecated. Please instead explicitly shut down your Bazel server using the command "bazel shutdown".
Loading:
Loading: 0 packages loaded
Analyzing: target //mace/proto:mace_py (12 packages loaded)
INFO: Analysed target //mace/proto:mace_py (17 packages loaded).
INFO: Found 1 target...
[0 / 15] [-----] BazelWorkspaceStatusAction stable-status.txt
Target //mace/proto:mace_py up-to-date:
bazel-genfiles/mace/proto/mace_pb2.py
INFO: Elapsed time: 1.515s, Critical Path: 0.03s
INFO: 0 processes.
INFO: Build completed successfully, 1 total action
INFO: Build completed successfully, 1 total action

CMD> cp -f bazel-genfiles/mace/proto/mace_pb2.py /home/umair/Desktop/umair/mace/tools/python/py_proto

CMD> bazel build //mace/proto:micro_mem_py
WARNING: --batch mode is deprecated. Please instead explicitly shut down your Bazel server using the command "bazel shutdown".
Loading:
Loading: 0 packages loaded
Analyzing: target //mace/proto:micro_mem_py (12 packages loaded)
INFO: Analysed target //mace/proto:micro_mem_py (17 packages loaded).
INFO: Found 1 target...
[2 / 14] [-----] BazelWorkspaceStatusAction stable-status.txt
Target //mace/proto:micro_mem_py up-to-date:
bazel-genfiles/mace/proto/micro_mem_pb2.py
INFO: Elapsed time: 1.541s, Critical Path: 0.02s
INFO: 0 processes.
INFO: Build completed successfully, 1 total action
INFO: Build completed successfully, 1 total action

CMD> cp -f bazel-genfiles/mace/proto/micro_mem_pb2.py /home/umair/Desktop/umair/mace/tools/python/py_proto

CMD> bazel build //third_party/caffe:caffe_py
WARNING: --batch mode is deprecated. Please instead explicitly shut down your Bazel server using the command "bazel shutdown".
Loading:
Loading: 0 packages loaded
Analyzing: target //third_party/caffe:caffe_py (12 packages loaded)
INFO: Analysed target //third_party/caffe:caffe_py (17 packages loaded).
INFO: Found 1 target...
[0 / 4] [-----] BazelWorkspaceStatusAction stable-status.txt
Target //third_party/caffe:caffe_py up-to-date:
bazel-genfiles/third_party/caffe/caffe_pb2.py
INFO: Elapsed time: 1.611s, Critical Path: 0.05s
INFO: 0 processes.
INFO: Build completed successfully, 1 total action
INFO: Build completed successfully, 1 total action

CMD> cp -f bazel-genfiles/third_party/caffe/caffe_pb2.py /home/umair/Desktop/umair/mace/tools/python/py_proto

CMD> bazel build //mace/proto:mace_py
WARNING: --batch mode is deprecated. Please instead explicitly shut down your Bazel server using the command "bazel shutdown".
Loading:
Loading: 0 packages loaded
Analyzing: target //mace/proto:mace_py (12 packages loaded)
INFO: Analysed target //mace/proto:mace_py (17 packages loaded).
INFO: Found 1 target...
[0 / 14] [-----] BazelWorkspaceStatusAction stable-status.txt
Target //mace/proto:mace_py up-to-date:
bazel-genfiles/mace/proto/mace_pb2.py
INFO: Elapsed time: 1.530s, Critical Path: 0.04s
INFO: 0 processes.
INFO: Build completed successfully, 1 total action
INFO: Build completed successfully, 1 total action

CMD> cp -f bazel-genfiles/mace/proto/mace_pb2.py tools/python/py_proto

CMD> bazel build //mace/proto:micro_mem_py
WARNING: --batch mode is deprecated. Please instead explicitly shut down your Bazel server using the command "bazel shutdown".
Loading:
Loading: 0 packages loaded
Analyzing: target //mace/proto:micro_mem_py (12 packages loaded)
INFO: Analysed target //mace/proto:micro_mem_py (17 packages loaded).
INFO: Found 1 target...
[0 / 4] [-----] BazelWorkspaceStatusAction stable-status.txt
Target //mace/proto:micro_mem_py up-to-date:
bazel-genfiles/mace/proto/micro_mem_pb2.py
INFO: Elapsed time: 1.591s, Critical Path: 0.03s
INFO: 0 processes.
INFO: Build completed successfully, 1 total action
INFO: Build completed successfully, 1 total action

CMD> cp -f bazel-genfiles/mace/proto/micro_mem_pb2.py tools/python/py_proto

CMD> bazel build //third_party/caffe:caffe_py
WARNING: --batch mode is deprecated. Please instead explicitly shut down your Bazel server using the command "bazel shutdown".
Loading:
Loading: 0 packages loaded
Analyzing: target //third_party/caffe:caffe_py (12 packages loaded)
INFO: Analysed target //third_party/caffe:caffe_py (17 packages loaded).
INFO: Found 1 target...
[0 / 8] [-----] BazelWorkspaceStatusAction stable-status.txt
Target //third_party/caffe:caffe_py up-to-date:
bazel-genfiles/third_party/caffe/caffe_pb2.py
INFO: Elapsed time: 1.636s, Critical Path: 0.03s
INFO: 0 processes.
INFO: Build completed successfully, 1 total action
INFO: Build completed successfully, 1 total action

CMD> cp -f bazel-genfiles/third_party/caffe/caffe_pb2.py tools/python/py_proto

* Build //mace/tools:mace_run_static with ABI host
WARNING: --batch mode is deprecated. Please instead explicitly shut down your Bazel server using the command "bazel shutdown".
write version v1.1.1-0-g8c75d39 to /root/.cache/bazel/_bazel_root/43e6d73defa22e2d485730359238d108/external/local_version_config/gen/version
INFO: Analysed target //mace/tools:mace_run_static (33 packages loaded).
INFO: Found 1 target...
[0 / 16] [-----] Creating source manifest for //mace/tools:mace_run_static
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.google.devtools.build.lib.unsafe.StringUnsafe (file:/root/.cache/bazel/_bazel_root/install/c25ea2c3043bcba07b93dde10595066c/_embedded_binaries/A-server.jar) to constructor java.lang.String(byte[],byte)
WARNING: Please consider reporting this to the maintainers of com.google.devtools.build.lib.unsafe.StringUnsafe
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
Target //mace/tools:mace_run_static up-to-date:
  bazel-bin/mace/tools/mace_run_static
INFO: Elapsed time: 46.065s, Critical Path: 12.57s
INFO: 314 processes: 314 local.
INFO: Build completed successfully, 332 total actions
('build', '--config', 'linux', '--define', 'cpu_enabled=true', '--define', 'quantize=false', '--define', 'bfloat16=false', '//mace/tools:mace_run_static', '--config', 'optimization', '--config', 'symbol_hidden')
Build done!

*******************************************************
          Run model mobilenet_v2_host on host          
*******************************************************

('The scope of generated data: ', ['-1,1'])
Generate input file:  build/mobilenet-v2-host/_tmp/mobilenet_v2_host/dc91df3d6f6a98dc0cd5d7906d5ea8db/host/model_input_input
Generate input file done.
* Run 'mobilenet_v2_host' with round=1, restart_round=1, tuning=False, out_of_range_check=False, num_threads=(-1,), cpu_affinity_policy=(1,), gpu_perf_hint=(3,), gpu_priority_hint=(3,)
I mace/tools/mace_run.cc:719] model name: mobilenet_v2_host
I mace/tools/mace_run.cc:720] mace version: v1.1.1-0-g8c75d39
I mace/tools/mace_run.cc:721] input node: input
I mace/tools/mace_run.cc:722] input shape: 1,224,224,3
I mace/tools/mace_run.cc:723] input data_type: float32
I mace/tools/mace_run.cc:724] input data_format: NHWC
I mace/tools/mace_run.cc:725] output node: MobilenetV2/Predictions/Reshape_1
I mace/tools/mace_run.cc:726] output shape: 1,1001
I mace/tools/mace_run.cc:727] output data_format: NHWC
I mace/tools/mace_run.cc:728] input_file: build/mobilenet-v2-host/_tmp/mobilenet_v2_host/dc91df3d6f6a98dc0cd5d7906d5ea8db/host/model_input
I mace/tools/mace_run.cc:729] output_file: build/mobilenet-v2-host/_tmp/mobilenet_v2_host/dc91df3d6f6a98dc0cd5d7906d5ea8db/host/model_out
I mace/tools/mace_run.cc:730] input dir: 
I mace/tools/mace_run.cc:731] output dir: 
I mace/tools/mace_run.cc:732] model_data_file: build/mobilenet-v2-host/model/mobilenet_v2_host.data
I mace/tools/mace_run.cc:733] model_file: build/mobilenet-v2-host/model/mobilenet_v2_host.pb
I mace/tools/mace_run.cc:734] accelerator_cache_policy: 0
I mace/tools/mace_run.cc:735] accelerator_binary_file: 
I mace/tools/mace_run.cc:736] accelerator_storage_file: 
I mace/tools/mace_run.cc:737] apu_boost_hint: 100
I mace/tools/mace_run.cc:738] apu_preference_hint: 1
I mace/tools/mace_run.cc:739] round: 1
I mace/tools/mace_run.cc:740] restart_round: 1
I mace/tools/mace_run.cc:741] gpu_perf_hint: 3
I mace/tools/mace_run.cc:742] gpu_priority_hint: 3
I mace/tools/mace_run.cc:743] num_threads: -1
I mace/tools/mace_run.cc:744] cpu_affinity_policy: 1
I mace/tools/mace_run.cc:786] raw_output_data_types[0] is float32
I mace/libmace/engines/serial_engine.cc:32] Creating SerialEngine, MACE version: v1.1.1-0-g8c75d39
I mace/tools/mace_run.cc:811] restart round 0
I mace/libmace/engines/serial_engine.cc:32] Creating SerialEngine, MACE version: v1.1.1-0-g8c75d39
F mace/core/runtime/runtime_registry.cc:66] Check failed: registry_.count(runtime_key) > 0 Current MACE doesn't support the runtime type. runtime_type: 2, runtime_sub_type: 0, perhaps you have specified A type runtime in yml file to convert model but specified B type runtime in yml file to run model
F mace/core/runtime/runtime_registry.cc:66] backtrace:
F mace/core/runtime/runtime_registry.cc:66]  build/mobilenet-v2-host/_tmp/host/mace_run_static() [0x5313bc]
F mace/core/runtime/runtime_registry.cc:66]  build/mobilenet-v2-host/_tmp/host/mace_run_static() [0x533ca1]
F mace/core/runtime/runtime_registry.cc:66]  build/mobilenet-v2-host/_tmp/host/mace_run_static() [0x534206]
F mace/core/runtime/runtime_registry.cc:66]  build/mobilenet-v2-host/_tmp/host/mace_run_static() [0x534254]
F mace/core/runtime/runtime_registry.cc:66]  build/mobilenet-v2-host/_tmp/host/mace_run_static() [0x4f925b]
F mace/core/runtime/runtime_registry.cc:66]  build/mobilenet-v2-host/_tmp/host/mace_run_static() [0x4e6d46]
F mace/core/runtime/runtime_registry.cc:66]  build/mobilenet-v2-host/_tmp/host/mace_run_static() [0x42f230]
F mace/core/runtime/runtime_registry.cc:66]  build/mobilenet-v2-host/_tmp/host/mace_run_static() [0x430159]
F mace/core/runtime/runtime_registry.cc:66]  build/mobilenet-v2-host/_tmp/host/mace_run_static() [0x430f96]
F mace/core/runtime/runtime_registry.cc:66]  build/mobilenet-v2-host/_tmp/host/mace_run_static() [0x431625]
F mace/core/runtime/runtime_registry.cc:66]  build/mobilenet-v2-host/_tmp/host/mace_run_static() [0x431ed3]
F mace/core/runtime/runtime_registry.cc:66]  build/mobilenet-v2-host/_tmp/host/mace_run_static() [0x432321]
F mace/core/runtime/runtime_registry.cc:66]  build/mobilenet-v2-host/_tmp/host/mace_run_static() [0x40b0e7]
F mace/core/runtime/runtime_registry.cc:66]  build/mobilenet-v2-host/_tmp/host/mace_run_static() [0x410824]
F mace/core/runtime/runtime_registry.cc:66]  build/mobilenet-v2-host/_tmp/host/mace_run_static() [0x406229]
F mace/core/runtime/runtime_registry.cc:66]  /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0) [0x7fd0f646b830]
F mace/core/runtime/runtime_registry.cc:66]  build/mobilenet-v2-host/_tmp/host/mace_run_static() [0x408749]

ERROR: [RUN] /home/umair/Desktop/umair/mace/tools/device.py:290: Failed to run the model

docker run:

nvidia-docker run -it --privileged -d --name mace-dev \
           -v /dev/bus/usb:/dev/bus/usb --net=host \
           -v $(pwd):$(pwd) \
           -v /usr/bin/docker:/usr/bin/docker \
           -v /var/run/docker.sock:/var/run/docker.sock \
           --gpus=all \
           registry.cn-hangzhou.aliyuncs.com/xiaomimace/mace-dev-lite
# Execute an interactive bash shell on the container
docker exec -it mace-dev /bin/bash

System information

  • UBUNTU 18.04:
  • MACE docker image
@lu229
Copy link
Collaborator

lu229 commented Mar 29, 2022

Perhaps you have specified A type runtime in yml file to convert model but specified B type runtime in yml file to run model

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@lu229 @umairjavaid