Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CAFFE validation fails when using WINOGRAD 4 #603

Closed
gasgallo opened this issue Mar 2, 2020 · 6 comments
Closed

CAFFE validation fails when using WINOGRAD 4 #603

gasgallo opened this issue Mar 2, 2020 · 6 comments

Comments

@gasgallo
Copy link
Contributor

gasgallo commented Mar 2, 2020 •
edited

Before you open an issue, please make sure you have tried the following steps:

  1. Make sure your environment is the same with (https://mace.readthedocs.io/en/latest/installation/env_requirement.html).
  2. Have you ever read the document for your usage?
  3. Check if your issue appears in HOW-TO-DEBUG or FAQ.
  4. The form below must be filled.

System information

  • OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 16.04
  • NDK version(e.g., 15c): 18b
  • GCC version(if compiling for host, e.g., 5.4.0): 5.4.0
  • MACE version (Use the command: git describe --long --tags): 0.12.0
  • Python version(2.7): 3.6
  • Bazel version (e.g., 0.13.0): 0.16.0
  • CMake version: 3.16.0

Model deploy file (*.yml)

# The name of library
library_name: FD
target_abis: [arm64-v8a]
model_graph_format: file
model_data_format: file
models:
  RF: # model tag, which will be used in model loading and must be specific.
    platform: caffe
    # path to your tensorflow model's pb file. Support local path, http:// and https://
    model_file_path: /models/model.prototxt
    weight_file_path: /models/model.caffemodel
    # sha256_checksum of your model's pb file.
    # use this command to get the sha256_checksum --> sha256sum path/to/your/pb/file
    model_sha256_checksum: 98f9b69a085e7d8f40704ac6b2fedae0fda876fff4658509dde3d74d883a9684
    weight_sha256_checksum: a9f5d4dfe944315511c6070e8556790409ae0f0bd9005c5db66b4fdd5c38b716
    subgraphs:
      - input_tensors:
          - data
        input_shapes:
          - 1,3,112,112
        input_data_formats:
          - NCHW
        output_tensors:
          - fc1
        output_shapes:
          - 1,1,1,512
    obfuscate: 0
    runtime: gpu
    winograd: 4

Describe the problem

Validation fails because of low similarity. This happens only when WINOGRAD is 4. With WINOGRAD 0 or 2, validation is successful.

To Reproduce

Steps to reproduce the problem:

1. cd /path/to/mace
2. python tools/converter.py convert --config_file=/path/to/your/model_deployment_file
2. python tools/converter.py run --validate --disable_tuning --config_file=/path/to/your/model_deployment_file

Error information / logs

Please include the full log and/or traceback here.

MACE v0.12.0 w/ WINOGRAD 0:

root@ds017:/mace# python tools/converter.py run --validate --disable_tuning --config /models/model/model.yml 
CMD> bazel build //mace/proto:mace_py
WARNING: --batch mode is deprecated. Please instead explicitly shut down your Bazel server using the command "bazel shutdown".
Loading:
Loading: 0 packages loaded
Analyzing: target //mace/proto:mace_py (6 packages loaded)
INFO: Analysed target //mace/proto:mace_py (17 packages loaded).
INFO: Found 1 target...
[2 / 8] [-----] BazelWorkspaceStatusAction stable-status.txt
Target //mace/proto:mace_py up-to-date:
bazel-genfiles/mace/proto/mace_pb2.py
INFO: Elapsed time: 2.382s, Critical Path: 0.06s
INFO: 0 processes.
INFO: Build completed successfully, 1 total action
INFO: Build completed successfully, 1 total action

CMD> cp -f bazel-genfiles/mace/proto/mace_pb2.py tools/python/py_proto

CMD> bazel build //third_party/caffe:caffe_py
WARNING: --batch mode is deprecated. Please instead explicitly shut down your Bazel server using the command "bazel shutdown".
Loading:
Loading: 0 packages loaded
Analyzing: target //third_party/caffe:caffe_py (6 packages loaded)
INFO: Analysed target //third_party/caffe:caffe_py (17 packages loaded).
INFO: Found 1 target...
[0 / 4] [-----] BazelWorkspaceStatusAction stable-status.txt
Target //third_party/caffe:caffe_py up-to-date:
bazel-genfiles/third_party/caffe/caffe_pb2.py
INFO: Elapsed time: 2.339s, Critical Path: 0.06s
INFO: 0 processes.
INFO: Build completed successfully, 1 total action
INFO: Build completed successfully, 1 total action

CMD> cp -f bazel-genfiles/third_party/caffe/caffe_pb2.py tools/python/py_proto

* Build //mace/tools:mace_run_static with ABI arm64-v8a
WARNING: --batch mode is deprecated. Please instead explicitly shut down your Bazel server using the command "bazel shutdown".
WARNING: The major revision of the Android NDK referenced by android_ndk_repository rule 'androidndk' is 19. The major revisions supported by Bazel are [10, 11, 12, 13, 14, 15, 16]. Bazel will attempt to treat the NDK as if it was r16. This may cause compilation and linkage problems. Please download a supported NDK version.
INFO: Analysed target //mace/tools:mace_run_static (32 packages loaded).
INFO: Found 1 target...
Target //mace/tools:mace_run_static up-to-date:
  bazel-bin/mace/tools/mace_run_static
INFO: Elapsed time: 11.926s, Critical Path: 0.60s
INFO: 0 processes.
INFO: Build completed successfully, 1 total action
('build', '//mace/tools:mace_run_static', '--config', 'android', '--cpu=arm64-v8a', '--define', 'neon=true', '--define', 'openmp=false', '--define', 'opencl=true', '--define', 'quantize=false', '--define', 'hexagon=false', '--define', 'hta=false', '--define', 'apu=false', '--config', 'optimization', '--config', 'symbol_hidden')
Build done!

*********************************************
          Run model model on MI9          
*********************************************

Generate input file:  build/model/_tmp/model/2ae7f4ae46d9383742a58f200f15a863/MI9_msmnile/arm64-v8a/model_input_data
Generate input file done.
* Run 'model' with round=1, restart_round=1, tuning=False, out_of_range_check=False, omp_num_threads=(-1,), cpu_affinity_policy=(1,), gpu_perf_hint=(3,), gpu_priority_hint=(3,)
Push build/model/_tmp/model/2ae7f4ae46d9383742a58f200f15a863/MI9_msmnile/arm64-v8a/model_input_data to /data/local/tmp/mace_run
Push build/model/model/model.data to /data/local/tmp/mace_run
Push build/model/model/model.pb to /data/local/tmp/mace_run/model.pb
Push build/model/_tmp/arm64-v8a/mace_run_static to /data/local/tmp/mace_run
Push /tmp/cmd_file-model-1583130719.9408345 to /data/local/tmp/mace_run/cmd_file-model-1583130719.9408345
I mace/tools/mace_run.cc:527] model name: model
I mace/tools/mace_run.cc:528] mace version: v0.12.0-0-ga610d50
I mace/tools/mace_run.cc:529] input node: data
I mace/tools/mace_run.cc:530] input shape: 1,3,112,112
I mace/tools/mace_run.cc:531] output node: fc1
I mace/tools/mace_run.cc:532] output shape: 1,1,1,512
I mace/tools/mace_run.cc:533] input_file: /data/local/tmp/mace_run/model_input
I mace/tools/mace_run.cc:534] output_file: /data/local/tmp/mace_run/model_out
I mace/tools/mace_run.cc:535] input dir:
I mace/tools/mace_run.cc:536] output dir:
I mace/tools/mace_run.cc:537] model_data_file: /data/local/tmp/mace_run/model.data
I mace/tools/mace_run.cc:538] model_file: /data/local/tmp/mace_run/model.pb
I mace/tools/mace_run.cc:539] device: CPU
I mace/tools/mace_run.cc:540] round: 1
I mace/tools/mace_run.cc:541] restart_round: 1
I mace/tools/mace_run.cc:542] gpu_perf_hint: 3
I mace/tools/mace_run.cc:543] gpu_priority_hint: 3
I mace/tools/mace_run.cc:544] omp_num_threads: -1
I mace/tools/mace_run.cc:545] cpu_affinity_policy: 1
I mace/tools/mace_run.cc:548] limit_opencl_kernel_time: 0
I mace/tools/mace_run.cc:553] opencl_queue_window_size: 0
I mace/libmace/mace.cc:464] Creating MaceEngine, MACE version: v0.12.0-0-ga610d50
I mace/libmace/mace.cc:503] Initializing MaceEngine
I mace/libmace/mace.cc:636] Destroying MaceEngine
I mace/tools/mace_run.cc:596] restart round 0
I mace/libmace/mace.cc:911] Create MaceEngine from model graph proto and weights data
I mace/libmace/mace.cc:464] Creating MaceEngine, MACE version: v0.12.0-0-ga610d50
I mace/libmace/mace.cc:503] Initializing MaceEngine
I mace/tools/mace_run.cc:269] Create Mace Engine latency: 117.143 ms
I mace/tools/mace_run.cc:276] Total init latency: 117.283 ms
I mace/tools/mace_run.cc:370] Warm up run
I mace/tools/mace_run.cc:406] 1st warm up run latency: 290.997 ms
I mace/tools/mace_run.cc:414] Run model
I mace/tools/mace_run.cc:476] Average latency: 124.496 ms
I mace/tools/mace_run.cc:491] Write output file /data/local/tmp/mace_run/model_out_fc1 with size 2048 done.
========================================================
     capability(CPU)        init      warmup     run_avg
========================================================
time          19.875     117.283     290.997     124.496
I mace/libmace/mace.cc:636] Destroying MaceEngine
Running finished!

* Validate with caffe
Pull /data/local/tmp/mace_run/model_out_fc1 to build/model/_tmp/model/2ae7f4ae46d9383742a58f200f15a863/MI9_msmnile/arm64-v8a
fc1 MACE VS CAFFE similarity: 0.999993100559901 , sqnr: 72424.4089891457 , pixel_accuracy: 1.0
******************************************
          Similarity Test Passed          
******************************************

Validation done!

Dana service is not available.
* Run 'model' with round=1, restart_round=1, tuning=False, out_of_range_check=False, omp_num_threads=(-1,), cpu_affinity_policy=(1,), gpu_perf_hint=(3,), gpu_priority_hint=(3,)
Push build/model/_tmp/model/2ae7f4ae46d9383742a58f200f15a863/MI9_msmnile/arm64-v8a/model_input_data to /data/local/tmp/mace_run
Push build/model/model/model.data to /data/local/tmp/mace_run
Push build/model/model/model.pb to /data/local/tmp/mace_run/model.pb
Push build/model/_tmp/arm64-v8a/mace_run_static to /data/local/tmp/mace_run
Push /tmp/cmd_file-model-1583130766.2653663 to /data/local/tmp/mace_run/cmd_file-model-1583130766.2653663
I mace/tools/mace_run.cc:527] model name: model
I mace/tools/mace_run.cc:528] mace version: v0.12.0-0-ga610d50
I mace/tools/mace_run.cc:529] input node: data
I mace/tools/mace_run.cc:530] input shape: 1,3,112,112
I mace/tools/mace_run.cc:531] output node: fc1
I mace/tools/mace_run.cc:532] output shape: 1,1,1,512
I mace/tools/mace_run.cc:533] input_file: /data/local/tmp/mace_run/model_input
I mace/tools/mace_run.cc:534] output_file: /data/local/tmp/mace_run/model_out
I mace/tools/mace_run.cc:535] input dir:
I mace/tools/mace_run.cc:536] output dir:
I mace/tools/mace_run.cc:537] model_data_file: /data/local/tmp/mace_run/model.data
I mace/tools/mace_run.cc:538] model_file: /data/local/tmp/mace_run/model.pb
I mace/tools/mace_run.cc:539] device: GPU
I mace/tools/mace_run.cc:540] round: 1
I mace/tools/mace_run.cc:541] restart_round: 1
I mace/tools/mace_run.cc:542] gpu_perf_hint: 3
I mace/tools/mace_run.cc:543] gpu_priority_hint: 3
I mace/tools/mace_run.cc:544] omp_num_threads: -1
I mace/tools/mace_run.cc:545] cpu_affinity_policy: 1
I mace/tools/mace_run.cc:548] limit_opencl_kernel_time: 0
I mace/tools/mace_run.cc:553] opencl_queue_window_size: 0
I mace/libmace/mace.cc:464] Creating MaceEngine, MACE version: v0.12.0-0-ga610d50
I mace/libmace/mace.cc:503] Initializing MaceEngine
I mace/libmace/mace.cc:636] Destroying MaceEngine
I mace/tools/mace_run.cc:596] restart round 0
W ./mace/utils/tuner.h:201] Failed to read tuned param file: /data/local/tmp/mace_run/model_tuned_opencl_parameter.MI9.msmnile.bin
I mace/libmace/mace.cc:911] Create MaceEngine from model graph proto and weights data
I mace/libmace/mace.cc:464] Creating MaceEngine, MACE version: v0.12.0-0-ga610d50
W mace/core/kv_storage.cc:109] Failed to read kv store file: /data/local/tmp/mace_run/interior//mace_cl_compiled_program.bin
W mace/core/runtime/opencl/opencl_runtime.cc:382] Load OpenCL cached compiled kernel file failed. Please make sure the storage directory exist and you have Write&Read permission
I mace/libmace/mace.cc:503] Initializing MaceEngine
I mace/tools/mace_run.cc:269] Create Mace Engine latency: 658.632 ms
I mace/tools/mace_run.cc:276] Total init latency: 658.807 ms
I mace/tools/mace_run.cc:370] Warm up run
I mace/tools/mace_run.cc:406] 1st warm up run latency: 724.763 ms
I mace/tools/mace_run.cc:414] Run model
I mace/tools/mace_run.cc:476] Average latency: 64.708 ms
I mace/tools/mace_run.cc:491] Write output file /data/local/tmp/mace_run/model_out_fc1 with size 2048 done.
========================================================
     capability(CPU)        init      warmup     run_avg
========================================================
time          21.066     658.807     724.763      64.708
I mace/libmace/mace.cc:636] Destroying MaceEngine
Running finished!

* Validate with caffe
Pull /data/local/tmp/mace_run/model_out_fc1 to build/model/_tmp/model/2ae7f4ae46d9383742a58f200f15a863/MI9_msmnile/arm64-v8a
fc1 MACE VS CAFFE similarity: 0.9996721272934489 , sqnr: 1507.9718956868915 , pixel_accuracy: 1.0
******************************************
          Similarity Test Passed          
******************************************

Validation done!

Elapse time: 1.577173 minutes.
* Package libs for model
Start packaging 'model' libs into build/model/libmace_model.tar.gz
build/model/model/
build/model/model/model.pb
build/model/model/cpu/
build/model/model/gpu/
build/model/model/model.data
Packaging Done!

------------------------------------------------------------
                          Library                           
------------------------------------------------------------
|          key           |              value              |
============================================================
| MACE Model package Path| build/model/libmace_model.tar.gz|
------------------------------------------------------------

MACE v0.12.0 w/ WINOGRAD 2:

root@ds017:/mace# python tools/converter.py run --validate --disable_tuning --config /models/model/model.yml 
CMD> bazel build //mace/proto:mace_py
WARNING: --batch mode is deprecated. Please instead explicitly shut down your Bazel server using the command "bazel shutdown".
Loading:
Loading: 0 packages loaded
Analyzing: target //mace/proto:mace_py (6 packages loaded)
INFO: Analysed target //mace/proto:mace_py (17 packages loaded).
INFO: Found 1 target...
[3 / 8] [-----] BazelWorkspaceStatusAction stable-status.txt
Target //mace/proto:mace_py up-to-date:
bazel-genfiles/mace/proto/mace_pb2.py
INFO: Elapsed time: 2.346s, Critical Path: 0.06s
INFO: 0 processes.
INFO: Build completed successfully, 1 total action
INFO: Build completed successfully, 1 total action

CMD> cp -f bazel-genfiles/mace/proto/mace_pb2.py tools/python/py_proto

CMD> bazel build //third_party/caffe:caffe_py
WARNING: --batch mode is deprecated. Please instead explicitly shut down your Bazel server using the command "bazel shutdown".
Loading:
Loading: 0 packages loaded
Analyzing: target //third_party/caffe:caffe_py (6 packages loaded)
INFO: Analysed target //third_party/caffe:caffe_py (17 packages loaded).
INFO: Found 1 target...
[0 / 1] [-----] BazelWorkspaceStatusAction stable-status.txt
Target //third_party/caffe:caffe_py up-to-date:
bazel-genfiles/third_party/caffe/caffe_pb2.py
INFO: Elapsed time: 2.520s, Critical Path: 0.07s
INFO: 0 processes.
INFO: Build completed successfully, 1 total action
INFO: Build completed successfully, 1 total action

CMD> cp -f bazel-genfiles/third_party/caffe/caffe_pb2.py tools/python/py_proto

* Build //mace/tools:mace_run_static with ABI arm64-v8a
WARNING: --batch mode is deprecated. Please instead explicitly shut down your Bazel server using the command "bazel shutdown".
WARNING: The major revision of the Android NDK referenced by android_ndk_repository rule 'androidndk' is 19. The major revisions supported by Bazel are [10, 11, 12, 13, 14, 15, 16]. Bazel will attempt to treat the NDK as if it was r16. This may cause compilation and linkage problems. Please download a supported NDK version.
INFO: Analysed target //mace/tools:mace_run_static (32 packages loaded).
INFO: Found 1 target...
Target //mace/tools:mace_run_static up-to-date:
  bazel-bin/mace/tools/mace_run_static
INFO: Elapsed time: 16.210s, Critical Path: 0.49s
INFO: 0 processes.
INFO: Build completed successfully, 1 total action
('build', '//mace/tools:mace_run_static', '--config', 'android', '--cpu=arm64-v8a', '--define', 'neon=true', '--define', 'openmp=false', '--define', 'opencl=true', '--define', 'quantize=false', '--define', 'hexagon=false', '--define', 'hta=false', '--define', 'apu=false', '--config', 'optimization', '--config', 'symbol_hidden')
Build done!

*********************************************
          Run model model on MI9          
*********************************************

Generate input file:  build/model/_tmp/model/2ae7f4ae46d9383742a58f200f15a863/MI9_msmnile/arm64-v8a/model_input_data
Generate input file done.
* Run 'model' with round=1, restart_round=1, tuning=False, out_of_range_check=False, omp_num_threads=(-1,), cpu_affinity_policy=(1,), gpu_perf_hint=(3,), gpu_priority_hint=(3,)
Push build/model/_tmp/model/2ae7f4ae46d9383742a58f200f15a863/MI9_msmnile/arm64-v8a/model_input_data to /data/local/tmp/mace_run
Push build/model/model/model.data to /data/local/tmp/mace_run
Push build/model/model/model.pb to /data/local/tmp/mace_run/model.pb
Push build/model/_tmp/arm64-v8a/mace_run_static to /data/local/tmp/mace_run
Push /tmp/cmd_file-model-1583131872.8136957 to /data/local/tmp/mace_run/cmd_file-model-1583131872.8136957
I mace/tools/mace_run.cc:527] model name: model
I mace/tools/mace_run.cc:528] mace version: v0.12.0-0-ga610d50
I mace/tools/mace_run.cc:529] input node: data
I mace/tools/mace_run.cc:530] input shape: 1,3,112,112
I mace/tools/mace_run.cc:531] output node: fc1
I mace/tools/mace_run.cc:532] output shape: 1,1,1,512
I mace/tools/mace_run.cc:533] input_file: /data/local/tmp/mace_run/model_input
I mace/tools/mace_run.cc:534] output_file: /data/local/tmp/mace_run/model_out
I mace/tools/mace_run.cc:535] input dir:
I mace/tools/mace_run.cc:536] output dir:
I mace/tools/mace_run.cc:537] model_data_file: /data/local/tmp/mace_run/model.data
I mace/tools/mace_run.cc:538] model_file: /data/local/tmp/mace_run/model.pb
I mace/tools/mace_run.cc:539] device: CPU
I mace/tools/mace_run.cc:540] round: 1
I mace/tools/mace_run.cc:541] restart_round: 1
I mace/tools/mace_run.cc:542] gpu_perf_hint: 3
I mace/tools/mace_run.cc:543] gpu_priority_hint: 3
I mace/tools/mace_run.cc:544] omp_num_threads: -1
I mace/tools/mace_run.cc:545] cpu_affinity_policy: 1
I mace/tools/mace_run.cc:548] limit_opencl_kernel_time: 0
I mace/tools/mace_run.cc:553] opencl_queue_window_size: 0
I mace/libmace/mace.cc:464] Creating MaceEngine, MACE version: v0.12.0-0-ga610d50
I mace/libmace/mace.cc:503] Initializing MaceEngine
I mace/libmace/mace.cc:636] Destroying MaceEngine
I mace/tools/mace_run.cc:596] restart round 0
I mace/libmace/mace.cc:911] Create MaceEngine from model graph proto and weights data
I mace/libmace/mace.cc:464] Creating MaceEngine, MACE version: v0.12.0-0-ga610d50
I mace/libmace/mace.cc:503] Initializing MaceEngine
I mace/tools/mace_run.cc:269] Create Mace Engine latency: 115.374 ms
I mace/tools/mace_run.cc:276] Total init latency: 115.49 ms
I mace/tools/mace_run.cc:370] Warm up run
I mace/tools/mace_run.cc:406] 1st warm up run latency: 313.807 ms
I mace/tools/mace_run.cc:414] Run model
I mace/tools/mace_run.cc:476] Average latency: 119.442 ms
I mace/tools/mace_run.cc:491] Write output file /data/local/tmp/mace_run/model_out_fc1 with size 2048 done.
========================================================
     capability(CPU)        init      warmup     run_avg
========================================================
time          20.525     115.490     313.807     119.442
I mace/libmace/mace.cc:636] Destroying MaceEngine
Running finished!

* Validate with caffe
Pull /data/local/tmp/mace_run/model_out_fc1 to build/model/_tmp/model/2ae7f4ae46d9383742a58f200f15a863/MI9_msmnile/arm64-v8a
fc1 MACE VS CAFFE similarity: 0.9999942339509533 , sqnr: 82863.20821899887 , pixel_accuracy: 1.0
******************************************
          Similarity Test Passed          
******************************************

Validation done!

Dana service is not available.
* Run 'model' with round=1, restart_round=1, tuning=False, out_of_range_check=False, omp_num_threads=(-1,), cpu_affinity_policy=(1,), gpu_perf_hint=(3,), gpu_priority_hint=(3,)
Push build/model/_tmp/model/2ae7f4ae46d9383742a58f200f15a863/MI9_msmnile/arm64-v8a/model_input_data to /data/local/tmp/mace_run
Push build/model/model/model.data to /data/local/tmp/mace_run
Push build/model/model/model.pb to /data/local/tmp/mace_run/model.pb
Push build/model/_tmp/arm64-v8a/mace_run_static to /data/local/tmp/mace_run
Push /tmp/cmd_file-model-1583131917.8780398 to /data/local/tmp/mace_run/cmd_file-model-1583131917.8780398
I mace/tools/mace_run.cc:527] model name: model
I mace/tools/mace_run.cc:528] mace version: v0.12.0-0-ga610d50
I mace/tools/mace_run.cc:529] input node: data
I mace/tools/mace_run.cc:530] input shape: 1,3,112,112
I mace/tools/mace_run.cc:531] output node: fc1
I mace/tools/mace_run.cc:532] output shape: 1,1,1,512
I mace/tools/mace_run.cc:533] input_file: /data/local/tmp/mace_run/model_input
I mace/tools/mace_run.cc:534] output_file: /data/local/tmp/mace_run/model_out
I mace/tools/mace_run.cc:535] input dir:
I mace/tools/mace_run.cc:536] output dir:
I mace/tools/mace_run.cc:537] model_data_file: /data/local/tmp/mace_run/model.data
I mace/tools/mace_run.cc:538] model_file: /data/local/tmp/mace_run/model.pb
I mace/tools/mace_run.cc:539] device: GPU
I mace/tools/mace_run.cc:540] round: 1
I mace/tools/mace_run.cc:541] restart_round: 1
I mace/tools/mace_run.cc:542] gpu_perf_hint: 3
I mace/tools/mace_run.cc:543] gpu_priority_hint: 3
I mace/tools/mace_run.cc:544] omp_num_threads: -1
I mace/tools/mace_run.cc:545] cpu_affinity_policy: 1
I mace/tools/mace_run.cc:548] limit_opencl_kernel_time: 0
I mace/tools/mace_run.cc:553] opencl_queue_window_size: 0
I mace/libmace/mace.cc:464] Creating MaceEngine, MACE version: v0.12.0-0-ga610d50
I mace/libmace/mace.cc:503] Initializing MaceEngine
I mace/libmace/mace.cc:636] Destroying MaceEngine
I mace/tools/mace_run.cc:596] restart round 0
W ./mace/utils/tuner.h:201] Failed to read tuned param file: /data/local/tmp/mace_run/model_tuned_opencl_parameter.MI9.msmnile.bin
I mace/libmace/mace.cc:911] Create MaceEngine from model graph proto and weights data
I mace/libmace/mace.cc:464] Creating MaceEngine, MACE version: v0.12.0-0-ga610d50
W mace/core/kv_storage.cc:109] Failed to read kv store file: /data/local/tmp/mace_run/interior//mace_cl_compiled_program.bin
W mace/core/runtime/opencl/opencl_runtime.cc:382] Load OpenCL cached compiled kernel file failed. Please make sure the storage directory exist and you have Write&Read permission
I mace/libmace/mace.cc:503] Initializing MaceEngine
I mace/tools/mace_run.cc:269] Create Mace Engine latency: 889.857 ms
I mace/tools/mace_run.cc:276] Total init latency: 890.038 ms
I mace/tools/mace_run.cc:370] Warm up run
I mace/tools/mace_run.cc:406] 1st warm up run latency: 1129.57 ms
I mace/tools/mace_run.cc:414] Run model
I mace/tools/mace_run.cc:476] Average latency: 38.121 ms
I mace/tools/mace_run.cc:491] Write output file /data/local/tmp/mace_run/model_out_fc1 with size 2048 done.
========================================================
     capability(CPU)        init      warmup     run_avg
========================================================
time          19.911     890.038    1129.568      38.121
I mace/libmace/mace.cc:636] Destroying MaceEngine
Running finished!

* Validate with caffe
Pull /data/local/tmp/mace_run/model_out_fc1 to build/model/_tmp/model/2ae7f4ae46d9383742a58f200f15a863/MI9_msmnile/arm64-v8a
fc1 MACE VS CAFFE similarity: 0.9991734955929524 , sqnr: 599.7702326224907 , pixel_accuracy: 1.0
******************************************
          Similarity Test Passed          
******************************************

Validation done!

Elapse time: 1.547288 minutes.
* Package libs for model
Start packaging 'model' libs into build/model/libmace_model.tar.gz
build/model/model/
build/model/model/model.pb
build/model/model/cpu/
build/model/model/gpu/
build/model/model/model.data
Packaging Done!

------------------------------------------------------------
                          Library                           
------------------------------------------------------------
|          key           |              value              |
============================================================
| MACE Model package Path| build/model/libmace_model.tar.gz|
------------------------------------------------------------

MACE v0.12.0 w/ WINOGRAD 4:

root@ds017:/mace# python tools/converter.py run --validate --disable_tuning --config /models/model/model_r34.yml 
CMD> bazel build //mace/proto:mace_py
WARNING: --batch mode is deprecated. Please instead explicitly shut down your Bazel server using the command "bazel shutdown".
Loading:
Loading: 0 packages loaded
Analyzing: target //mace/proto:mace_py (6 packages loaded)
INFO: Analysed target //mace/proto:mace_py (17 packages loaded).
INFO: Found 1 target...
[1 / 5] [-----] BazelWorkspaceStatusAction stable-status.txt
Target //mace/proto:mace_py up-to-date:
bazel-genfiles/mace/proto/mace_pb2.py
INFO: Elapsed time: 2.319s, Critical Path: 0.04s
INFO: 0 processes.
INFO: Build completed successfully, 1 total action
INFO: Build completed successfully, 1 total action

CMD> cp -f bazel-genfiles/mace/proto/mace_pb2.py tools/python/py_proto

CMD> bazel build //third_party/caffe:caffe_py
WARNING: --batch mode is deprecated. Please instead explicitly shut down your Bazel server using the command "bazel shutdown".
Loading:
Loading: 0 packages loaded
Analyzing: target //third_party/caffe:caffe_py (5 packages loaded)
INFO: Analysed target //third_party/caffe:caffe_py (17 packages loaded).
INFO: Found 1 target...
[3 / 6] [-----] BazelWorkspaceStatusAction stable-status.txt
Target //third_party/caffe:caffe_py up-to-date:
bazel-genfiles/third_party/caffe/caffe_pb2.py
INFO: Elapsed time: 2.392s, Critical Path: 0.04s
INFO: 0 processes.
INFO: Build completed successfully, 1 total action
INFO: Build completed successfully, 1 total action

CMD> cp -f bazel-genfiles/third_party/caffe/caffe_pb2.py tools/python/py_proto

* Build //mace/tools:mace_run_static with ABI arm64-v8a
WARNING: --batch mode is deprecated. Please instead explicitly shut down your Bazel server using the command "bazel shutdown".
WARNING: The major revision of the Android NDK referenced by android_ndk_repository rule 'androidndk' is 19. The major revisions supported by Bazel are [10, 11, 12, 13, 14, 15, 16]. Bazel will attempt to treat the NDK as if it was r16. This may cause compilation and linkage problems. Please download a supported NDK version.
INFO: Analysed target //mace/tools:mace_run_static (32 packages loaded).
INFO: Found 1 target...
Target //mace/tools:mace_run_static up-to-date:
  bazel-bin/mace/tools/mace_run_static
INFO: Elapsed time: 12.108s, Critical Path: 0.39s
INFO: 0 processes.
INFO: Build completed successfully, 1 total action
('build', '//mace/tools:mace_run_static', '--config', 'android', '--cpu=arm64-v8a', '--define', 'neon=true', '--define', 'openmp=false', '--define', 'opencl=true', '--define', 'quantize=false', '--define', 'hexagon=false', '--define', 'hta=false', '--define', 'apu=false', '--config', 'optimization', '--config', 'symbol_hidden')
Build done!

*********************************************
          Run model model on MI9          
*********************************************

Generate input file:  build/model/_tmp/model/2ae7f4ae46d9383742a58f200f15a863/MI9_msmnile/arm64-v8a/model_input_data
Generate input file done.
* Run 'model' with round=1, restart_round=1, tuning=False, out_of_range_check=False, omp_num_threads=(-1,), cpu_affinity_policy=(1,), gpu_perf_hint=(3,), gpu_priority_hint=(3,)
Push build/model/_tmp/model/2ae7f4ae46d9383742a58f200f15a863/MI9_msmnile/arm64-v8a/model_input_data to /data/local/tmp/mace_run
Push build/model/model/model.data to /data/local/tmp/mace_run
Push build/model/model/model.pb to /data/local/tmp/mace_run/model.pb
Push build/model/_tmp/arm64-v8a/mace_run_static to /data/local/tmp/mace_run
Push /tmp/cmd_file-model-1583130426.6526062 to /data/local/tmp/mace_run/cmd_file-model-1583130426.6526062
I mace/tools/mace_run.cc:527] model name: model
I mace/tools/mace_run.cc:528] mace version: v0.12.0-0-ga610d50
I mace/tools/mace_run.cc:529] input node: data
I mace/tools/mace_run.cc:530] input shape: 1,3,112,112
I mace/tools/mace_run.cc:531] output node: fc1
I mace/tools/mace_run.cc:532] output shape: 1,1,1,512
I mace/tools/mace_run.cc:533] input_file: /data/local/tmp/mace_run/model_input
I mace/tools/mace_run.cc:534] output_file: /data/local/tmp/mace_run/model_out
I mace/tools/mace_run.cc:535] input dir:
I mace/tools/mace_run.cc:536] output dir:
I mace/tools/mace_run.cc:537] model_data_file: /data/local/tmp/mace_run/model.data
I mace/tools/mace_run.cc:538] model_file: /data/local/tmp/mace_run/model.pb
I mace/tools/mace_run.cc:539] device: CPU
I mace/tools/mace_run.cc:540] round: 1
I mace/tools/mace_run.cc:541] restart_round: 1
I mace/tools/mace_run.cc:542] gpu_perf_hint: 3
I mace/tools/mace_run.cc:543] gpu_priority_hint: 3
I mace/tools/mace_run.cc:544] omp_num_threads: -1
I mace/tools/mace_run.cc:545] cpu_affinity_policy: 1
I mace/tools/mace_run.cc:548] limit_opencl_kernel_time: 0
I mace/tools/mace_run.cc:553] opencl_queue_window_size: 0
I mace/libmace/mace.cc:464] Creating MaceEngine, MACE version: v0.12.0-0-ga610d50
I mace/libmace/mace.cc:503] Initializing MaceEngine
I mace/libmace/mace.cc:636] Destroying MaceEngine
I mace/tools/mace_run.cc:596] restart round 0
I mace/libmace/mace.cc:911] Create MaceEngine from model graph proto and weights data
I mace/libmace/mace.cc:464] Creating MaceEngine, MACE version: v0.12.0-0-ga610d50
I mace/libmace/mace.cc:503] Initializing MaceEngine
I mace/tools/mace_run.cc:269] Create Mace Engine latency: 115.89 ms
I mace/tools/mace_run.cc:276] Total init latency: 116.025 ms
I mace/tools/mace_run.cc:370] Warm up run
I mace/tools/mace_run.cc:406] 1st warm up run latency: 346.642 ms
I mace/tools/mace_run.cc:414] Run model
I mace/tools/mace_run.cc:476] Average latency: 119.829 ms
I mace/tools/mace_run.cc:491] Write output file /data/local/tmp/mace_run/model_out_fc1 with size 2048 done.
========================================================
     capability(CPU)        init      warmup     run_avg
========================================================
time          19.646     116.025     346.642     119.829
I mace/libmace/mace.cc:636] Destroying MaceEngine
Running finished!

* Validate with caffe
Pull /data/local/tmp/mace_run/model_out_fc1 to build/model/_tmp/model/2ae7f4ae46d9383742a58f200f15a863/MI9_msmnile/arm64-v8a
fc1 MACE VS CAFFE similarity: 0.9999931550298183 , sqnr: 69466.28993624987 , pixel_accuracy: 1.0
******************************************
          Similarity Test Passed          
******************************************

Validation done!

Dana service is not available.
* Run 'model' with round=1, restart_round=1, tuning=False, out_of_range_check=False, omp_num_threads=(-1,), cpu_affinity_policy=(1,), gpu_perf_hint=(3,), gpu_priority_hint=(3,)
Push build/model/_tmp/model/2ae7f4ae46d9383742a58f200f15a863/MI9_msmnile/arm64-v8a/model_input_data to /data/local/tmp/mace_run
Push build/model/model/model.data to /data/local/tmp/mace_run
Push build/model/model/model.pb to /data/local/tmp/mace_run/model.pb
Push build/model/_tmp/arm64-v8a/mace_run_static to /data/local/tmp/mace_run
Push /tmp/cmd_file-model-1583130473.4097192 to /data/local/tmp/mace_run/cmd_file-model-1583130473.4097192
I mace/tools/mace_run.cc:527] model name: model
I mace/tools/mace_run.cc:528] mace version: v0.12.0-0-ga610d50
I mace/tools/mace_run.cc:529] input node: data
I mace/tools/mace_run.cc:530] input shape: 1,3,112,112
I mace/tools/mace_run.cc:531] output node: fc1
I mace/tools/mace_run.cc:532] output shape: 1,1,1,512
I mace/tools/mace_run.cc:533] input_file: /data/local/tmp/mace_run/model_input
I mace/tools/mace_run.cc:534] output_file: /data/local/tmp/mace_run/model_out
I mace/tools/mace_run.cc:535] input dir:
I mace/tools/mace_run.cc:536] output dir:
I mace/tools/mace_run.cc:537] model_data_file: /data/local/tmp/mace_run/model.data
I mace/tools/mace_run.cc:538] model_file: /data/local/tmp/mace_run/model.pb
I mace/tools/mace_run.cc:539] device: GPU
I mace/tools/mace_run.cc:540] round: 1
I mace/tools/mace_run.cc:541] restart_round: 1
I mace/tools/mace_run.cc:542] gpu_perf_hint: 3
I mace/tools/mace_run.cc:543] gpu_priority_hint: 3
I mace/tools/mace_run.cc:544] omp_num_threads: -1
I mace/tools/mace_run.cc:545] cpu_affinity_policy: 1
I mace/tools/mace_run.cc:548] limit_opencl_kernel_time: 0
I mace/tools/mace_run.cc:553] opencl_queue_window_size: 0
I mace/libmace/mace.cc:464] Creating MaceEngine, MACE version: v0.12.0-0-ga610d50
I mace/libmace/mace.cc:503] Initializing MaceEngine
I mace/libmace/mace.cc:636] Destroying MaceEngine
I mace/tools/mace_run.cc:596] restart round 0
W ./mace/utils/tuner.h:201] Failed to read tuned param file: /data/local/tmp/mace_run/model_tuned_opencl_parameter.MI9.msmnile.bin
I mace/libmace/mace.cc:911] Create MaceEngine from model graph proto and weights data
I mace/libmace/mace.cc:464] Creating MaceEngine, MACE version: v0.12.0-0-ga610d50
W mace/core/kv_storage.cc:109] Failed to read kv store file: /data/local/tmp/mace_run/interior//mace_cl_compiled_program.bin
W mace/core/runtime/opencl/opencl_runtime.cc:382] Load OpenCL cached compiled kernel file failed. Please make sure the storage directory exist and you have Write&Read permission
I mace/libmace/mace.cc:503] Initializing MaceEngine
I mace/tools/mace_run.cc:269] Create Mace Engine latency: 1084.34 ms
I mace/tools/mace_run.cc:276] Total init latency: 1084.52 ms
I mace/tools/mace_run.cc:370] Warm up run
I mace/tools/mace_run.cc:406] 1st warm up run latency: 1395.71 ms
I mace/tools/mace_run.cc:414] Run model
I mace/tools/mace_run.cc:476] Average latency: 35.978 ms
I mace/tools/mace_run.cc:491] Write output file /data/local/tmp/mace_run/model_out_fc1 with size 2048 done.
========================================================
     capability(CPU)        init      warmup     run_avg
========================================================
time          19.563    1084.516    1395.707      35.978
I mace/libmace/mace.cc:636] Destroying MaceEngine
Running finished!

* Validate with caffe
Pull /data/local/tmp/mace_run/model_out_fc1 to build/model/_tmp/model/2ae7f4ae46d9383742a58f200f15a863/MI9_msmnile/arm64-v8a
fc1 MACE VS CAFFE similarity: 0.9832035111649001 , sqnr: 29.693451591115913 , pixel_accuracy: 1.0
ERROR: [] /mace/validate.py:117: ******************************************
          Similarity Test Failed          
******************************************

Traceback (most recent call last):
  File "tools/converter.py", line 1151, in <module>
    flags.func(flags)
  File "tools/converter.py", line 938, in run_mace
    device.run_specify_abi(flags, configs, target_abi)
  File "/mace/tools/device.py", line 782, in run_specify_abi
    log_file=log_file,
  File "/mace/tools/sh_commands.py", line 756, in validate_model
    _fg=True)
  File "/root/.pyenv/versions/3.6.3/lib/python3.6/site-packages/sh.py", line 1413, in __call__
    raise exc
sh.ErrorReturnCode_1: 

  RAN: /usr/bin/docker exec mace_caffe_lastest_validator python -u /mace/validate.py --platform=caffe --model_file=/mace/arc-34.prototxt --weight_file=/mace/arc-34.caffemodel --input_file=/mace/model_input --mace_out_file=/mace/model_out --device_type=GPU --input_node=data --output_node=fc1 --input_shape=1,3,112,112 --output_shape=1,1,1,512 --input_data_format=NCHW --output_data_format=NHWC --validation_threshold=0.995000 --input_data_type=float32 --backend=tensorflow --validation_outputs_data= --log_file=

  STDOUT:


  STDERR:

Additional context

Model to reproduce the issue can be found here
@lu229
Copy link
Collaborator

lu229 commented Mar 3, 2020

@gasgallo When the winograd block size is large, the accuracy of calculation will decrease, This is a normal phenomenon.

@gasgallo
Copy link
Contributor Author

gasgallo commented Mar 4, 2020

@lu229 that's correct.

I was just surprised that it happens with that specific model only, while with other models (e.g this model) that has the same convolution filter sizes, but 50 layers instead of 34, validation is successful.

@lu229
Copy link
Collaborator

lu229 commented Mar 9, 2020

@gasgallo Generally it can be passed, which is related to the weight value of the model....

@gasgallo gasgallo closed this as completed Mar 9, 2020
@gasgallo
Copy link
Contributor Author

@lu229 I've just noticed that the same model passes validation if run on a difference device. Any idea why this happens?

@lu229
Copy link
Collaborator

lu229 commented May 11, 2020

@gasgallo There are many possible reasons for this, such as:

  1. The input data is different between the devices, you can configure the same input data in YML and try it.
  2. The calculation accuracy of each equipment is different.
  3. Sometimes there is a bug in the OpenCL driver, which is rare

@gasgallo
Copy link
Contributor Author

We can continue at #638

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@lu229 @gasgallo