-
Notifications
You must be signed in to change notification settings - Fork 154
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Crashes in the middle of the optimization process (KeyError: 'throughput') #90
Comments
Hi @shonigs , I tried with the model in notebook tutorials and no issues appeared. I am not sure if the issue is related to your ONNX model. Could you please share the model you used ? Thanks. |
Hi, I am getting the same error: KeyError: 'throughput'. The complete error log is:
I am executing the optimization with The above error message is the contents of log.txt (see final part of the execution command above). Please find my ONNX model here: https://get.hidrive.com/2qErePEy (Link valid until August 10, 2022) |
Hi @kbraun-axio , |
Hi @leqiao-1, |
Hi @leqiao-1, Today, I tried to run the optimization again. This time, I increased the max_latency_ms to 10,000. However, I got the same error. Do you think max_latency_ms of 10,000 is still not enough? Inference with the onnx runtime and the same onnx model that I am trying to optimize takes about 7.5 seconds. |
Hi @kbraun-axio |
Okay, thank you. The machine on which we want to run the inference is a 6 core AMD CPU with 8 GB RAM from 2012. It runs in a manufacturing / shop floor environment. They do not have the newest hardware. But maybe it would be better to use a more powerful machine, like an Nvidia Jetson device, which supports CUDA. Besides that, I realized the optimization takes lots of RAM. Watching the processes with |
Hi @kbraun-axio, If so, it's maybe because when checking model input info with ort infenrece sessions, OLive will try to create session with cuda. I think it's a bug in OLive, and we will fix it. As a workaround, you can uninstall onnxruntime gpu package and install the cpu version. If not, please let me know your onnxruntime package version with |
Hi @leqiao-1, But the other issue, the KeyError: 'throughput', persists even with the cpu package and even if we set the max_latency_ms to higher values. Maybe it fails because the system is too old, it is from 2012. |
Hi @kbraun-axio |
I have same issue with log:
Running by: opt_config = OptimizationConfig(
model_path = "./model.onnx",
sample_input_data_path="./input.npz",
result_path = "olive_opt_latency_result",
throughput_tuning_enabled=True,
openmp_enabled=False,
max_latency_percentile = 0.95,
max_latency_ms = 1000000,
threads_num = 1,
min_duration_sec=10000,
providers_list = ["cpu", "dnnl"],
inter_thread_num_list = [1],
intra_thread_num_list=[1],
execution_mode_list = ["sequential"],
ort_opt_level_list=['all'],
concurrency_num=4,
warmup_num = 20,
test_num = 200
)
result = optimize(opt_config) Model is huge and inference is over 15secs but what do I wrong ? What means
|
Hi @PasaOpasen, Q: |
@leqiao-1 Thank u for fast response! Can u please try to do anything with this model https://github.com/PasaOpasen/_olive_craft ? I tried several configurations but nothing changed. Its inference is about 15sec with 2 cores and the optimization works too long with huge Also the optimization uses 6-8 cores with |
If you have any further concerns or questions, please reopen this issue. |
Hi,
The program crashes while optimizing -
Steps to reproduce
installation
Use
This runs for sometime, then crashes
I am not sure about the exact issue but could this be maybe be in a try-except so the whole process doesn't fail?
P.S. is there any details about the environment that I should add?
The text was updated successfully, but these errors were encountered: