Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature] add onnxruntime test tool #498

Merged
merged 10 commits into from
Apr 29, 2021
Merged

Conversation

grimoire
Copy link
Member

@grimoire grimoire commented Apr 20, 2021

Hi
This PR adds an ONNXRuntime test tool to align results between PyTorch and ONNXRuntime.
Pytorch2onnx is also updated to support rescale output shape (a large output shape might cause OOM).

TODO

  • Align performance of segmentation model, update doc.
Model Config Dataset Metric PyTorch ONNXRuntime
FCN fcn_r50-d8_512x1024_40k_cityscapes.py cityscapes mIOU 72.2 72.2
PSPNet pspnet_r50-d8_769x769_40k_cityscapes.py cityscapes mIOU 78.2 78.1
deeplabv3 deeplabv3_r50-d8_769x769_40k_cityscapes.py cityscapes mIOU 78.5 78.3
deeplabv3+ deeplabv3plus_r50-d8_769x769_40k_cityscapes.py cityscapes mIOU 78.9 78.7

@codecov
Copy link

codecov bot commented Apr 20, 2021

Codecov Report

Merging #498 (61071d9) into master (83d312e) will increase coverage by 0.11%.
The diff coverage is n/a.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #498      +/-   ##
==========================================
+ Coverage   86.48%   86.60%   +0.11%     
==========================================
  Files          97       99       +2     
  Lines        4974     5160     +186     
  Branches      807      834      +27     
==========================================
+ Hits         4302     4469     +167     
- Misses        519      533      +14     
- Partials      153      158       +5     
Flag Coverage Δ
unittests 86.60% <ø> (+0.11%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
mmseg/datasets/pipelines/transforms.py 97.12% <0.00%> (-0.87%) ⬇️
mmseg/models/necks/__init__.py 100.00% <0.00%> (ø)
mmseg/models/backbones/__init__.py 100.00% <0.00%> (ø)
mmseg/models/backbones/vit.py 87.97% <0.00%> (ø)
mmseg/models/necks/multilevel_neck.py 100.00% <0.00%> (ø)
mmseg/models/builder.py 91.30% <0.00%> (+9.82%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 83d312e...61071d9. Read the comment docs.

tools/ort_test.py Outdated Show resolved Hide resolved
@grimoire grimoire changed the title [WIP] add onnxruntime test tool [Feature] add onnxruntime test tool Apr 25, 2021
@xvjiarui
Copy link
Collaborator

LGTM. Ping @RunningLeon if there is any comments.

@grimoire grimoire changed the title [Feature] add onnxruntime test tool [WIP] add onnxruntime test tool Apr 26, 2021
@grimoire grimoire changed the title [WIP] add onnxruntime test tool [Feature] add onnxruntime test tool Apr 27, 2021
docs/useful_tools.md Outdated Show resolved Hide resolved
Copy link
Collaborator

@RunningLeon RunningLeon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@xvjiarui xvjiarui merged commit bdf5adf into open-mmlab:master Apr 29, 2021
bowenroom pushed a commit to bowenroom/mmsegmentation that referenced this pull request Feb 25, 2022
* add onnxruntime test tool, update pytorch2onnx to support slice export

* onnx convert with custom output shape, update test code

* update pytorch2onnx, add rescale_shape support, add document

* update doc for lint error fixing

* remove cpu flag in ort_test.py

* change class name, fix cuda error

* remote comment

* fix bug of torch2onnx

* mIOU to mIoU
aravind-h-v pushed a commit to aravind-h-v/mmsegmentation that referenced this pull request Mar 27, 2023
Fix `disable_attention_slicing` in pipelines.
wjkim81 pushed a commit to wjkim81/mmsegmentation that referenced this pull request Dec 3, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants