Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[multimodal]Upgrade object detection backend to mmdet 3.0 #3188

Merged
merged 39 commits into from
Jun 1, 2023
Merged

[multimodal]Upgrade object detection backend to mmdet 3.0 #3188

merged 39 commits into from
Jun 1, 2023

Conversation

FANGAreNotGnu
Copy link
Contributor

@FANGAreNotGnu FANGAreNotGnu commented May 2, 2023 •
edited

Change MMDetection backend to 3.0.

Changes:

  • Internal data structures
  • Pipeline logics
  • Yolox Configs
  • Mosaic/mixup tranform logics
  • Multigpu support logics
  • Basically everything

Benefits:

  • Able to use latest work like DINO, RTMDet, Detic, etc.
  • Have better data structure now (DataContainer in 2.0 is not as good), and also aligns better with open vocabulary detection's needs.
  • Will be stable and do not need big refactors for a long period.

Future works:

  • Benchmarking
  • Update Presets
  • Update examples and tutorials
  • Refine tests

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@review-notebook-app
Copy link

Check out this pull request on  ReviewNB

See visual diffs & provide feedback on Jupyter Notebooks.

Powered by ReviewNB

@tonyhoo
Copy link
Collaborator

tonyhoo commented May 5, 2023

Thanks for updating the version. Can you provide some benchmark results and attach them in the description? Thanks

@FANGAreNotGnu FANGAreNotGnu added the model list checked You have updated the model list after modifying multimodal unit tests/docs label May 10, 2023
@github-actions
Copy link

Job PR-3188-7cd1dc2 is done.
Docs are uploaded to http://autogluon-staging.s3-website-us-west-2.amazonaws.com/PR-3188/7cd1dc2/index.html

@FANGAreNotGnu FANGAreNotGnu changed the title [WIP][multimodal]Upgrade object detection backend to mmdet 3.0 [multimodal]Upgrade object detection backend to mmdet 3.0 May 11, 2023
@FANGAreNotGnu
Copy link
Contributor Author

Thanks for updating the version. Can you provide some benchmark results and attach them in the description? Thanks

Just completed the basic changes. I'll run a small benchmark after working on a PR in autogluon-bench. And will do a complete benchmarking after upgrading presets to more recent models (RTMDet/DINO).

@github-actions
Copy link

Job PR-3188-50bedc8 is done.
Docs are uploaded to http://autogluon-staging.s3-website-us-west-2.amazonaws.com/PR-3188/50bedc8/index.html

@github-actions
Copy link

Job PR-3188-70feb9d is done.
Docs are uploaded to http://autogluon-staging.s3-website-us-west-2.amazonaws.com/PR-3188/70feb9d/index.html

@github-actions
Copy link

Job PR-3188-78157d1 is done.
Docs are uploaded to http://autogluon-staging.s3-website-us-west-2.amazonaws.com/PR-3188/78157d1/index.html

@github-actions
Copy link

Job PR-3188-0635abd is done.
Docs are uploaded to http://autogluon-staging.s3-website-us-west-2.amazonaws.com/PR-3188/0635abd/index.html

zhiqiangdon
zhiqiangdon reviewed May 25, 2023
View reviewed changes
CI/docker/full_install_image.sh Show resolved Hide resolved
...utorials/multimodal/object_detection/finetune/detection_high_performance_finetune_coco.ipynb Show resolved Hide resolved
multimodal/setup.py Outdated Show resolved Hide resolved
multimodal/src/autogluon/multimodal/configs/pretrain/detection/yolox/yolox_l_8xb8-300e_coco.py Show resolved Hide resolved
multimodal/src/autogluon/multimodal/data/process_mmlab/process_mmdet.py Outdated Show resolved Hide resolved
multimodal/src/autogluon/multimodal/models/mmdet_image.py
return rets
elif mode == "predict":
# for detailed data structure, see https://github.com/open-mmlab/mmdetection/blob/main/mmdet/structures/det_data_sample.py
return [{BBOX: ret.pred_instances, LABEL: ret.gt_instances} for ret in rets]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why does predict mode require groundtruth labels?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

model(mode="predict") is used for both predicting and evaluation. It's easier to include the ground truth here than merging ground truth with result in lit module / predictor. Here gt_instances will be None if there is no label available.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The evaluation is done after prediction. Why do we need labels in the prediction?

multimodal/src/autogluon/multimodal/predictor.py Show resolved Hide resolved
multimodal/src/autogluon/multimodal/utils/environment.py Outdated Show resolved Hide resolved
multimodal/src/autogluon/multimodal/configs/model/fusion_mlp_image_text_tabular.yaml Show resolved Hide resolved
@github-actions
Copy link

Job PR-3188-0c266ee is done.
Docs are uploaded to http://autogluon-staging.s3-website-us-west-2.amazonaws.com/PR-3188/0c266ee/index.html

@github-actions
Copy link

Job PR-3188-c00a1e1 is done.
Docs are uploaded to http://autogluon-staging.s3-website-us-west-2.amazonaws.com/PR-3188/c00a1e1/index.html

@github-actions
Copy link

Job PR-3188-f54327f is done.
Docs are uploaded to http://autogluon-staging.s3-website-us-west-2.amazonaws.com/PR-3188/f54327f/index.html

@github-actions
Copy link

github-actions bot commented Jun 1, 2023

Job PR-3188-5990923 is done.
Docs are uploaded to http://autogluon-staging.s3-website-us-west-2.amazonaws.com/PR-3188/5990923/index.html

@github-actions
Copy link

github-actions bot commented Jun 1, 2023

Job PR-3188-e2ee022 is done.
Docs are uploaded to http://autogluon-staging.s3-website-us-west-2.amazonaws.com/PR-3188/e2ee022/index.html

zhiqiangdon
zhiqiangdon approved these changes Jun 1, 2023
View reviewed changes
Copy link
Contributor

@zhiqiangdon zhiqiangdon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks for the upgrade.

@FANGAreNotGnu FANGAreNotGnu merged commit cccad2f into autogluon:master Jun 1, 2023
29 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@zhiqiangdon zhiqiangdon zhiqiangdon approved these changes

@yongxinw yongxinw Awaiting requested review from yongxinw

@tonyhoo tonyhoo Awaiting requested review from tonyhoo

Assignees
No one assigned
Labels
model list checked You have updated the model list after modifying multimodal unit tests/docs
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@FANGAreNotGnu @tonyhoo @zhiqiangdon