Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Fix ] MMDetWandbHook does not support offline mode #8755

Merged
merged 1 commit into from
Sep 19, 2022
Merged

[Fix ] MMDetWandbHook does not support offline mode #8755

merged 1 commit into from
Sep 19, 2022

Conversation

i-aki-y
Copy link
Contributor

@i-aki-y i-aki-y commented Sep 8, 2022

Motivation

I got the following error when I used MMDetWandbHook in the offline mode:

wandb: Tracking run with wandb version 0.13.2
wandb: W&B syncing is set to `offline` in this directory.  
wandb: Run `wandb online` or set WANDB_MODE=online to enable cloud syncing.
Traceback (most recent call last):
  File "tools/train.py", line 242, in <module>
    main()
  File "tools/train.py", line 238, in main
    meta=meta)
  File "/content/mmdetection/mmdet/apis/train.py", line 244, in train_detector
    runner.run(data_loaders, cfg.workflow)
  File "/usr/local/lib/python3.7/dist-packages/mmcv/runner/epoch_based_runner.py", line 117, in run
    self.call_hook('before_run')
  File "/usr/local/lib/python3.7/dist-packages/mmcv/runner/base_runner.py", line 317, in call_hook
    getattr(hook, fn_name)(self)
  File "/usr/local/lib/python3.7/dist-packages/mmcv/runner/dist_utils.py", line 135, in wrapper
    return func(*args, **kwargs)
  File "/content/mmdetection/mmdet/core/hook/wandblogger_hook.py", line 203, in before_run
    self._log_data_table()
  File "/content/mmdetection/mmdet/core/hook/wandblogger_hook.py", line 569, in _log_data_table
    self.wandb.run.use_artifact(data_artifact)
  File "/usr/local/lib/python3.7/dist-packages/wandb/sdk/wandb_run.py", line 255, in wrapper
    return func(self, *args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/wandb/sdk/wandb_run.py", line 2491, in use_artifact
    raise TypeError("Cannot use artifact when in offline mode.")
TypeError: Cannot use artifact when in offline mode.
wandb: Waiting for W&B process to finish... (failed 1).

The log_config is the following:

log_config = dict(
    hooks=[
        dict(type='TextLoggerHook'),
        dict(type='MMDetWandbHook',
             init_kwargs={'project':'mmdet','mode':'offline'},
             interval=1,
             log_checkpoint=True,
             log_checkpoint_metadata=True,
             num_eval_images=1)
    ]
)

Modification

I think the use_artifact and wait methods in the following codes can be skipped when the wandb is offline:

def _log_data_table(self):
"""Log the W&B Tables for validation data as artifact and calls
`use_artifact` on it so that the evaluation table can use the reference
of already uploaded images.
This allows the data to be uploaded just once.
"""
data_artifact = self.wandb.Artifact('val', type='dataset')
data_artifact.add(self.data_table, 'val_data')
self.wandb.run.use_artifact(data_artifact)
data_artifact.wait()
self.data_table_ref = data_artifact.get('val_data')

BC-breaking (Optional)

When the wandb is in the online mode (it is the default mode), there is no change.

Use cases (Optional)

I use the offline mode when I debug and test the codes.

Checklist

  1. Pre-commit or other linting tools are used to fix the potential lint issues.
  2. The modification is covered by complete unit tests. If not, please add more unit test to ensure the correctness.
  3. If the modification has potential influence on downstream projects, this PR should be tested with downstream projects, like MMDet or MMCls.
  4. The documentation has been modified accordingly, like docstring or example tutorials.

About unit test:
To test the MMDetwandbHook, we need a full training setup (prepare dataset and install wandb and login), but I am not sure how to write this type of test efficiently.
I would appreciate any advice about this.

@i-aki-y i-aki-y changed the title [Fix ] WandbLogger does not support offline mode [Fix ] MMDetWandbHook does not support offline mode Sep 8, 2022
@RangiLyu RangiLyu changed the base branch from master to dev September 14, 2022 06:17
@RangiLyu RangiLyu added enhancement New feature or request v-2.x labels Sep 14, 2022
@ZwwWayne ZwwWayne added this to the 2.26.0 milestone Sep 14, 2022
@codecov
Copy link

codecov bot commented Sep 14, 2022

Codecov Report

Base: 64.08% // Head: 64.06% // Decreases project coverage by -0.02% ⚠️

Coverage data is based on head (63b9ae1) compared to base (f8bbba2).
Patch coverage: 0.00% of modified lines in pull request are covered.

Additional details and impacted files
@@            Coverage Diff             @@
##              dev    #8755      +/-   ##
==========================================
- Coverage   64.08%   64.06%   -0.03%     
==========================================
  Files         361      361              
  Lines       29525    29527       +2     
  Branches     5020     5021       +1     
==========================================
- Hits        18922    18916       -6     
- Misses       9589     9593       +4     
- Partials     1014     1018       +4     
Flag Coverage Δ
unittests 64.06% <0.00%> (-0.03%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
mmdet/core/hook/wandblogger_hook.py 14.91% <0.00%> (-0.13%) ⬇️
mmdet/core/post_processing/bbox_nms.py 74.62% <0.00%> (-4.48%) ⬇️
mmdet/datasets/pipelines/transforms.py 75.72% <0.00%> (-0.25%) ⬇️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

@ZwwWayne ZwwWayne merged commit 2825598 into open-mmlab:dev Sep 19, 2022
@ayulockin
Copy link
Contributor

Hey @i-aki-y, just curious to know if you are using W&B Tables when using MMDetWandbHook in offline mode?

@i-aki-y
Copy link
Contributor Author

i-aki-y commented Sep 23, 2022

@ayulockin Hi, Thank you for your review.

No, I do not use W&B Tables intentionally in offline mode.
I use offline mode when I debug and test my code.
So I want my code to work independently of WANDB_MODE, without any other code change as follows.

# for debug
WANDB_MODE=offline python tools/train.py ${CONFIG}

@ayulockin
Copy link
Contributor

Awesome, makes sense. Thanks for clarifying.

YangXiaoYang1997 added a commit to YangXiaoYang1997/mmdetection that referenced this pull request Dec 5, 2022
SakiRinn pushed a commit to SakiRinn/mmdetection-locount that referenced this pull request Mar 17, 2023
@jason102811
Copy link

Dear i-aki-y,
First of all, we want to express our gratitude for your significant PR in the MMDet . Your contribution is highly appreciated, and we are grateful for your efforts in helping improve this open-source project during your personal time. We believe that many developers will benefit from your PR.
As a valued contributor, we would also like to invite you to join our Special Interest Group (SIG) private channel, where you can share your experiences, ideas, and build connections with like-minded peers.
Join us :https://discord.gg/raweFPmdzG
To join the SIG channel, simply message moderator— OpenMMLab on Discord or briefly share your open-source contributions in the #introductions channel and we will assist you. We look forward to seeing you there!
Also,if you have WeChat,welcome to join our community on WeChat. You can add our assistant :openmmlabwx,or scan the following QR code to join us. Please add "mmsig + Github ID" as a remark when adding friends:)
Best regards! @i-aki-y

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request v-2.x
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants