[Feature] Upload checkpoints and logs to ceph

[zhouzaida]

Thanks for your contribution and we appreciate it a lot. The following instructions would make your pull request more healthy and more easily get feedback. If you do not understand some items, don't worry, just make the pull request and seek help from maintainers.

Motivation

Note: The PR is based on #1330

The motivation of the PR is to upload checkpoints and logs to PetrelOSS and load checkpoints from PetrelOSS.

It includes four parts:

• Refactor save_checkpoint (mmcv/runner/checkpoint.py) and CheckpointHook (mmcv/runner/hooks/checkpoint.py). They support uploading checkpoints to PetrelOSS.
• Refactor load_checkpoint_ceph. It supports loading checkpoints from PetrelOSS.
• Refactor TextLoggerHook. It supports uploading logs to PetrelOSS.
• Refactor EvalHook. It supports uploading the best checkpoint to PetrelOSS.

Modification

• save_checkpoint (mmcv/runner/checkpoint.py)

  • Add an arugment file_client_args which is used to instantiate a FileClient

• CheckPointHook (mmcv/runner/hooks/checkpoint.py)

  • Add an arugment file_client_args which is used to instantiate a FileClient
  • Refactor the method _save_checkpoint which saves the new checkpoint and removes those deprecated checkpoints

• load_checkpoint_ceph
  Original load_checkpoint_ceph only supports loading checkpoints from Ceph. The prefix of it is s3:// and it is the same as Petrel. Therefore, we need to refactor the function and make it support Petrel too.

• TextLoggerHook

  • Add three arguments out_dir, keep_local and file_client_args
  • Add after_run which will upload the logs to Petrel after the end of the training

• EvalHook (mmcv/runner/hooks/evalhook.py)

  • Add two arguments out_dir and file_client_args
  • Update before_run which will infer the out_dir and file_client.

BC-breaking (Optional)

No

Use cases (Optional)

1. Upload checkpoints to PetrelOSS

   checkpoint_config = dict(interval=1, out_dir='s3://path/of/your/directory')  # defined in default_runtime.py
   evaluation = dict(interval=1, metric='accuracy', out_dir='s3://path/of/your/directory')

2. Save the best checkpoint to PetrelOSS

   evaluation = dict(interval=1, save_best='auto', out_dir='s3://bucket_name')

3. Resume the checkpoint from PetrelOSS

    resume_from = 's3://path/of/your/resumed_checkpoint.pth'

4. Upload logs to PetrelOss
   Only the log_config needs to be modified, which is defined in default_runtime.py

   log_config = dict(
       interval=50,
       hooks=[
           dict(type='TextLoggerHook', out_dir='s3://path/of/your/directory/to/save/log'),
       ])

Tested by downstream codebases

MMClassification

Training a model with resnet101_b16x8_cifar10

	Save to PetrelOSS	Save to Lustre
Training time (20 epochs, 1 GPU)	15min	14min

• [Save to PetrelOSS] config for uploading checkpoints, best checkpoint, and logs to PetrelOSS

  # checkpoint saving
  checkpoint_config = dict(interval=1, max_keep_ckpts=5, out_dir='s3://bucket_name')
  # yapf:disable
  log_config = dict(
      interval=100,
      hooks=[
          dict(type='TextLoggerHook', out_dir='s3://bucket_name'),
          # dict(type='TensorboardLoggerHook')
      ])
  # yapf:enable
  evaluation = dict(interval=1, save_best='auto', out_dir='s3://bucket_name')

  • CheckpointHook works as expected
    Include uploading checkpoints and deleting outdated checkpoints (max_keep_ckpts=5)
  • Resuming from checkpoints works as expected
  • TextLoggerHook works as expected
    Include uploading logs or deleting the local logs after uploading logs (keep_local=False)
  • EvaluationHook works as expected
    Include evaluating the validation dataset, saving and updating the best checkpoint (save_best='auto')

• [Save to Lustre] config for saving checkpoints, best checkpoint, and logs to Lustre

  # checkpoint saving
  checkpoint_config = dict(interval=1, max_keep_ckpts=5)
  # yapf:disable
  log_config = dict(
      interval=100,
      hooks=[
          dict(type='TextLoggerHook'),
          # dict(type='TensorboardLoggerHook')
      ])
  # yapf:enable
  evaluation = dict(interval=1, save_best='auto')

MMDetection

Training a model with retinanet_r50_fpn_1x_coco

	Save to PetrelOSS	Save to Lustre
Training time (8 epochs, 1 GPU)	2h20m

• [Save to PetrelOSS] config for uploading checkpoints, best checkpoint, and logs to PetrelOSS

  # checkpoint saving
  checkpoint_config = dict(interval=1, max_keep_ckpts=2, out_dir='s3://bucket_name')
  # yapf:disable
  log_config = dict(
      interval=50,
      hooks=[
          dict(type='TextLoggerHook', out_dir='s3://bucket_name'),
          # dict(type='TensorboardLoggerHook')
      ])
  # yapf:enable
  evaluation = dict(interval=1, save_best='bbox', out_dir='s3://bucket_name')
  data = dict(
      samples_per_gpu=2,
      workers_per_gpu=2,
      train=dict(
          type=dataset_type,
          ann_file=data_root + 'annotations/instances_val2017.json',  # reduce the training time
          img_prefix=data_root + 'train2017/',
          pipeline=train_pipeline),
      val=dict(
          type=dataset_type,
          ann_file=data_root + 'annotations/instances_val2017.json',
          img_prefix=data_root + 'val2017/',
          pipeline=test_pipeline),
      test=dict(
          type=dataset_type,
          ann_file=data_root + 'annotations/instances_val2017.json',
          img_prefix=data_root + 'val2017/',
          pipeline=test_pipeline))

• [Save to Lustre] config for saving checkpoints, best checkpoint, and logs to Lustre

  # checkpoint saving
  checkpoint_config = dict(interval=1, max_keep_ckpts=5)
  # yapf:disable
  log_config = dict(
      interval=100,
      hooks=[
          dict(type='TextLoggerHook'),
          # dict(type='TensorboardLoggerHook')
      ])
  # yapf:enable
  evaluation = dict(interval=1, save_best='bbox')
  data = dict(
      samples_per_gpu=2,
      workers_per_gpu=2,
      train=dict(
          type=dataset_type,
          ann_file=data_root + 'annotations/instances_val2017.json',  # reduce the training time
          img_prefix=data_root + 'train2017/',
          pipeline=train_pipeline),
      val=dict(
          type=dataset_type,
          ann_file=data_root + 'annotations/instances_val2017.json',
          img_prefix=data_root + 'val2017/',
          pipeline=test_pipeline),
      test=dict(
          type=dataset_type,
          ann_file=data_root + 'annotations/instances_val2017.json',
          img_prefix=data_root + 'val2017/',
          pipeline=test_pipeline))

Checklist

Before PR:

• I have read and followed the workflow indicated in the CONTRIBUTING.md to create this PR.
• Pre-commit or linting tools indicated in CONTRIBUTING.md are used to fix the potential lint issues.
• Bug fixes are covered by unit tests, the case that causes the bug should be added in the unit tests.
• New functionalities are covered by complete unit tests. If not, please add more unit test to ensure the correctness.
• The documentation has been modified accordingly, including docstring or example tutorials.

After PR:

• If the modification has potential influence on downstream or other related projects, this PR should be tested with some of those projects, like MMDet or MMCls.
• CLA has been signed and all committers have signed the CLA in this PR.

[codecov]

Codecov Report

Merging #1375 (81520af) into master (8cac7c2) will increase coverage by 0.14%.
The diff coverage is 77.22%.

❗ Current head 81520af differs from pull request most recent head 39a58a3. Consider uploading reports for the commit 39a58a3 to get more accurate results

@@            Coverage Diff             @@
##           master    #1375      +/-   ##
==========================================
+ Coverage   68.59%   68.74%   +0.14%     
==========================================
  Files         164      164              
  Lines       10891    11029     +138     
  Branches     1991     2018      +27     
==========================================
+ Hits         7471     7582     +111     
- Misses       3030     3051      +21     
- Partials      390      396       +6     

Flag	Coverage Δ
unittests	68.74% <77.22%> (+0.14%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
mmcv/runner/hooks/logger/text.py	71.07% <31.81%> (-8.73%)	⬇️
mmcv/runner/checkpoint.py	73.77% <41.17%> (-0.65%)	⬇️
mmcv/fileio/file_client.py	86.66% <85.29%> (-3.41%)	⬇️
mmcv/runner/hooks/checkpoint.py	84.12% <90.90%> (+24.86%)	⬆️
mmcv/fileio/handlers/base.py	53.33% <100.00%> (-25.24%)	⬇️
mmcv/fileio/handlers/pickle_handler.py	87.50% <100.00%> (-12.50%)	⬇️
mmcv/fileio/io.py	76.56% <100.00%> (+6.56%)	⬆️
mmcv/fileio/parse.py	100.00% <100.00%> (ø)
... and 3 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5b5b47f...39a58a3. Read the comment docs.

[ZwwWayne]

Can be merged after resolving the final comments.