Add SimCLR trainer #1252

isaaccorley · 2023-04-15T23:01:36Z

This PR adds a SimCLR trainer which uses implementations from the lightly package

Reboot of #1195

adamjstewart

Messaged on Slack, need to update to new style.

requirements/required.txt

setup.cfg

adamjstewart · 2023-04-25T17:38:12Z

Really sick of codecov/feedback#126

calebrob6 · 2023-04-26T06:00:29Z

I tried this trainer with multigpu and got ValueError: bad value(s) in fds_to_keep, however the MultiLabel trainer worked fine with the same DataModule.

adamjstewart · 2023-04-26T15:13:47Z

I tried this trainer with multigpu and got ValueError: bad value(s) in fds_to_keep, however the MultiLabel trainer worked fine with the same DataModule.

With or without gather_distributed?

calebrob6 · 2023-04-26T15:42:16Z

I don't touch gather_distributed as I would expect this to work by specifying multiple GPUs in the trainer.

E.g. this config works

module:
  _target_: torchgeo.trainers.MultiLabelClassificationTask
  loss: "bce"
  model: "resnet18"
  learning_rate: 1e-3
  learning_rate_schedule_patience: 6
  weights: null
  in_channels: 12
  num_classes: 19

datamodule:
  _target_: torchgeo.datamodules.BigEarthNetDataModule
  root: "data/BigEarthNet"
  batch_size: 256
  bands: s2
  num_workers: 16

trainer:
  _target_: lightning.pytorch.Trainer
  accelerator: gpu
  devices:
    - 4
    - 6
  min_epochs: 100
  max_epochs: 100

however, if I change the module to the SimCLR trainer, then it doesn't work.

adamjstewart · 2023-04-26T15:44:42Z

What's the stack trace for the error? Only thing I can do is search repos for that error message.

calebrob6 · 2023-04-26T15:47:03Z

Can you try running train.py with multiple GPUs?

calebrob6 · 2023-04-26T15:47:46Z

(geospatiallib) calebrobinson@cdf00374c70c:~/ssdprivate/torchgeo$ python train.py config_file=conf/custom/simclr_bigearthnet2.yaml
Global seed set to 0
GPU available: True (cuda), used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
Traceback (most recent call last):                                                                                                                                                                   File "/home/calebrobinson/ssdprivate/torchgeo/train.py", line 156, in <module>
    main(conf)
  File "/home/calebrobinson/ssdprivate/torchgeo/train.py", line 129, in main
    trainer.fit(model=task, datamodule=datamodule)
  File "/home/calebrobinson/.conda/envs/geospatiallib/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py", line 608, in fit
    call._call_and_handle_interrupt(
  File "/home/calebrobinson/.conda/envs/geospatiallib/lib/python3.10/site-packages/lightning/pytorch/trainer/call.py", line 36, in _call_and_handle_interrupt
    return trainer.strategy.launcher.launch(trainer_fn, *args, trainer=trainer, **kwargs)
  File "/home/calebrobinson/.conda/envs/geospatiallib/lib/python3.10/site-packages/lightning/pytorch/strategies/launchers/multiprocessing.py", line 113, in launch
    mp.start_processes(
  File "/home/calebrobinson/.conda/envs/geospatiallib/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 188, in start_processes
    process.start()
  File "/home/calebrobinson/.conda/envs/geospatiallib/lib/python3.10/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
  File "/home/calebrobinson/.conda/envs/geospatiallib/lib/python3.10/multiprocessing/context.py", line 288, in _Popen
    return Popen(process_obj)
  File "/home/calebrobinson/.conda/envs/geospatiallib/lib/python3.10/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)                                                                                                                                                                    File "/home/calebrobinson/.conda/envs/geospatiallib/lib/python3.10/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)                                                                                                                                                                        File "/home/calebrobinson/.conda/envs/geospatiallib/lib/python3.10/multiprocessing/popen_spawn_posix.py", line 58, in _launch
    self.pid = util.spawnv_passfds(spawn.get_executable(),
  File "/home/calebrobinson/.conda/envs/geospatiallib/lib/python3.10/multiprocessing/util.py", line 452, in spawnv_passfds                                                                             return _posixsubprocess.fork_exec(
ValueError: bad value(s) in fds_to_keep

adamjstewart · 2023-04-26T16:21:26Z

Our node is still down. Been going back and forth with the sys admins for days 😭

adamjstewart · 2023-05-08T19:47:40Z

torchgeo/trainers/simclr.py

+        loss = self.criterion(z1, z2)
+
+        # Calculate the mean normalized standard deviation over features dimensions.
+        # If this is << 1 / sqrt(h1.shape[1]), then the model is not learning anything.


@calebrob6 do you want this in every SSL trainer or no SSL trainers? Want to be consistent.

It is a good thing to monitor in training per the SimSiam paper so we shouldn't remove it just for convenience, however it also isn't urgent to add it to other trainers

I'll add it to MoCo, should only take a minute.

calebrob6 · 2023-05-11T16:37:35Z

torchgeo/trainers/simclr.py

+        loss = self.criterion(z1, z2)
+
+        # Calculate the mean normalized standard deviation over features dimensions.
+        # If this is << 1 / sqrt(h1.shape[1]), then the model is not learning anything.


It is a good thing to monitor in training per the SimSiam paper so we shouldn't remove it just for convenience, however it also isn't urgent to add it to other trainers

isaaccorley added 2 commits April 15, 2023 22:56

add simclr and tests

0d5279d

add lightly to reqs

490b863

isaaccorley requested a review from adamjstewart April 15, 2023 23:01

isaaccorley self-assigned this Apr 15, 2023

github-actions bot added dependencies Packaging and dependencies testing Continuous integration testing trainers PyTorch Lightning trainers labels Apr 15, 2023

adamjstewart reviewed Apr 15, 2023

View reviewed changes

requirements/required.txt Outdated Show resolved Hide resolved

setup.cfg Outdated Show resolved Hide resolved

adamjstewart added this to the 0.5.0 milestone Apr 15, 2023

adamjstewart added this to In progress in SSL4EO-L via automation Apr 15, 2023

adamjstewart added 7 commits April 16, 2023 12:43

Merge branch 'main' into trainers/simclr

a7d6e71

pyupgrade

1dabbcd

Copy things from prior implementation

42f3be0

Add SimCLR v2 projection head

554aa92

Remove kwargs

900d378

Call __init__ explicitly

b580f19

Fix mypy and docs

5a4408f

github-actions bot added the documentation Improvements or additions to documentation label Apr 16, 2023

adamjstewart added 12 commits April 16, 2023 15:53

Can't test newer setuptools

cda57df

Default to output dim of model

5bec352

Add memory bank

b1a51f8

Ignore erroneous warning

4b1a15d

Fix configs, test SSL4EO

845519d

Fix a few layer bugs

776071e

mypy fixes

7b86686

kernel_size must be an integer

3fba197

Fix SeCo in_channels

3b48ed5

Get more coverage

19c5052

Bump min lightly

395fe37

Default logging

1edc96a

adamjstewart closed this Apr 25, 2023

SSL4EO-L automation moved this from In progress to Done Apr 25, 2023

adamjstewart reopened this Apr 25, 2023

SSL4EO-L automation moved this from Done to In progress Apr 25, 2023

adamjstewart closed this Apr 25, 2023

SSL4EO-L automation moved this from In progress to Done Apr 25, 2023

adamjstewart reopened this Apr 25, 2023

SSL4EO-L automation moved this from Done to In progress Apr 25, 2023

adamjstewart marked this pull request as draft May 3, 2023 03:05

adamjstewart added 7 commits May 3, 2023 09:08

Bump lightly version

a93cc23

Merge branch 'main' into trainers/simclr

a42902d

Add RandomGrayscale

27b823d

Flake8 fixes

eb7b912

Placate pydocstyle

19f6935

Clarify docs

3b68504

Pass correct weights

c3349c1

adamjstewart marked this pull request as ready for review May 3, 2023 21:12

adamjstewart reviewed May 8, 2023

View reviewed changes

calebrob6 approved these changes May 11, 2023

View reviewed changes

calebrob6 merged commit ef7a9ad into microsoft:main May 11, 2023
18 checks passed

SSL4EO-L automation moved this from In progress to Done May 11, 2023

isaaccorley deleted the trainers/simclr branch May 23, 2023 18:10

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add SimCLR trainer #1252

Add SimCLR trainer #1252

isaaccorley commented Apr 15, 2023 •

edited by adamjstewart

adamjstewart left a comment

adamjstewart commented Apr 25, 2023

calebrob6 commented Apr 26, 2023

adamjstewart commented Apr 26, 2023

calebrob6 commented Apr 26, 2023 •

edited

adamjstewart commented Apr 26, 2023

calebrob6 commented Apr 26, 2023

calebrob6 commented Apr 26, 2023

adamjstewart commented Apr 26, 2023

adamjstewart May 8, 2023

calebrob6 May 11, 2023

adamjstewart May 11, 2023

calebrob6 May 11, 2023

Add SimCLR trainer #1252

Add SimCLR trainer #1252

Conversation

isaaccorley commented Apr 15, 2023 • edited by adamjstewart

adamjstewart left a comment

Choose a reason for hiding this comment

adamjstewart commented Apr 25, 2023

calebrob6 commented Apr 26, 2023

adamjstewart commented Apr 26, 2023

calebrob6 commented Apr 26, 2023 • edited

adamjstewart commented Apr 26, 2023

calebrob6 commented Apr 26, 2023

calebrob6 commented Apr 26, 2023

adamjstewart commented Apr 26, 2023

adamjstewart May 8, 2023

Choose a reason for hiding this comment

calebrob6 May 11, 2023

Choose a reason for hiding this comment

adamjstewart May 11, 2023

Choose a reason for hiding this comment

calebrob6 May 11, 2023

Choose a reason for hiding this comment

isaaccorley commented Apr 15, 2023 •

edited by adamjstewart

calebrob6 commented Apr 26, 2023 •

edited