[WIP] Engines 3.0 #1230

Scitator · 2021-06-06T10:10:55Z

Before submitting (checklist)

Description

Related Issue

Type of Change

Examples / docs / tutorials / contributors update
Bug fix (non-breaking change which fixes an issue)
Improvement (non-breaking change which improves an existing feature)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to change)

PR review

Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in Github issues there's a high chance it will not be merged.

FAQ

Please review the FAQ before submitting an issue:

I have read the documentation and FAQ
I have reviewed the minimal examples section
I have checked the changelog for main framework updates
I have read the contribution guide
I have joined Catalyst slack (#__questions channel) for issue discussion

github-actions · 2021-06-06T10:18:18Z

catalyst/engines/fairscale.py

+
+if SETTINGS.fairscale_required:
+    from fairscale.nn import Pipe
+    from fairscale.nn.data_parallel import (


[pep8] _{reported by reviewdog 🐶}
N817 camelcase 'FullyShardedDataParallel' imported as acronym 'FSDP'

github-actions · 2021-06-06T10:18:19Z

catalyst/utils/distributed.py

@@ -35,6 +35,14 @@ def _is_ddp_wrapped(model: nn.Module) -> bool:

        parallel_wrappers = parallel_wrappers + (apex_DDP,)

+    if SETTINGS.fairscale_required:
+        from fairscale.nn.data_parallel import (


[pep8] _{reported by reviewdog 🐶}
N817 camelcase 'FullyShardedDataParallel' imported as acronym 'FSDP'

github-actions · 2021-06-06T10:18:19Z

catalyst/utils/distributed.py

+            ShardedDataParallel as ShardedDDP,
+        )
+
+        parallel_wrappers = parallel_wrappers + (ShardedDDP, FSDP,)


[pep8] _{reported by reviewdog 🐶}
C819 trailing comma prohibited

github-actions · 2021-06-07T07:22:43Z

examples/engines/fairscale_ddp.py

+from torch import nn, optim
+from torch.utils.data import DataLoader
+
+from catalyst import dl, utils


[pep8] _{reported by reviewdog 🐶}
F401 'catalyst.utils' imported but unused

github-actions · 2021-06-07T07:22:44Z

examples/engines/fairscale_ddp.py

+from catalyst.data import ToTensor
+
+
+class CustomRunner(dl.IRunner):


[pep8] _{reported by reviewdog 🐶}
D101 Missing docstring in public class

github-actions · 2021-06-07T07:22:44Z

examples/engines/fairscale_ddp.py

+        super().__init__()
+        self._logdir = logdir
+
+    def get_engine(self):


[pep8] _{reported by reviewdog 🐶}
D102 Missing docstring in public method

github-actions · 2021-06-07T07:22:44Z

examples/engines/fairscale_ddp.py

+        # return dl.SharedDataParallelFairScaleAMPEngine()
+        return dl.FullySharedDataParallelFairScaleEngine()
+
+    def get_loggers(self):


[pep8] _{reported by reviewdog 🐶}
D102 Missing docstring in public method

github-actions · 2021-06-07T07:22:44Z

examples/engines/fairscale_ddp.py

+        }
+
+    @property
+    def stages(self):


[pep8] _{reported by reviewdog 🐶}
D102 Missing docstring in public method

github-actions · 2021-06-07T07:22:45Z

examples/engines/fairscale_pipe.py

+        )
+        return model
+
+    def get_criterion(self, stage: str):


[pep8] _{reported by reviewdog 🐶}
D102 Missing docstring in public method

github-actions · 2021-06-07T07:22:45Z

examples/engines/fairscale_pipe.py

+    def get_criterion(self, stage: str):
+        return nn.CrossEntropyLoss()
+
+    def get_optimizer(self, stage: str, model):


[pep8] _{reported by reviewdog 🐶}
D102 Missing docstring in public method

github-actions · 2021-06-07T07:22:46Z

examples/engines/fairscale_pipe.py

+    def get_optimizer(self, stage: str, model):
+        return optim.Adam(model.parameters(), lr=1e-3)
+
+    def get_scheduler(self, stage: str, optimizer):


[pep8] _{reported by reviewdog 🐶}
D102 Missing docstring in public method

github-actions · 2021-06-07T07:22:46Z

examples/engines/fairscale_pipe.py

+    def get_scheduler(self, stage: str, optimizer):
+        return None
+
+    def get_callbacks(self, stage: str):


[pep8] _{reported by reviewdog 🐶}
D102 Missing docstring in public method

github-actions · 2021-06-07T07:22:46Z

examples/engines/fairscale_pipe.py

+            ),
+        }
+
+    def handle_batch(self, batch):


[pep8] _{reported by reviewdog 🐶}
D102 Missing docstring in public method

pep8speaks · 2021-06-08T07:03:57Z

Hello @Scitator! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

In the file examples/engines/multi_gpu.py:

Line 41:61: E231 missing whitespace after ','

Comment last updated at 2021-06-13 14:21:23 UTC

github-actions · 2021-06-08T07:11:17Z

examples/engines/multi_gpu.py

+if SETTINGS.fairscale_required:
+    E2E.update(
+        {
+            "fs-pp": partial(dl.PipelineParallelFairScaleEngine, pipe_kwargs=dict(balance=[3, 4])),


[pep8] _{reported by reviewdog 🐶}
C408 Unnecessary dict call - rewrite as a literal.

github-actions · 2021-06-08T07:11:17Z

examples/engines/multi_gpu.py

+# tested with `docker pull deepspeed/deepspeed:v031_torch17_cuda11`
+if SETTINGS.deepspeed_required:
+    E2E.update(
+        {"ds-ddp": dl.DistributedDataParallelDeepSpeedEngine,}


[pep8] _{reported by reviewdog 🐶}
E231 missing whitespace after ','

github-actions · 2021-06-08T07:11:17Z

examples/engines/multi_gpu.py

+# tested with `docker pull deepspeed/deepspeed:v031_torch17_cuda11`
+if SETTINGS.deepspeed_required:
+    E2E.update(
+        {"ds-ddp": dl.DistributedDataParallelDeepSpeedEngine,}


[pep8] _{reported by reviewdog 🐶}
C819 trailing comma prohibited

github-actions · 2021-06-08T07:11:17Z

examples/engines/multi_gpu.py

+    )
+
+
+def conv_block(in_channels, out_channels, pool=False):


[pep8] _{reported by reviewdog 🐶}
D103 Missing docstring in public function

github-actions · 2021-06-08T07:11:17Z

examples/engines/multi_gpu.py

+    return nn.Sequential(*layers)
+
+
+def resnet9(in_channels: int, num_classes: int, size: int = 16):


[pep8] _{reported by reviewdog 🐶}
D103 Missing docstring in public function

github-actions · 2021-06-08T07:11:19Z

catalyst/contrib/datasets/cifar.py

+                + " You can use download=True to download it"
+            )
+        with open(path, "rb") as infile:
+            data = pickle.load(infile, encoding="latin1")


[pep8] _{reported by reviewdog 🐶}
WPS442 Found outer scope names shadowing: data

github-actions · 2021-06-08T07:11:19Z

catalyst/contrib/datasets/cifar.py

+
+        return img, target
+
+    def __len__(self) -> int:


[pep8] _{reported by reviewdog 🐶}
D105 Missing docstring in magic method

github-actions · 2021-06-08T07:11:19Z

catalyst/contrib/datasets/cifar.py

+                return False
+        return True
+
+    def download(self) -> None:


[pep8] _{reported by reviewdog 🐶}
D102 Missing docstring in public method

github-actions · 2021-06-08T07:11:19Z

catalyst/contrib/datasets/cifar.py

+            return
+        download_and_extract_archive(self.url, self.root, filename=self.filename, md5=self.tgz_md5)
+
+    def extra_repr(self) -> str:


[pep8] _{reported by reviewdog 🐶}
D102 Missing docstring in public method

github-actions · 2021-06-08T07:11:20Z

catalyst/utils/distributed.py

@@ -207,11 +219,21 @@ def all_gather(data: Any) -> List[Any]:
    return data_list


+def ddp_sync_run(function: Callable):


[pep8] _{reported by reviewdog 🐶}
D103 Missing docstring in public function

review-notebook-app · 2021-06-11T06:52:50Z

Check out this pull request on

See visual diffs & provide feedback on Jupyter Notebooks.

Powered by ReviewNB

github-actions · 2021-06-13T10:05:05Z

examples/engines/tmp.py

+from catalyst.data import ToTensor
+
+
+class CustomRunner(dl.IRunner):


[pep8] _{reported by reviewdog 🐶}
D101 Missing docstring in public class

github-actions · 2021-06-13T10:05:05Z

examples/engines/tmp.py

+        super().__init__()
+        self._logdir = logdir
+
+    def get_engine(self):


[pep8] _{reported by reviewdog 🐶}
D102 Missing docstring in public method

github-actions · 2021-06-13T10:05:05Z

examples/engines/tmp.py

+    def get_engine(self):
+        return dl.DistributedDataParallelEngine()
+
+    def get_loggers(self):


[pep8] _{reported by reviewdog 🐶}
D102 Missing docstring in public method

github-actions · 2021-06-13T10:05:05Z

examples/engines/tmp.py

+        }
+
+    @property
+    def stages(self):


[pep8] _{reported by reviewdog 🐶}
D102 Missing docstring in public method

github-actions · 2021-06-13T10:05:05Z

examples/engines/tmp.py

+    def stages(self):
+        return ["train_freezed", "train_unfreezed"]
+
+    def get_stage_len(self, stage: str) -> int:


[pep8] _{reported by reviewdog 🐶}
D102 Missing docstring in public method

github-actions · 2021-06-13T10:05:06Z

examples/engines/tmp.py

+            utils.set_requires_grad(model, True)
+        return model
+
+    def get_criterion(self, stage: str):


[pep8] _{reported by reviewdog 🐶}
D102 Missing docstring in public method

github-actions · 2021-06-13T10:05:06Z

examples/engines/tmp.py

+    def get_criterion(self, stage: str):
+        return nn.CrossEntropyLoss()
+
+    def get_optimizer(self, stage: str, model):


[pep8] _{reported by reviewdog 🐶}
D102 Missing docstring in public method

github-actions · 2021-06-13T10:05:06Z

examples/engines/tmp.py

+        else:
+            return optim.SGD(model.parameters(), lr=1e-1)
+
+    def get_callbacks(self, stage: str):


[pep8] _{reported by reviewdog 🐶}
D102 Missing docstring in public method

github-actions · 2021-06-13T10:05:06Z

examples/engines/tmp.py

+            "verbose": dl.TqdmCallback(),
+        }
+
+    def handle_batch(self, batch):


[pep8] _{reported by reviewdog 🐶}
D102 Missing docstring in public method

github-actions · 2021-06-13T10:05:06Z

catalyst/engines/fairscale.py

+        scheduler = self.sync_device(scheduler)
+
+
+        return model, criterion, optimizer, scheduler


[pep8] _{reported by reviewdog 🐶}
E303 too many blank lines (2)

github-actions · 2021-06-13T12:36:31Z

catalyst/engines/fairscale.py

+    def init_components(
+        self, model_fn=None, criterion_fn=None, optimizer_fn=None, scheduler_fn=None,
+    ):
+        """Inits the runs components."""


[pep8] _{reported by reviewdog 🐶}
D202 No blank lines allowed after function docstring

github-actions · 2021-06-13T12:36:32Z

catalyst/engines/fairscale.py

+        return pipe_model, criterion, optimizer, scheduler
+
+    def deinit_components(self, runner):
+        """Deinits the runs components.


[pep8] _{reported by reviewdog 🐶}
DAR101 Missing parameter(s) in Docstring: - runner

github-actions · 2021-06-13T13:01:47Z

catalyst/contrib/datasets/cifar.py

+        self.targets = []
+
+        # now load the picked numpy arrays
+        for file_name, checksum in downloaded_list:


[pep8] _{reported by reviewdog 🐶}
B007 Loop control variable 'checksum' not used within the loop body. If this is intended, start the name with an underscore.

github-actions · 2021-06-13T13:01:47Z

catalyst/contrib/datasets/cifar.py

+                + " You can use download=True to download it"
+            )
+        with open(path, "rb") as infile:
+            data = pickle.load(infile, encoding="latin1")


[pep8] _{reported by reviewdog 🐶}
WPS442 Found outer scope names shadowing: data

github-actions · 2021-06-13T13:01:47Z

catalyst/contrib/datasets/cifar.py

+
+        return img, target
+
+    def __len__(self) -> int:


[pep8] _{reported by reviewdog 🐶}
D105 Missing docstring in magic method

github-actions · 2021-06-13T13:01:47Z

catalyst/contrib/datasets/cifar.py

+                return False
+        return True
+
+    def download(self) -> None:


[pep8] _{reported by reviewdog 🐶}
D102 Missing docstring in public method

github-actions · 2021-06-13T13:01:47Z

catalyst/contrib/datasets/cifar.py

+            return
+        download_and_extract_archive(self.url, self.root, filename=self.filename, md5=self.tgz_md5)
+
+    def extra_repr(self) -> str:


[pep8] _{reported by reviewdog 🐶}
D102 Missing docstring in public method

github-actions · 2021-06-13T13:01:48Z

catalyst/engines/fairscale.py

+
+if SETTINGS.fairscale_required:
+    from fairscale.nn import Pipe
+    from fairscale.nn.data_parallel import (


[pep8] _{reported by reviewdog 🐶}
N817 camelcase 'FullyShardedDataParallel' imported as acronym 'FSDP'

github-actions · 2021-06-13T13:01:48Z

catalyst/engines/fairscale.py

+    def init_components(
+        self, model_fn=None, criterion_fn=None, optimizer_fn=None, scheduler_fn=None,
+    ):
+        """Inits the runs components."""


[pep8] _{reported by reviewdog 🐶}
D202 No blank lines allowed after function docstring

github-actions · 2021-06-13T13:01:48Z

catalyst/engines/fairscale.py

+        return pipe_model, criterion, optimizer, scheduler
+
+    def deinit_components(self, runner):
+        """Deinits the runs components.


[pep8] _{reported by reviewdog 🐶}
DAR101 Missing parameter(s) in Docstring: - runner

github-actions · 2021-06-13T13:52:15Z

catalyst/utils/distributed.py

+    if SETTINGS.fairscale_required:
+        from fairscale.nn.data_parallel import FullyShardedDataParallel, ShardedDataParallel
+
+        parallel_wrappers = parallel_wrappers + (ShardedDataParallel, FullyShardedDataParallel,)


[pep8] _{reported by reviewdog 🐶}
C819 trailing comma prohibited

fairscale draft

da0c03c

Scitator requested review from bagxi and ditwoo as code owners June 6, 2021 10:10

github-actions bot reviewed Jun 6, 2021

View reviewed changes

Scitator added 2 commits June 6, 2021 14:27

deepspeed draft

5d5372e

fairscale draft examples

1bf80e3

github-actions bot reviewed Jun 7, 2021

View reviewed changes

example update

259ff07

github-actions bot reviewed Jun 8, 2021

View reviewed changes

deepspeed tested

4126fd1

Scitator added 5 commits June 11, 2021 10:07

deepspeed updated

48f79b8

codestyle

1e3afa2

codestyle

1068c3e

dpp logge fix

0a2e482

dpp update

ca7b156

github-actions bot reviewed Jun 13, 2021

View reviewed changes

Scitator added 2 commits June 13, 2021 15:28

dpp update

6913c71

dpp update

1c2b89d

github-actions bot reviewed Jun 13, 2021

View reviewed changes

Scitator added 3 commits June 13, 2021 15:48

dpp update

879a1ad

codestyle

f8aad1f

fairscale update

0ef1a0e

github-actions bot reviewed Jun 13, 2021

View reviewed changes

fairscale update

3b625ca

github-actions bot reviewed Jun 13, 2021

View reviewed changes

Scitator added 3 commits June 13, 2021 16:56

codestyle

3f9b956

hydra hotfix

b67c3cf

hydra hotfix

fbef2f2

Scitator merged commit ed363dd into master Jun 13, 2021

mergify bot deleted the feature/engines-3 branch June 13, 2021 14:39

		from catalyst.data import ToTensor


		class CustomRunner(dl.IRunner):

		return nn.Sequential(*layers)


		def resnet9(in_channels: int, num_classes: int, size: int = 16):

		@@ -207,11 +219,21 @@ def all_gather(data: Any) -> List[Any]:
		return data_list


		def ddp_sync_run(function: Callable):

		scheduler = self.sync_device(scheduler)


		return model, criterion, optimizer, scheduler

[WIP] Engines 3.0 #1230

[WIP] Engines 3.0 #1230

Conversation

Scitator commented Jun 6, 2021

Before submitting (checklist)

Description

Related Issue

Type of Change

PR review

FAQ

github-actions bot Jun 6, 2021

Choose a reason for hiding this comment

github-actions bot Jun 6, 2021

Choose a reason for hiding this comment

github-actions bot Jun 6, 2021

Choose a reason for hiding this comment

github-actions bot Jun 7, 2021

Choose a reason for hiding this comment

github-actions bot Jun 7, 2021

Choose a reason for hiding this comment

github-actions bot Jun 7, 2021

Choose a reason for hiding this comment

github-actions bot Jun 7, 2021

Choose a reason for hiding this comment

github-actions bot Jun 7, 2021

Choose a reason for hiding this comment

github-actions bot Jun 7, 2021

Choose a reason for hiding this comment

github-actions bot Jun 7, 2021

Choose a reason for hiding this comment

github-actions bot Jun 7, 2021

Choose a reason for hiding this comment

github-actions bot Jun 7, 2021

Choose a reason for hiding this comment

github-actions bot Jun 7, 2021

Choose a reason for hiding this comment

pep8speaks commented Jun 8, 2021 • edited

Comment last updated at 2021-06-13 14:21:23 UTC

github-actions bot Jun 8, 2021

Choose a reason for hiding this comment

github-actions bot Jun 8, 2021

Choose a reason for hiding this comment

github-actions bot Jun 8, 2021

Choose a reason for hiding this comment

github-actions bot Jun 8, 2021

Choose a reason for hiding this comment

github-actions bot Jun 8, 2021

Choose a reason for hiding this comment

github-actions bot Jun 8, 2021

Choose a reason for hiding this comment

github-actions bot Jun 8, 2021

Choose a reason for hiding this comment

github-actions bot Jun 8, 2021

Choose a reason for hiding this comment

github-actions bot Jun 8, 2021

Choose a reason for hiding this comment

github-actions bot Jun 8, 2021

Choose a reason for hiding this comment

review-notebook-app bot commented Jun 11, 2021

github-actions bot Jun 13, 2021

Choose a reason for hiding this comment

github-actions bot Jun 13, 2021

Choose a reason for hiding this comment

github-actions bot Jun 13, 2021

Choose a reason for hiding this comment

github-actions bot Jun 13, 2021

Choose a reason for hiding this comment

github-actions bot Jun 13, 2021

Choose a reason for hiding this comment

github-actions bot Jun 13, 2021

Choose a reason for hiding this comment

github-actions bot Jun 13, 2021

Choose a reason for hiding this comment

github-actions bot Jun 13, 2021

Choose a reason for hiding this comment

github-actions bot Jun 13, 2021

Choose a reason for hiding this comment

github-actions bot Jun 13, 2021

Choose a reason for hiding this comment

github-actions bot Jun 13, 2021

pep8speaks commented Jun 8, 2021 •

edited