Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

XLA PoC #1275

Merged
merged 16 commits into from Aug 20, 2021
Merged

XLA PoC #1275

merged 16 commits into from Aug 20, 2021

Conversation

Scitator
Copy link
Member

@Scitator Scitator commented Aug 15, 2021 •
edited

Before submitting (checklist)

  • Was this discussed/approved via a Github issue? (no need for typos and docs improvements)
  • Did you read the contribution guide?
  • Did you check the code style? catalyst-make-codestyle && catalyst-check-codestyle (pip install -U catalyst-codestyle).
  • Did you make sure to update the docs? We use Google format for all the methods and classes.
  • Did you check the docs with make check-docs?
  • Did you write any new necessary tests?
  • Did you check that your code passes the unit tests pytest . ?
  • Did you add your new functionality to the docs?
  • Did you update the CHANGELOG?
  • Did you run colab minimal CI/CD with latest and minimal requirements?

Description

PoC: https://colab.research.google.com/drive/1jG8qcp8plm7YEsFoeIInxtKZoxIRz1CT#scrollTo=r9mDrhuETaPG

Related Issue

Type of Change

  • Examples / docs / tutorials / contributors update
  • Bug fix (non-breaking change which fixes an issue)
  • Improvement (non-breaking change which improves an existing feature)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)

PR review

Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in Github issues there's a high chance it will not be merged.

FAQ

Please review the FAQ before submitting an issue:

github-actions[bot]
github-actions bot reviewed Aug 15, 2021
View reviewed changes
catalyst/engines/xla.py Show resolved Hide resolved
catalyst/engines/xla.py Show resolved Hide resolved
@Scitator Scitator force-pushed the engines/tpu branch 3 times, most recently from b03b044 to 8cb5eab Compare August 15, 2021 10:32
github-actions[bot]
github-actions bot reviewed Aug 15, 2021
View reviewed changes
catalyst/core/engine.py Show resolved Hide resolved
catalyst/engines/xla.py Outdated Show resolved Hide resolved
catalyst/engines/xla.py Outdated Show resolved Hide resolved
catalyst/engines/torch.py Outdated
@@ -1,4 +1,4 @@
from typing import Any, Dict, List, Mapping, Tuple, Union
from typing import Any, Callable, Dict, List, Mapping, Tuple, Union
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[pep8] reported by reviewdog 🐶
F401 'typing.Callable' imported but unused

@Scitator Scitator force-pushed the engines/tpu branch 3 times, most recently from fe1c5b9 to dfd6ae6 Compare August 15, 2021 11:47
@Scitator Scitator force-pushed the engines/tpu branch 6 times, most recently from 57f5adb to 53d9ddd Compare August 15, 2021 13:01
@pep8speaks
Copy link

pep8speaks commented Aug 15, 2021 •
edited

Hello @Scitator! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

Line 34:61: E231 missing whitespace after ','

Comment last updated at 2021-08-20 06:52:53 UTC

@Scitator Scitator force-pushed the engines/tpu branch 10 times, most recently from 2538074 to 758ec82 Compare August 15, 2021 14:03
github-actions[bot]
github-actions bot reviewed Aug 18, 2021
View reviewed changes
catalyst/core/engine.py Show resolved Hide resolved
catalyst/core/engine.py Show resolved Hide resolved
catalyst/engines/deepspeed.py
def spawn(self, fn: Callable, *args: Any, **kwargs: Any) -> None:
"""Spawns abstraction for``nprocs`` creation with specified ``fn`` and ``args``/``kwargs``.

Args:
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[pep8] reported by reviewdog 🐶
DAR101 Missing parameter(s) in Docstring: - **kwargs

catalyst/engines/deepspeed.py
def spawn(self, fn: Callable, *args: Any, **kwargs: Any) -> None:
"""Spawns abstraction for``nprocs`` creation with specified ``fn`` and ``args``/``kwargs``.

Args:
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[pep8] reported by reviewdog 🐶
DAR101 Missing parameter(s) in Docstring: - *args

catalyst/engines/xla.py
def spawn(self, fn: Callable, *args: Any, **kwargs: Any) -> None:
"""Spawns abstraction for``nprocs`` creation with specified ``fn`` and ``args``/``kwargs``.

Args:
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[pep8] reported by reviewdog 🐶
DAR101 Missing parameter(s) in Docstring: - **kwargs

catalyst/engines/xla.py
def spawn(self, fn: Callable, *args: Any, **kwargs: Any) -> None:
"""Spawns abstraction for``nprocs`` creation with specified ``fn`` and ``args``/``kwargs``.

Args:
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[pep8] reported by reviewdog 🐶
DAR101 Missing parameter(s) in Docstring: - *args

catalyst/engines/torch.py Show resolved Hide resolved
catalyst/engines/torch.py Show resolved Hide resolved
github-actions[bot]
github-actions bot reviewed Aug 18, 2021
View reviewed changes
catalyst/utils/distributed.py
@@ -64,20 +72,57 @@ def get_nn_from_ddp_module(model: nn.Module) -> nn.Module:
return model


def get_backend() -> Optional[str]:
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[pep8] reported by reviewdog 🐶
D103 Missing docstring in public function

catalyst/utils/distributed.py
# pass


def get_world_size() -> int:
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[pep8] reported by reviewdog 🐶
D103 Missing docstring in public function

@Scitator Scitator force-pushed the engines/tpu branch 8 times, most recently from b865672 to 3c88377 Compare August 18, 2021 19:03
@mergify
Copy link

mergify bot commented Aug 19, 2021

This pull request is now in conflicts. @Scitator, could you fix it? 🙏

@mergify
Copy link

mergify bot commented Aug 19, 2021

This pull request is now in conflicts. @Scitator, could you fix it? 🙏

@review-notebook-app
Copy link

Check out this pull request on  ReviewNB

See visual diffs & provide feedback on Jupyter Notebooks.

Powered by ReviewNB

github-actions[bot]
github-actions bot reviewed Aug 20, 2021
View reviewed changes
catalyst/engines/amp.py Outdated
@@ -238,12 +304,15 @@ def get_engine(self):
stages:
...

.. _convert_sync_batchnorm: https://pytorch.org/docs/stable/generated/torch.nn.SyncBatchNorm.html#
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[pep8] reported by reviewdog 🐶
E501 line too long (102 > 99 characters)

catalyst/engines/amp.py Outdated
@@ -238,12 +304,15 @@ def get_engine(self):
stages:
...

.. _convert_sync_batchnorm: https://pytorch.org/docs/stable/generated/torch.nn.SyncBatchNorm.html#
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[pep8] reported by reviewdog 🐶
W505 doc line too long (102 > 99 characters)

catalyst/engines/fairscale.py Outdated
@@ -203,103 +199,18 @@ def get_engine(self):

stages:
...

.. _convert_sync_batchnorm: https://pytorch.org/docs/stable/generated/torch.nn.SyncBatchNorm.html#
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[pep8] reported by reviewdog 🐶
E501 line too long (102 > 99 characters)

catalyst/engines/fairscale.py Outdated
@@ -203,103 +199,18 @@ def get_engine(self):

stages:
...

.. _convert_sync_batchnorm: https://pytorch.org/docs/stable/generated/torch.nn.SyncBatchNorm.html#
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[pep8] reported by reviewdog 🐶
W505 doc line too long (102 > 99 characters)

catalyst/engines/fairscale.py Outdated
@@ -424,6 +326,8 @@ def get_engine(self):
stages:
...

.. _convert_sync_batchnorm: https://pytorch.org/docs/stable/generated/torch.nn.SyncBatchNorm.html#
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[pep8] reported by reviewdog 🐶
E501 line too long (102 > 99 characters)

catalyst/engines/fairscale.py Outdated
@@ -526,6 +433,8 @@ def get_engine(self):
stages:
...

.. _convert_sync_batchnorm: https://pytorch.org/docs/stable/generated/torch.nn.SyncBatchNorm.html#
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[pep8] reported by reviewdog 🐶
W505 doc line too long (102 > 99 characters)

catalyst/engines/torch.py Outdated
@@ -357,19 +369,24 @@ def get_engine(self):
stages:
...

.. _convert_sync_batchnorm: https://pytorch.org/docs/stable/generated/torch.nn.SyncBatchNorm.html#
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[pep8] reported by reviewdog 🐶
E501 line too long (102 > 99 characters)

catalyst/engines/torch.py Outdated
@@ -357,19 +369,24 @@ def get_engine(self):
stages:
...

.. _convert_sync_batchnorm: https://pytorch.org/docs/stable/generated/torch.nn.SyncBatchNorm.html#
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[pep8] reported by reviewdog 🐶
W505 doc line too long (102 > 99 characters)

catalyst/engines/apex.py Outdated
@@ -439,19 +442,27 @@ def get_engine(self):

stages:
...

.. _convert_sync_batchnorm: https://pytorch.org/docs/stable/generated/torch.nn.SyncBatchNorm.html#
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[pep8] reported by reviewdog 🐶
E501 line too long (102 > 99 characters)

catalyst/engines/apex.py Outdated
@@ -439,19 +442,27 @@ def get_engine(self):

stages:
...

.. _convert_sync_batchnorm: https://pytorch.org/docs/stable/generated/torch.nn.SyncBatchNorm.html#
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[pep8] reported by reviewdog 🐶
W505 doc line too long (102 > 99 characters)

@Scitator Scitator merged commit 6a2b929 into master Aug 20, 2021
@mergify mergify bot deleted the engines/tpu branch August 20, 2021 07:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@github-actions github-actions[bot] github-actions left review comments

@bagxi bagxi Awaiting requested review from bagxi bagxi is a code owner

@ditwoo ditwoo Awaiting requested review from ditwoo

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@Scitator @pep8speaks