add colocation identifier #16796

nicolai86 · 2023-02-17T15:39:27Z

What does this PR do?

Expose colocation_group_id; this will allow users of the lightning platform to use high speed interconnects.
As this has a couple of moving pieces I'll circle back and add documentation once this feature works end to end in production.

Fixes #<issue_number>

Before submitting

Was this discussed/agreed via a GitHub issue? (not for typos and docs)
Did you read the contributor guideline, Pull Request section?
Did you make sure your PR does only one thing, instead of bundling different changes together?
Did you make sure to update the documentation with your changes? (if necessary)
Did you write any new necessary tests? (not for typos and docs)
Did you verify new and existing tests pass locally with your changes?
Did you list all the breaking changes introduced by this pull request?
Did you update the CHANGELOG? (not for typos, docs, test updates, or minor internal changes/refactors)

PR review

Anyone in the community is welcome to review the PR.
Before you start reviewing, make sure you have read the review guidelines. In short, see the following bullet-list:

Reviewer checklist

Is this pull request ready for review? (if not, please submit in draft mode)
Check that all items from Before submitting are resolved
Make sure the title is self-explanatory and the description concisely explains the PR
Add labels and milestones (and optionally projects) to the PR so it can be classified

cc @Borda

github-actions · 2023-02-17T15:44:22Z

⚡ Required checks status: All passing 🟢

Groups summary

🟢 lightning_app: Tests workflow

Check ID	Status
app-pytest (macOS-11, lightning, 3.8, latest)	success	✅
app-pytest (macOS-11, lightning, 3.8, oldest)	success	✅
app-pytest (macOS-11, app, 3.9, latest)	success	✅
app-pytest (ubuntu-20.04, lightning, 3.8, latest)	success	✅
app-pytest (ubuntu-20.04, lightning, 3.8, oldest)	success	✅
app-pytest (ubuntu-20.04, app, 3.9, latest)	success	✅
app-pytest (windows-2022, lightning, 3.8, latest)	success	✅
app-pytest (windows-2022, lightning, 3.8, oldest)	success	✅
app-pytest (windows-2022, app, 3.8, latest)	success	✅

These checks are required after the changes to src/lightning/app/runners/cloud.py, src/lightning/app/storage/orchestrator.py, src/lightning/app/utilities/packaging/cloud_compute.py, tests/tests_app/core/test_lightning_app.py, tests/tests_app/core/test_lightning_flow.py, tests/tests_app/structures/test_structures.py, tests/tests_app/utilities/test_load_app.py.

🟢 lightning_app: Examples

Check ID	Status
app-examples (macOS-11, lightning, 3.9, latest)	success	✅
app-examples (macOS-11, lightning, 3.9, oldest)	success	✅
app-examples (macOS-11, app, 3.9, latest)	success	✅
app-examples (ubuntu-20.04, lightning, 3.9, latest)	success	✅
app-examples (ubuntu-20.04, lightning, 3.9, oldest)	success	✅
app-examples (ubuntu-20.04, app, 3.9, latest)	success	✅
app-examples (windows-2022, lightning, 3.9, latest)	success	✅
app-examples (windows-2022, lightning, 3.9, oldest)	success	✅
app-examples (windows-2022, app, 3.9, latest)	success	✅

These checks are required after the changes to src/lightning/app/runners/cloud.py, src/lightning/app/storage/orchestrator.py, src/lightning/app/utilities/packaging/cloud_compute.py.

🟢 lightning_app: Docs

Check ID	Status
make-doctest (app)	success	✅
make-html (app)	success	✅

These checks are required after the changes to src/lightning/app/runners/cloud.py, src/lightning/app/storage/orchestrator.py, src/lightning/app/utilities/packaging/cloud_compute.py.

🟢 mypy

Check ID	Status
mypy	success	✅

These checks are required after the changes to src/lightning/app/runners/cloud.py, src/lightning/app/storage/orchestrator.py, src/lightning/app/utilities/packaging/cloud_compute.py.

🟢 install

Check ID	Status
install-pkg (ubuntu-22.04, app, 3.8)	success	✅
install-pkg (ubuntu-22.04, app, 3.10)	success	✅
install-pkg (ubuntu-22.04, fabric, 3.8)	success	✅
install-pkg (ubuntu-22.04, fabric, 3.10)	success	✅
install-pkg (ubuntu-22.04, pytorch, 3.8)	success	✅
install-pkg (ubuntu-22.04, pytorch, 3.10)	success	✅
install-pkg (ubuntu-22.04, lightning, 3.8)	success	✅
install-pkg (ubuntu-22.04, lightning, 3.10)	success	✅
install-pkg (ubuntu-22.04, notset, 3.8)	success	✅
install-pkg (ubuntu-22.04, notset, 3.10)	success	✅
install-pkg (macOS-12, app, 3.8)	success	✅
install-pkg (macOS-12, app, 3.10)	success	✅
install-pkg (macOS-12, fabric, 3.8)	success	✅
install-pkg (macOS-12, fabric, 3.10)	success	✅
install-pkg (macOS-12, pytorch, 3.8)	success	✅
install-pkg (macOS-12, pytorch, 3.10)	success	✅
install-pkg (macOS-12, lightning, 3.8)	success	✅
install-pkg (macOS-12, lightning, 3.10)	success	✅
install-pkg (macOS-12, notset, 3.8)	success	✅
install-pkg (macOS-12, notset, 3.10)	success	✅
install-pkg (windows-2022, app, 3.8)	success	✅
install-pkg (windows-2022, app, 3.10)	success	✅
install-pkg (windows-2022, fabric, 3.8)	success	✅
install-pkg (windows-2022, fabric, 3.10)	success	✅
install-pkg (windows-2022, pytorch, 3.8)	success	✅
install-pkg (windows-2022, pytorch, 3.10)	success	✅
install-pkg (windows-2022, lightning, 3.8)	success	✅
install-pkg (windows-2022, lightning, 3.10)	success	✅
install-pkg (windows-2022, notset, 3.8)	success	✅
install-pkg (windows-2022, notset, 3.10)	success	✅

These checks are required after the changes to src/lightning/app/runners/cloud.py, src/lightning/app/storage/orchestrator.py, src/lightning/app/utilities/packaging/cloud_compute.py.

Thank you for your contribution! 💜

Note
This comment is automatically generated and updates for 60 minutes every 180 seconds. If you have any other questions, contact carmocca for help.

tchaton

LGTM !
Maybe a test ;)

src/lightning/app/utilities/packaging/cloud_compute.py

awaelchli · 2023-02-17T23:29:22Z

@nicolai86 Some tests need adjustment because of the new attribute in the state.

Borda · 2023-03-29T12:13:39Z

@awaelchli could you help here to finish it?

awaelchli · 2023-04-02T21:25:59Z

@nicolai86 Should this work be continued?

lantiga · 2023-05-05T06:36:04Z

Yes we are going to need it as a general attribute, but let's align with the rest of the team before continuing.

Co-authored-by: Adrian Wälchli <aedu.waelchli@gmail.com>

Co-authored-by: Adrian Wälchli <aedu.waelchli@gmail.com> Co-authored-by: Justus Schock <12886177+justusschock@users.noreply.github.com> Co-authored-by: Jirka Borovec <6035284+Borda@users.noreply.github.com> Co-authored-by: thomas chaton <thomas@grid.ai> Co-authored-by: thomas <thomas@thomass-MacBook-Pro.local> (cherry picked from commit b9dd977)

justusschock and others added 2 commits January 19, 2023 23:21

start colocation

31ba41f

Merge branch 'master' into feat/colocation

610d940

github-actions bot added the app Generic label for Lightning App package label Feb 17, 2023

remove merge conflict

694f162

nicolai86 marked this pull request as ready for review February 17, 2023 15:43

nicolai86 requested review from tchaton, lantiga, awaelchli, hhsecond and ethanwharris as code owners February 17, 2023 15:43

nicolai86 changed the title ~~add colocation identifier~~ [app] add colocation identifier Feb 17, 2023

nicolai86 changed the title ~~[app] add colocation identifier~~ add colocation identifier Feb 17, 2023

justusschock approved these changes Feb 17, 2023

View reviewed changes

doc fmt

cc8857e

tchaton approved these changes Feb 17, 2023

View reviewed changes

src/lightning/app/utilities/packaging/cloud_compute.py Show resolved Hide resolved

mergify bot added the ready PRs ready to be merged label Feb 17, 2023

awaelchli approved these changes Feb 17, 2023

View reviewed changes

src/lightning/app/utilities/packaging/cloud_compute.py Outdated Show resolved Hide resolved

src/lightning/app/utilities/packaging/cloud_compute.py Outdated Show resolved Hide resolved

awaelchli added this to the v1.9.x milestone Feb 17, 2023

Borda added 2 commits February 28, 2023 01:20

Merge branch 'master' into feat/colocation

a531878

Merge branch 'master' into feat/colocation

43dff3b

Borda modified the milestones: v1.9.x, 2.0.x Mar 29, 2023

Merge branch 'master' into feat/colocation

7187efb

Borda marked this pull request as draft May 5, 2023 07:02

mergify bot removed the ready PRs ready to be merged label May 5, 2023

Merge branch 'master' into feat/colocation

f456705

tchaton marked this pull request as ready for review June 28, 2023 04:35

Update src/lightning/app/utilities/packaging/cloud_compute.py

c605ae9

Co-authored-by: Adrian Wälchli <aedu.waelchli@gmail.com>

mergify bot added the ready PRs ready to be merged label Jun 28, 2023

tchaton and others added 3 commits June 28, 2023 05:36

Update src/lightning/app/utilities/packaging/cloud_compute.py

4da9838

Co-authored-by: Adrian Wälchli <aedu.waelchli@gmail.com>

update

dcac043

update

6f4c184

Borda approved these changes Jun 28, 2023

View reviewed changes

thomas added 3 commits June 28, 2023 09:43

update

899bbb7

update

f8b820a

update

345735a

tchaton enabled auto-merge (squash) June 28, 2023 09:27

tchaton disabled auto-merge June 28, 2023 09:27

thomas added 3 commits June 28, 2023 10:35

update

3045ef7

update

d7e6aa7

update

508c8e1

ethanwharris approved these changes Jun 28, 2023

View reviewed changes

tchaton enabled auto-merge (squash) June 28, 2023 10:42

tchaton merged commit b9dd977 into master Jun 28, 2023
86 checks passed

tchaton deleted the feat/colocation branch June 28, 2023 10:58

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add colocation identifier #16796

add colocation identifier #16796

nicolai86 commented Feb 17, 2023 •

edited by github-actions bot

github-actions bot commented Feb 17, 2023 •

edited

tchaton left a comment

awaelchli commented Feb 17, 2023

Borda commented Mar 29, 2023

awaelchli commented Apr 2, 2023

lantiga commented May 5, 2023 •

edited

add colocation identifier #16796

add colocation identifier #16796

Conversation

nicolai86 commented Feb 17, 2023 • edited by github-actions bot

What does this PR do?

PR review

github-actions bot commented Feb 17, 2023 • edited

⚡ Required checks status: All passing 🟢

Groups summary

tchaton left a comment

Choose a reason for hiding this comment

awaelchli commented Feb 17, 2023

Borda commented Mar 29, 2023

awaelchli commented Apr 2, 2023

lantiga commented May 5, 2023 • edited

nicolai86 commented Feb 17, 2023 •

edited by github-actions bot

github-actions bot commented Feb 17, 2023 •

edited

lantiga commented May 5, 2023 •

edited