[App] Add support for private data #16738

tchaton · 2023-02-13T08:56:15Z

What does this PR do?

This PR enables to create private data connection to s3.

LIGHTNING_CLOUD_URL=http://localhost:9800/ lightning connect data private_bucket_2 --region us-east-1 --source s3://private-data-demo --secret_arn_name role

Fixes #<issue_number>

Before submitting

Was this discussed/agreed via a GitHub issue? (not for typos and docs)
Did you read the contributor guideline, Pull Request section?
Did you make sure your PR does only one thing, instead of bundling different changes together?
Did you make sure to update the documentation with your changes? (if necessary)
Did you write any new necessary tests? (not for typos and docs)
Did you verify new and existing tests pass locally with your changes?
Did you list all the breaking changes introduced by this pull request?
Did you update the CHANGELOG? (not for typos, docs, test updates, or minor internal changes/refactors)

PR review

Anyone in the community is welcome to review the PR.
Before you start reviewing, make sure you have read the review guidelines. In short, see the following bullet-list:

Reviewer checklist

Is this pull request ready for review? (if not, please submit in draft mode)
Check that all items from Before submitting are resolved
Make sure the title is self-explanatory and the description concisely explains the PR
Add labels and milestones (and optionally projects) to the PR so it can be classified

cc @Borda

…o mount

github-actions · 2023-02-13T08:56:54Z

⚡ Required checks status: All passing 🟢

Groups summary

🟢 lightning_app: Tests workflow

Check ID	Status
app-pytest (macOS-11, lightning, 3.8, latest)	success	✅
app-pytest (macOS-11, lightning, 3.8, oldest)	success	✅
app-pytest (macOS-11, app, 3.9, latest)	success	✅
app-pytest (ubuntu-20.04, lightning, 3.8, latest)	success	✅
app-pytest (ubuntu-20.04, lightning, 3.8, oldest)	success	✅
app-pytest (ubuntu-20.04, app, 3.9, latest)	success	✅
app-pytest (windows-2022, lightning, 3.8, latest)	success	✅
app-pytest (windows-2022, lightning, 3.8, oldest)	success	✅
app-pytest (windows-2022, app, 3.8, latest)	success	✅

These checks are required after the changes to src/lightning/app/cli/commands/ls.py, src/lightning/app/cli/connect/data.py, tests/tests_app/cli/test_connect_data.py, requirements/app/base.txt.

🟢 lightning_app: Examples

Check ID	Status
app-examples (macOS-11, lightning, 3.9, latest)	success	✅
app-examples (macOS-11, lightning, 3.9, oldest)	success	✅
app-examples (macOS-11, app, 3.9, latest)	success	✅
app-examples (ubuntu-20.04, lightning, 3.9, latest)	success	✅
app-examples (ubuntu-20.04, lightning, 3.9, oldest)	success	✅
app-examples (ubuntu-20.04, app, 3.9, latest)	success	✅
app-examples (windows-2022, lightning, 3.9, latest)	success	✅
app-examples (windows-2022, lightning, 3.9, oldest)	success	✅
app-examples (windows-2022, app, 3.9, latest)	success	✅

These checks are required after the changes to src/lightning/app/cli/commands/ls.py, src/lightning/app/cli/connect/data.py, requirements/app/base.txt.

🟢 lightning_app: Azure

Check ID	Status
App.cloud-e2e	success	✅

These checks are required after the changes to src/lightning/app/cli/commands/ls.py, src/lightning/app/cli/connect/data.py, requirements/app/base.txt.

🟢 lightning_app: Docs

Check ID	Status
make-doctest (lightning)	success	✅
make-html (lightning)	success	✅

These checks are required after the changes to src/lightning/app/cli/commands/ls.py, src/lightning/app/cli/connect/data.py, requirements/app/base.txt.

🟢 mypy

Check ID	Status
mypy	success	✅

These checks are required after the changes to requirements/app/base.txt, src/lightning/app/cli/commands/ls.py, src/lightning/app/cli/connect/data.py.

🟢 install

Check ID	Status
install-pkg (ubuntu-22.04, app, 3.8)	success	✅
install-pkg (ubuntu-22.04, app, 3.10)	success	✅
install-pkg (ubuntu-22.04, fabric, 3.8)	success	✅
install-pkg (ubuntu-22.04, fabric, 3.10)	success	✅
install-pkg (ubuntu-22.04, pytorch, 3.8)	success	✅
install-pkg (ubuntu-22.04, pytorch, 3.10)	success	✅
install-pkg (ubuntu-22.04, lightning, 3.8)	success	✅
install-pkg (ubuntu-22.04, lightning, 3.10)	success	✅
install-pkg (ubuntu-22.04, notset, 3.8)	success	✅
install-pkg (ubuntu-22.04, notset, 3.10)	success	✅
install-pkg (macOS-12, app, 3.8)	success	✅
install-pkg (macOS-12, app, 3.10)	success	✅
install-pkg (macOS-12, fabric, 3.8)	success	✅
install-pkg (macOS-12, fabric, 3.10)	success	✅
install-pkg (macOS-12, pytorch, 3.8)	success	✅
install-pkg (macOS-12, pytorch, 3.10)	success	✅
install-pkg (macOS-12, lightning, 3.8)	success	✅
install-pkg (macOS-12, lightning, 3.10)	success	✅
install-pkg (macOS-12, notset, 3.8)	success	✅
install-pkg (macOS-12, notset, 3.10)	success	✅
install-pkg (windows-2022, app, 3.8)	success	✅
install-pkg (windows-2022, app, 3.10)	success	✅
install-pkg (windows-2022, fabric, 3.8)	success	✅
install-pkg (windows-2022, fabric, 3.10)	success	✅
install-pkg (windows-2022, pytorch, 3.8)	success	✅
install-pkg (windows-2022, pytorch, 3.10)	success	✅
install-pkg (windows-2022, lightning, 3.8)	success	✅
install-pkg (windows-2022, lightning, 3.10)	success	✅
install-pkg (windows-2022, notset, 3.8)	success	✅
install-pkg (windows-2022, notset, 3.10)	success	✅

These checks are required after the changes to src/lightning/app/cli/commands/ls.py, src/lightning/app/cli/connect/data.py, requirements/app/base.txt.

🟢 link-check

Check ID	Status
markdown-link-check	success	✅

These checks are required after the changes to src/lightning/app/CHANGELOG.md.

Thank you for your contribution! 💜

Note
This comment is automatically generated and updates for 60 minutes every 180 seconds. If you have any other questions, contact carmocca for help.

src/lightning/app/cli/connect/data.py

Co-authored-by: thomas <thomas@thomass-MacBook-Pro.local>

* Add .git-blame-ignore-revs (#16709) Co-authored-by: Jirka Borovec <6035284+Borda@users.noreply.github.com> * Fix strategy type validation in connectors (#16693) * Disable strict loading in multiprocessing launcher (#16365) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Jirka <jirka.borovec@seznam.cz> * Fix min-epochs and early-stopping triggering too many validation runs (#16719) Co-authored-by: Jirka Borovec <6035284+Borda@users.noreply.github.com> * Update hydra-core requirement from <1.3.0,>=1.0.5 to >=1.0.5,<1.4.0 in /requirements (#16736) Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * [App] Add support for private data (#16738) Co-authored-by: thomas <thomas@thomass-MacBook-Pro.local> * [App] Add rm one level below project level (#16740) Co-authored-by: Ethan Harris <ethanwharris@gmail.com> Co-authored-by: Justus Schock <12886177+justusschock@users.noreply.github.com> Co-authored-by: thomas <thomas@thomass-MacBook-Pro.local> * ci: cleaning caches (#16752) * CI: Update colossalai version (#16747) Co-authored-by: Carlos Mocholí <carlossmocholi@gmail.com> type * Update version and changelog for 1.9.2 --------- Co-authored-by: Akihiro Nitta <nitta@akihironitta.com> Co-authored-by: Jirka Borovec <6035284+Borda@users.noreply.github.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Jirka <jirka.borovec@seznam.cz> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: thomas chaton <thomas@grid.ai> Co-authored-by: thomas <thomas@thomass-MacBook-Pro.local> Co-authored-by: Ethan Harris <ethanwharris@gmail.com> Co-authored-by: Justus Schock <12886177+justusschock@users.noreply.github.com>

thomas and others added 28 commits February 6, 2023 12:43

update

23825b4

update

1478c4f

update

41919c5

update

4488c4b

update

1b41080

update

4ed3a48

update

98e3734

Merge branch 'master' into mount

1d423f9

update

9447dd4

Merge branch 'mount' of https://github.com/Lightning-AI/lightning int…

646fe65

…o mount

update

9aeaaff

update

a939bd2

update

50d7f21

update

87e76ea

update

b30c522

Merge branch 'master' into mount

3f39949

update

ad7046c

Merge branch 'mount' of https://github.com/Lightning-AI/lightning int…

6d983e0

…o mount

update

0bf0f8b

update

39ae28e

update

9086167

Merge branch 'master' into mount

033772b

update

3ebacf1

update

e480767

update

6ac9476

update

bcdef0f

update

4a19ed9

update

1130e91

tchaton requested review from lantiga and awaelchli as code owners February 13, 2023 08:56

tchaton requested review from hhsecond and ethanwharris as code owners February 13, 2023 08:56

github-actions bot added the app Generic label for Lightning App package label Feb 13, 2023

update

06708b7

tchaton requested a review from Borda as a code owner February 13, 2023 08:57

thomas added 3 commits February 13, 2023 09:00

update

4920c7f

update

365e309

update

2c7eaa8

mergify bot added the has conflicts label Feb 13, 2023

update

e88dafc

tchaton self-assigned this Feb 13, 2023

Merge branch 'master' into private_s3

53b7f79

mergify bot removed the has conflicts label Feb 13, 2023

Merge branch 'master' into private_s3

d433540

ethanwharris approved these changes Feb 14, 2023

View reviewed changes

Merge branch 'master' into private_s3

49afdc8

justusschock approved these changes Feb 14, 2023

View reviewed changes

tchaton enabled auto-merge (squash) February 14, 2023 11:28

mergify bot added the ready PRs ready to be merged label Feb 14, 2023

Borda approved these changes Feb 14, 2023

View reviewed changes

src/lightning/app/cli/connect/data.py Show resolved Hide resolved

Merge branch 'master' into private_s3

42f73aa

tchaton merged commit 7e8400d into master Feb 14, 2023

tchaton deleted the private_s3 branch February 14, 2023 12:34

tchaton added a commit that referenced this pull request Feb 14, 2023

[App] Add support for private data (#16738)

bbbb89a

Co-authored-by: thomas <thomas@thomass-MacBook-Pro.local>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[App] Add support for private data #16738

[App] Add support for private data #16738

tchaton commented Feb 13, 2023 •

edited by github-actions bot

github-actions bot commented Feb 13, 2023 •

edited

[App] Add support for private data #16738

[App] Add support for private data #16738

Conversation

tchaton commented Feb 13, 2023 • edited by github-actions bot

What does this PR do?

PR review

github-actions bot commented Feb 13, 2023 • edited

⚡ Required checks status: All passing 🟢

Groups summary

tchaton commented Feb 13, 2023 •

edited by github-actions bot

github-actions bot commented Feb 13, 2023 •

edited