Support Injecting Secrets into Apps Running in the Cloud #14612

alecmerdler · 2022-09-08T15:06:38Z

What does this PR do?

Adds a new '--secret' flag to 'lightning run app':

lightning run app --cloud --secret MY_SECRET=my-secret-name app.py

When the Lightning App runs in the cloud, the 'MY_SECRET'
environment variable will be populated with the value of the
referenced Secret. The value of the Secret is encrypted in the
database, and will only be decrypted and accessible to the
Flow/Work processes in the cloud.

Before submitting

Was this discussed/approved via a GitHub issue? (not for typos and docs)
Did you read the contributor guideline, Pull Request section?
Did you make sure your PR does only one thing, instead of bundling different changes together?
Did you make sure to update the documentation with your changes? (if necessary)
Did you write any new necessary tests? (not for typos and docs)
Did you verify new and existing tests pass locally with your changes?
Did you list all the breaking changes introduced by this pull request?
Did you update the CHANGELOG? (not for typos, docs, test updates, or minor internal changes/refactors)

PR review

Anyone in the community is welcome to review the PR.
Before you start reviewing, make sure you have read the review guidelines. In short, see the following bullet-list:

Is this pull request ready for review? (if not, please submit in draft mode)
Check that all items from Before submitting are resolved
Make sure the title is self-explanatory and the description concisely explains the PR
Add labels and milestones (and optionally projects) to the PR so it can be classified

Did you have fun?

Make sure you had fun coding 🙃

awaelchli

It's finally coming <3

src/lightning_app/runners/cloud.py

src/lightning_app/utilities/secrets.py

src/lightning_app/cli/lightning_cli.py

src/lightning_app/runners/cloud.py

edenlightning · 2022-09-08T20:35:44Z

This needs documentation @Felonious-Spellfire

src/lightning_app/cli/lightning_cli.py

src/lightning_app/runners/runtime.py

tests/tests_app/cli/test_run_app.py

docs/source-app/glossary/environment_variables.rst

src/lightning_app/CHANGELOG.md

alecmerdler · 2022-09-14T17:52:32Z

@hhsecond The App.cloud-e2e tests are failing again. Could you help out when you get a chance?

docs/source-app/glossary/secrets.rst

Felonious-Spellfire

LGTM

Adds a new '--secret' flag to 'lightning run app': lightning run app --cloud --secret MY_SECRET=my-secret-name app.py When the Lightning App runs in the cloud, the 'MY_SECRET' environment variable will be populated with the value of the referenced Secret. The value of the Secret is encrypted in the database, and will only be decrypted and accessible to the Flow/Work processes in the cloud.

edenlightning

@Felonious-Spellfire

edenlightning · 2022-09-15T19:13:55Z

docs/source-app/glossary/secrets.rst

+Encrypted Secrets
+#################
+
+We understand that many Apps require access to private data like API keys, access tokens, database passwords, or other credentials. And that you need to protect this data.


I would remove "We understand", just say "Many apps require access..."

@Felonious-Spellfire

Please review my PR. The structure was completely changed

edenlightning · 2022-09-15T19:14:03Z

docs/source-app/glossary/secrets.rst

+
+We understand that many Apps require access to private data like API keys, access tokens, database passwords, or other credentials. And that you need to protect this data.
+
+Secrets provie a secure way to make private data like API keys or passwords accessible to your app, without hardcoding. You can use secrets to authenticate third-party services/solutions.


@Felonious-Spellfire

Please review my PR. The structure was completely changed and this typo came across when it was.

edenlightning · 2022-09-15T19:14:22Z

docs/source-app/glossary/secrets.rst

+*******************
+Overview of Secrets
+*******************
+
+The ``--secret`` option has been added to the **lightning run app** command. ``--secret`` can be used by itself or alongside ``--env``.
+
+When a Lightning App (App) **runs in the cloud**, the Secret can be exposed to the App using environment variables.
+The value of the Secret is encrypted in the Lightning.ai database, and is only decrypted and accessible to
+LightningFlow (Flow) or LightningWork (Work) processes in the cloud (when you use the ``--cloud`` option running your App).
+


@Felonious-Spellfire

Please review my PR. The structure was completely changed

edenlightning · 2022-09-15T19:14:54Z

docs/source-app/glossary/secrets.rst

+----
+
+*********************
+Use Encrypted Secrets


If this is specific to lightning cloud, we should say "Add secrets to Lightning Cloud"

@Felonious-Spellfire

I'm gonna disagree on this. It's called out that this is only available in the cloud in several places. What we want to point out is how they should be using the feature correctly.

edenlightning · 2022-09-15T19:15:51Z

docs/source-app/glossary/secrets.rst

+Use Encrypted Secrets
+*********************
+
+First, a Secret must be created using the admin web UI. Once you create a Secret, you can bind it to any of your Apps. You do not need to create a new Secret for each App if the Secret value is the same.


Step 1- Log into lighning.ai to add your secrets.
Avatar > Profile > secrets > add a secret

Add a screenshot/gif of how to do that

@Felonious-Spellfire

edenlightning · 2022-09-15T19:16:07Z

docs/source-app/glossary/secrets.rst

+
+First, a Secret must be created using the admin web UI. Once you create a Secret, you can bind it to any of your Apps. You do not need to create a new Secret for each App if the Secret value is the same.
+
+.. note::


Step 2- Add a secret

@Felonious-Spellfire

edenlightning · 2022-09-15T19:16:25Z

docs/source-app/glossary/secrets.rst

+
+In the example below, we already used the admin UI to create a Secret named ``my-secret`` with the value ``some-value``` and will bind it to the environment variable ``MY_APP_SECRET`` within our App. The binding is accomplished by using the ``--secret`` option when running the App from the Lightning CLI.
+
+The ``--secret``` option works similar to ``--env``, but instead of providing a value, you provide the name of the Secret which will be replaced with with the value that you want to bind to the environment variable:


Step 3- Add env variable to your app

@Felonious-Spellfire

edenlightning · 2022-09-15T19:16:47Z

docs/source-app/glossary/secrets.rst

+
+The ``--secret``` option works similar to ``--env``, but instead of providing a value, you provide the name of the Secret which will be replaced with with the value that you want to bind to the environment variable:
+
+.. code:: bash


Step 4- add the secret to the lighnging app

@Felonious-Spellfire

Felonious-Spellfire · 2022-09-21T15:42:51Z

also @alecmerdler @Felonious-Spellfire I didn't see anything in the doc about the secret name formatting constraints i.e. that you can't use underscores in the secret name

@nohalon
It's here: #14734
My changes kept getting overwritten so I made my own PR so that wouldn't happen.

Adds a new '--secret' flag to 'lightning run app': lightning run app --cloud --secret MY_SECRET=my-secret-name app.py When the Lightning App runs in the cloud, the 'MY_SECRET' environment variable will be populated with the value of the referenced Secret. The value of the Secret is encrypted in the database, and will only be decrypted and accessible to the Flow/Work processes in the cloud. Co-authored-by: Sherin Thomas <sherin@grid.ai> Co-authored-by: Noha Alon <nohalon@gmail.com> Co-authored-by: thomas chaton <thomas@grid.ai> (cherry picked from commit 71719b9)

* Support Injecting Secrets into Apps Running in the Cloud (#14612) Adds a new '--secret' flag to 'lightning run app': lightning run app --cloud --secret MY_SECRET=my-secret-name app.py When the Lightning App runs in the cloud, the 'MY_SECRET' environment variable will be populated with the value of the referenced Secret. The value of the Secret is encrypted in the database, and will only be decrypted and accessible to the Flow/Work processes in the cloud. Co-authored-by: Sherin Thomas <sherin@grid.ai> Co-authored-by: Noha Alon <nohalon@gmail.com> Co-authored-by: thomas chaton <thomas@grid.ai> (cherry picked from commit 71719b9) * secrets docs (#14951) * secrets docs * Update docs/source-app/glossary/secrets.rst Co-authored-by: Yurij Mikhalevich <yurij@grid.ai> * Apply suggestions from code review Co-authored-by: Adrian Wälchli <aedu.waelchli@gmail.com> * Update secrets.rst * links Co-authored-by: Yurij Mikhalevich <yurij@grid.ai> Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com> Co-authored-by: Adrian Wälchli <aedu.waelchli@gmail.com> Co-authored-by: Jirka <jirka.borovec@seznam.cz> (cherry picked from commit 8715cd0) # Conflicts: # docs/source-app/glossary/secrets.rst * Add support for command descriptions (#15193) (cherry picked from commit 4acb10f) * docs: temp drop S3 from index (#15099) Co-authored-by: awaelchli <aedu.waelchli@gmail.com> (cherry picked from commit 05d91c8) * version 0.7.0 * chlog join 0.6.3 & 0.7 Co-authored-by: Alec Merdler <alecmerdler@gmail.com> Co-authored-by: edenlightning <66261195+edenlightning@users.noreply.github.com> Co-authored-by: Ethan Harris <ethanwharris@gmail.com>

alecmerdler self-assigned this Sep 8, 2022

github-actions bot added the app Generic label for Lightning App package label Sep 8, 2022

alecmerdler mentioned this pull request Sep 8, 2022

Support Injecting Secrets into Apps Running in the Cloud #14592

Closed

12 tasks

alecmerdler force-pushed the LAI2-10393 branch from 3fcff82 to e51a6cc Compare September 8, 2022 15:07

Lightning-AI deleted a comment from mergify bot Sep 8, 2022

awaelchli approved these changes Sep 8, 2022

View reviewed changes

src/lightning_app/runners/cloud.py Outdated Show resolved Hide resolved

src/lightning_app/utilities/secrets.py Show resolved Hide resolved

src/lightning_app/cli/lightning_cli.py Outdated Show resolved Hide resolved

Borda added this to the app:0.7 milestone Sep 8, 2022

manskx reviewed Sep 8, 2022

View reviewed changes

src/lightning_app/cli/lightning_cli.py Show resolved Hide resolved

manskx reviewed Sep 8, 2022

View reviewed changes

src/lightning_app/runners/cloud.py Outdated Show resolved Hide resolved

manskx approved these changes Sep 8, 2022

View reviewed changes

alecmerdler marked this pull request as ready for review September 8, 2022 17:41

alecmerdler requested review from tchaton, hhsecond, williamFalcon, Borda, kaushikb11 and rohitgr7 as code owners September 8, 2022 17:41

mergify bot added the ready PRs ready to be merged label Sep 8, 2022

alecmerdler force-pushed the LAI2-10393 branch from f34bd5f to b2c75eb Compare September 8, 2022 19:20

alecmerdler force-pushed the LAI2-10393 branch from eafc08b to d1dcf86 Compare September 12, 2022 17:49

Felonious-Spellfire reviewed Sep 12, 2022

View reviewed changes

src/lightning_app/cli/lightning_cli.py Show resolved Hide resolved

Felonious-Spellfire reviewed Sep 12, 2022

View reviewed changes

src/lightning_app/runners/runtime.py Show resolved Hide resolved

Felonious-Spellfire reviewed Sep 12, 2022

View reviewed changes

tests/tests_app/cli/test_run_app.py Show resolved Hide resolved

alecmerdler force-pushed the LAI2-10393 branch from 61c91db to 5d286d0 Compare September 12, 2022 19:10

rohitgr7 approved these changes Sep 12, 2022

View reviewed changes

Felonious-Spellfire mentioned this pull request Sep 12, 2022

Added content for encrypted secrets #14669

Closed

12 tasks

alecmerdler force-pushed the LAI2-10393 branch 3 times, most recently from 8751291 to f9529e5 Compare September 12, 2022 21:07

alecmerdler force-pushed the LAI2-10393 branch 6 times, most recently from 4a6c0c2 to abfd373 Compare September 14, 2022 15:56

krshrimali approved these changes Sep 14, 2022

View reviewed changes

docs/source-app/glossary/environment_variables.rst Show resolved Hide resolved

alecmerdler force-pushed the LAI2-10393 branch from abfd373 to b3d0cab Compare September 14, 2022 16:59

Borda reviewed Sep 14, 2022

View reviewed changes

src/lightning_app/CHANGELOG.md Outdated Show resolved Hide resolved

alecmerdler force-pushed the LAI2-10393 branch 2 times, most recently from e1acedd to e807d20 Compare September 14, 2022 17:24

hhsecond approved these changes Sep 14, 2022

View reviewed changes

docs/source-app/glossary/secrets.rst Outdated Show resolved Hide resolved

mergify bot added the ready PRs ready to be merged label Sep 14, 2022

alecmerdler force-pushed the LAI2-10393 branch from e807d20 to d7829ce Compare September 14, 2022 19:41

Felonious-Spellfire approved these changes Sep 14, 2022

View reviewed changes

alecmerdler force-pushed the LAI2-10393 branch from d7829ce to 60f4013 Compare September 14, 2022 21:11

Sherin Thomas and others added 5 commits September 15, 2022 10:31

Merge branch 'master' into LAI2-10393

6674357

Merge branch 'master' into LAI2-10393

444718e

Merge branch 'master' into LAI2-10393

20c25a4

Merge branch 'master' into LAI2-10393

2b5626c

Merge branch 'master' into LAI2-10393

c80799b

lexierule merged commit 71719b9 into master Sep 15, 2022

lexierule deleted the LAI2-10393 branch September 15, 2022 13:16

edenlightning reviewed Sep 15, 2022

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support Injecting Secrets into Apps Running in the Cloud #14612

Support Injecting Secrets into Apps Running in the Cloud #14612

alecmerdler commented Sep 8, 2022 •

edited by Borda

awaelchli left a comment

edenlightning commented Sep 8, 2022

alecmerdler commented Sep 14, 2022

Felonious-Spellfire left a comment

edenlightning left a comment

edenlightning Sep 15, 2022

edenlightning Sep 15, 2022

Felonious-Spellfire Sep 15, 2022

edenlightning Sep 15, 2022

edenlightning Sep 15, 2022

Felonious-Spellfire Sep 15, 2022

edenlightning Sep 15, 2022

edenlightning Sep 15, 2022

Felonious-Spellfire Sep 15, 2022

edenlightning Sep 15, 2022

edenlightning Sep 15, 2022

Felonious-Spellfire Sep 15, 2022

edenlightning Sep 15, 2022

edenlightning Sep 15, 2022

edenlightning Sep 15, 2022

edenlightning Sep 15, 2022

edenlightning Sep 15, 2022

edenlightning Sep 15, 2022

edenlightning Sep 15, 2022

edenlightning Sep 15, 2022

Felonious-Spellfire commented Sep 21, 2022


		We understand that many Apps require access to private data like API keys, access tokens, database passwords, or other credentials. And that you need to protect this data.

		Secrets provie a secure way to make private data like API keys or passwords accessible to your app, without hardcoding. You can use secrets to authenticate third-party services/solutions.


		First, a Secret must be created using the admin web UI. Once you create a Secret, you can bind it to any of your Apps. You do not need to create a new Secret for each App if the Secret value is the same.

		.. note::


		In the example below, we already used the admin UI to create a Secret named ``my-secret`` with the value ``some-value``` and will bind it to the environment variable ``MY_APP_SECRET`` within our App. The binding is accomplished by using the ``--secret`` option when running the App from the Lightning CLI.

		The ``--secret``` option works similar to ``--env``, but instead of providing a value, you provide the name of the Secret which will be replaced with with the value that you want to bind to the environment variable:


		The ``--secret``` option works similar to ``--env``, but instead of providing a value, you provide the name of the Secret which will be replaced with with the value that you want to bind to the environment variable:

		.. code:: bash

Support Injecting Secrets into Apps Running in the Cloud #14612

Support Injecting Secrets into Apps Running in the Cloud #14612

Conversation

alecmerdler commented Sep 8, 2022 • edited by Borda

What does this PR do?

Before submitting

PR review

Did you have fun?

awaelchli left a comment

Choose a reason for hiding this comment

edenlightning commented Sep 8, 2022

alecmerdler commented Sep 14, 2022

Felonious-Spellfire left a comment

Choose a reason for hiding this comment

edenlightning left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Felonious-Spellfire commented Sep 21, 2022

alecmerdler commented Sep 8, 2022 •

edited by Borda