Support Injecting Secrets into Apps Running in the Cloud (#14612)

Adds a new '--secret' flag to 'lightning run app':

lightning run app --cloud --secret MY_SECRET=my-secret-name app.py

When the Lightning App runs in the cloud, the 'MY_SECRET'
environment variable will be populated with the value of the
referenced Secret. The value of the Secret is encrypted in the
database, and will only be decrypted and accessible to the
Flow/Work processes in the cloud.

Co-authored-by: Sherin Thomas <sherin@grid.ai>
Co-authored-by: Noha Alon <nohalon@gmail.com>
Co-authored-by: thomas chaton <thomas@grid.ai>

(cherry picked from commit 71719b9)

docs/source-app/glossary/environment_variables.rst

@@ -4,17 +4,17 @@
 Environment Variables
 *********************
 
-If your app is using secrets or values you don't want to expose in your app code such as API keys or access tokens, you can use environment variables.
+If your App is using configuration values you don't want to commit with your App source code, you can use environment variables.
 
-Lightning allows you to set environment variables when running the app from the CLI with the `lightning run app` command. You can use environment variables to pass any value such as API keys or other similar configurations to the app, avoiding having to stick them in the source code.
+Lightning allows you to set environment variables when running the App from the CLI with the `lightning run app` command. You can use environment variables to pass any values to the App, and avoiding sticking those values in the source code.
 
 Set one or multiple variables using the **--env** option:
 
 .. code:: bash
 
     lightning run app app.py --cloud --env FOO=BAR --env BAZ=FAZ
 
-The environment variables are available in all flows and works, and can be accessed as follows:
+Environment variables are available in all Flows and Works, and can be accessed as follows:
 
 .. code:: python
 
@@ -24,4 +24,4 @@ The environment variables are available in all flows and works, and can be acces
     print(os.environ["BAZ"])  # FAZ
 
 .. note::
-	Environment variables are currently not encrypted.
+	Environment variables are not encrypted. For sensitive values, we recommend using :ref:`Encrypted Secrets <secrets>`.

docs/source-app/glossary/secrets.rst

@@ -0,0 +1,57 @@
+.. _secrets:
+
+#################
+Encrypted Secrets
+#################
+
+We understand that many Apps require access to private data like API keys, access tokens, database passwords, or other credentials. And that you need to protect this data.
+
+Secrets provie a secure way to make private data like API keys or passwords accessible to your app, without hardcoding. You can use secrets to authenticate third-party services/solutions.
+
+.. tip::
+	For non-sensitive configuration values, we recommend using :ref:`plain-text Environment Variables <environment_variables>`.
+
+*******************
+Overview of Secrets
+*******************
+
+The ``--secret`` option has been added to the **lightning run app** command. ``--secret`` can be used by itself or alongside ``--env``.
+
+When a Lightning App (App) **runs in the cloud**, the Secret can be exposed to the App using environment variables.
+The value of the Secret is encrypted in the Lightning.ai database, and is only decrypted and accessible to
+LightningFlow (Flow) or LightningWork (Work) processes in the cloud (when you use the ``--cloud`` option running your App).
+
+----
+
+*********************
+Use Encrypted Secrets
+*********************
+
+First, a Secret must be created using the admin web UI. Once you create a Secret, you can bind it to any of your Apps. You do not need to create a new Secret for each App if the Secret value is the same.
+
+.. note::
+    Secret names must start with a letter and can only contain letters, numbers, dashes, and periods. The Secret names must comply with `RFC1123 naming conventions <https://www.rfc-editor.org/rfc/rfc1123>`_. The Secret value has no restrictions.
+
+In the example below, we already used the admin UI to create a Secret named ``my-secret`` with the value ``some-value``` and will bind it to the environment variable ``MY_APP_SECRET`` within our App. The binding is accomplished by using the ``--secret`` option when running the App from the Lightning CLI.
+
+The ``--secret``` option works similar to ``--env``, but instead of providing a value, you provide the name of the Secret which will be replaced with with the value that you want to bind to the environment variable:
+
+.. code:: bash
+
+    lightning run app app.py --cloud --secret MY_APP_SECRET=my-secret
+
+The environment variables are available in all Flows and Works, and can be accessed as follows:
+
+.. code:: python
+
+    import os
+
+    print(os.environ["MY_APP_SECRET"])
+
+The code above will print out ``some-value``.
+
+The ``--secret`` option can be used for multiple Secrets, and alongside the ``--env`` option:
+
+.. code:: bash
+
+    lightning run app app.py --cloud --env FOO=bar --secret MY_APP_SECRET=my-secret --secret ANOTHER_SECRET=another-secret

docs/source-app/index.rst

@@ -270,6 +270,7 @@ Keep Learning
    DAG <glossary/dag>
    Event Loop <glossary/event_loop>
    Environment Variables <glossary/environment_variables>
+   Encrypted Secrets <glossary/secrets>
    Frontend <workflows/add_web_ui/glossary_front_end.rst>
    Apple and Android mobile devices with Lighting Apps <glossary/ios_and_android>
    REST API <glossary/restful_api/restful_api>

docs/source-lit/glossary/secrets.rst

@@ -0,0 +1 @@
+../../source-app/glossary/secrets.rst

docs/source-lit/index.rst

@@ -113,6 +113,7 @@ Welcome to ⚡ Lightning Apps
    DAG <glossary/dag>
    Event Loop <glossary/event_loop>
    Environment Variables <glossary/environment_variables>
+   Encrypted Secrets <glossary/secrets>
    Frontend <workflows/add_web_ui/glossary_front_end.rst>
    Sharing Components <glossary/sharing_components>
    Scheduling <glossary/scheduling.rst>

docs/source-lit/workflows/build_command_line_interface/commands/notebook/run.py

@@ -0,0 +1 @@
+../../../../../source-app/workflows/build_command_line_interface/commands/notebook/run.py

src/lightning_app/CHANGELOG.md

@@ -4,6 +4,13 @@ All notable changes to this project will be documented in this file.
 
 The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/).
 
+## [UnReleased] - 2022-MM-DD
+
+### Added
+
+- Add `--secret` option to CLI to allow binding Secrets to app environment variables when running in the cloud ([#14612](https://github.com/Lightning-AI/lightning/pull/14612))
+
+
 ## [0.6.2] - 2022-09-21
 
 ### Changed

src/lightning_app/cli/lightning_cli.py

@@ -304,6 +304,7 @@ def _run_app(
     blocking: bool,
     open_ui: bool,
     env: tuple,
+    secret: tuple,
 ) -> None:
     file = _prepare_file(file)
 
@@ -319,11 +320,18 @@ def _run_app(
                 "Caching is a property of apps running in cloud. "
                 "Using the flag --no-cache in local execution is not supported."
             )
+        if secret:
+            raise click.ClickException(
+                "Secrets can only be used for apps running in cloud. "
+                "Using the option --secret in local execution is not supported."
+            )
 
     env_vars = _format_input_env_variables(env)
     os.environ.update(env_vars)
 
-    def on_before_run(*args: Any) -> None:
+    secrets = _format_input_env_variables(secret)
+
+    def on_before_run(*args, **kwargs) -> None:
         if open_ui and not without_server:
             click.launch(get_app_url(runtime_type, *args))
 
@@ -341,6 +349,7 @@ def on_before_run(*args: Any) -> None:
         on_before_run=on_before_run,
         name=name,
         env_vars=env_vars,
+        secrets=secrets,
         cluster_id=cluster_id,
     )
     if runtime_type == RuntimeType.CLOUD:
@@ -364,8 +373,14 @@ def run() -> None:
     "--no-cache", is_flag=True, default=False, help="Disable caching of packages " "installed from requirements.txt"
 )
 @click.option("--blocking", "blocking", type=bool, default=False)
-@click.option("--open-ui", type=bool, default=True, help="Decide whether to launch the app UI in a web browser")
-@click.option("--env", type=str, default=[], multiple=True, help="Env variables to be set for the app.")
+@click.option(
+    "--open-ui",
+    type=bool,
+    default=True,
+    help="Decide whether to launch the app UI in a web browser"
+)
+@click.option("--env", type=str, default=[], multiple=True, help="Environment variables to be set for the app.")
+@click.option("--secret", type=str, default=[], multiple=True, help="Secret variables to be set for the app.")
 @click.option("--app_args", type=str, default=[], multiple=True, help="Collection of arguments for the app.")
 def run_app(
     file: str,
@@ -377,10 +392,11 @@ def run_app(
     blocking: bool,
     open_ui: bool,
     env: tuple,
+    secret: tuple,
     app_args: tuple,
 ) -> None:
     """Run an app from a file."""
-    _run_app(file, cloud, cluster_id, without_server, no_cache, name, blocking, open_ui, env)
+    _run_app(file, cloud, cluster_id, without_server, no_cache, name, blocking, open_ui, env, secret)
 
 
 @_main.group(hidden=True)

src/lightning_app/runners/cloud.py

@@ -48,6 +48,7 @@
 from lightning_app.utilities.dependency_caching import get_hash
 from lightning_app.utilities.packaging.app_config import AppConfig, find_config_file
 from lightning_app.utilities.packaging.lightning_utils import _prepare_lightning_wheels_and_requirements
+from lightning_app.utilities.secrets import _names_to_ids
 
 logger = Logger(__name__)
 
@@ -96,8 +97,16 @@ def dispatch(
 
         print(f"The name of the app is: {app_config.name}")
 
-        work_reqs: List[V1Work] = []
         v1_env_vars = [V1EnvVar(name=k, value=v) for k, v in self.env_vars.items()]
+
+        if len(self.secrets.values()) > 0:
+            secret_names_to_ids = _names_to_ids(self.secrets.values())
+            env_vars_from_secrets = [
+                V1EnvVar(name=k, from_secret=secret_names_to_ids[v]) for k, v in self.secrets.items()
+            ]
+            v1_env_vars.extend(env_vars_from_secrets)
+
+        work_reqs: List[V1Work] = []
         for flow in self.app.flows:
             for work in flow.works(recurse=False):
                 work_requirements = "\n".join(work.cloud_build_config.requirements)

src/lightning_app/runners/runtime.py

@@ -27,7 +27,8 @@ def dispatch(
     blocking: bool = True,
     on_before_run: Optional[Callable] = None,
     name: str = "",
-    env_vars: Dict[str, str] = {},
+    env_vars: Dict[str, str] = None,
+    secrets: Dict[str, str] = None,
     cluster_id: str = None,
 ) -> Optional[Any]:
     """Bootstrap and dispatch the application to the target.
@@ -43,6 +44,7 @@ def dispatch(
         on_before_run: Callable to be executed before run.
         name: Name of app execution
         env_vars: Dict of env variables to be set on the app
+        secrets: Dict of secrets to be passed as environment variables to the app
         cluster_id: the Lightning AI cluster to run the app on. Defaults to managed Lightning AI cloud
     """
     from lightning_app.runners.runtime_type import RuntimeType
@@ -54,11 +56,20 @@ def dispatch(
     runtime_cls: Type[Runtime] = runtime_type.get_runtime()
     app = load_app_from_file(str(entrypoint_file))
 
+    env_vars = {} if env_vars is None else env_vars
+    secrets = {} if secrets is None else secrets
+
     if blocking:
         app.stage = AppStage.BLOCKING
 
     runtime = runtime_cls(
-        app=app, entrypoint_file=entrypoint_file, start_server=start_server, host=host, port=port, env_vars=env_vars
+        app=app,
+        entrypoint_file=entrypoint_file,
+        start_server=start_server,
+        host=host,
+        port=port,
+        env_vars=env_vars,
+        secrets=secrets,
     )
     # a cloud dispatcher will return the result while local
     # dispatchers will be running the app in the main process
@@ -78,6 +89,7 @@ class Runtime:
     done: bool = False
     backend: Optional[Union[str, Backend]] = "multiprocessing"
     env_vars: Dict[str, str] = field(default_factory=dict)
+    secrets: Dict[str, str] = field(default_factory=dict)
 
     def __post_init__(self):
         if isinstance(self.backend, str):

src/lightning_app/utilities/secrets.py

@@ -0,0 +1,26 @@
+from typing import Dict, List
+
+from lightning_app.utilities.cloud import _get_project
+from lightning_app.utilities.network import LightningClient
+
+
+def _names_to_ids(secret_names: List[str]) -> Dict[str, str]:
+    """Returns the name/ID pair for each given Secret name.
+
+    Raises a `ValueError` if any of the given Secret names do not exist.
+    """
+    lightning_client = LightningClient()
+
+    project = _get_project(lightning_client)
+    secrets = lightning_client.secret_service_list_secrets(project_id=project.project_id)
+
+    secret_names_to_ids: Dict[str, str] = {}
+    for secret in secrets.secrets:
+        if secret.name in secret_names:
+            secret_names_to_ids[secret.name] = secret.id
+
+    for secret_name in secret_names:
+        if secret_name not in secret_names_to_ids.keys():
+            raise ValueError(f"Secret with name '{secret_name}' not found")
+
+    return secret_names_to_ids

tests/tests_app/cli/test_run_app.py

@@ -33,7 +33,6 @@ def _lightning_app_run_and_logging(self, *args, **kwargs):
 
     with caplog.at_level(logging.INFO):
         with mock.patch("lightning_app.LightningApp._run", _lightning_app_run_and_logging):
-
             runner = CliRunner()
             result = runner.invoke(
                 run_app,
@@ -70,6 +69,7 @@ def test_lightning_run_cluster_without_cloud(monkeypatch):
             open_ui=False,
             no_cache=True,
             env=("FOO=bar",),
+            secret=(),
         )
 
 
@@ -80,7 +80,7 @@ def test_lightning_run_app_cloud(mock_dispatch: mock.MagicMock, open_ui, caplog,
     """This test validates the command has ran properly when --cloud argument is passed.
 
     It tests it by checking if the click.launch is called with the right url if --open-ui was true and also checks the
-    call to `dispatch` for the right arguments
+    call to `dispatch` for the right arguments.
     """
     monkeypatch.setattr("lightning_app.runners.cloud.logger", logging.getLogger())
 
@@ -95,6 +95,7 @@ def test_lightning_run_app_cloud(mock_dispatch: mock.MagicMock, open_ui, caplog,
             open_ui=open_ui,
             no_cache=True,
             env=("FOO=bar",),
+            secret=("BAR=my-secret",),
         )
     # capture logs.
     # TODO(yurij): refactor the test, check if the actual HTTP request is being sent and that the proper admin
@@ -108,5 +109,25 @@ def test_lightning_run_app_cloud(mock_dispatch: mock.MagicMock, open_ui, caplog,
         name="",
         no_cache=True,
         env_vars={"FOO": "bar"},
+        secrets={"BAR": "my-secret"},
         cluster_id="",
     )
+
+
+def test_lightning_run_app_secrets(monkeypatch):
+    """Validates that running apps only supports the `--secrets` argument if the `--cloud` argument is passed."""
+    monkeypatch.setattr("lightning_app.runners.cloud.logger", logging.getLogger())
+
+    with pytest.raises(click.exceptions.ClickException):
+        _run_app(
+            file=os.path.join(_PROJECT_ROOT, "tests/tests_app/core/scripts/app_metadata.py"),
+            cloud=False,
+            cluster_id="test-cluster",
+            without_server=False,
+            name="",
+            blocking=False,
+            open_ui=False,
+            no_cache=True,
+            env=(),
+            secret=("FOO=my-secret"),
+        )

tests/tests_app/utilities/test_secrets.py

@@ -0,0 +1,49 @@
+from typing import Dict, List
+from unittest import mock
+from unittest.mock import MagicMock
+
+import pytest
+from lightning_cloud.openapi import V1ListMembershipsResponse, V1ListSecretsResponse, V1Membership, V1Secret
+
+from lightning_app.utilities.secrets import _names_to_ids
+
+
+@pytest.mark.parametrize(
+    "secret_names, secrets, expected, expected_exception",
+    [
+        ([], [], {}, False),
+        (
+            ["first-secret", "second-secret"],
+            [
+                V1Secret(name="first-secret", id="1234"),
+            ],
+            {},
+            True,
+        ),
+        (
+            ["first-secret", "second-secret"],
+            [V1Secret(name="first-secret", id="1234"), V1Secret(name="second-secret", id="5678")],
+            {"first-secret": "1234", "second-secret": "5678"},
+            False,
+        ),
+    ],
+)
+@mock.patch("lightning_cloud.login.Auth.authenticate", MagicMock())
+@mock.patch("lightning_app.utilities.network.LightningClient.secret_service_list_secrets")
+@mock.patch("lightning_app.utilities.network.LightningClient.projects_service_list_memberships")
+def test_names_to_ids(
+    list_memberships: MagicMock,
+    list_secrets: MagicMock,
+    secret_names: List[str],
+    secrets: List[V1Secret],
+    expected: Dict[str, str],
+    expected_exception: bool,
+):
+    list_memberships.return_value = V1ListMembershipsResponse(memberships=[V1Membership(project_id="default-project")])
+    list_secrets.return_value = V1ListSecretsResponse(secrets=secrets)
+
+    if expected_exception:
+        with pytest.raises(ValueError):
+            _names_to_ids(secret_names)
+    else:
+        assert _names_to_ids(secret_names) == expected