Support Injecting Secrets into Apps Running in the Cloud

docs/source-app/glossary/environment_variables.rst

@@ -4,17 +4,17 @@
 Environment Variables
 *********************
 
-If your app is using secrets or values you don't want to expose in your app code such as API keys or access tokens, you can use environment variables.
+If your App is using configuration values you don't want to commit with your App source code, you can use environment variables.
 
-Lightning allows you to set environment variables when running the app from the CLI with the `lightning run app` command. You can use environment variables to pass any value such as API keys or other similar configurations to the app, avoiding having to stick them in the source code.
+Lightning allows you to set environment variables when running the App from the CLI with the `lightning run app` command. You can use environment variables to pass any values to the App, and avoiding sticking those values in the source code.
 
 Set one or multiple variables using the **--env** option:
 
 .. code:: bash
 
     lightning run app app.py --cloud --env FOO=BAR --env BAZ=FAZ
 
-The environment variables are available in all flows and works, and can be accessed as follows:
+Environment variables are available in all Flows and Works, and can be accessed as follows:
 
 .. code:: python
 
@@ -24,4 +24,4 @@ The environment variables are available in all flows and works, and can be acces
     print(os.environ["BAZ"])  # FAZ
 
 .. note::
-	Environment variables are currently not encrypted.
+	Environment variables are not encrypted. For sensitive values, we recommend using :ref:`Encrypted Secrets <secrets>`.

docs/source-app/glossary/secrets.rst

@@ -0,0 +1,57 @@
+.. _secrets:
+
+#################
+Encrypted Secrets
+#################
+
+We understand that many Apps require access to private data like API keys, access tokens, database passwords, or other credentials. And that you need to protect this data.
+
+Secrets provie a secure way to make private data like API keys or passwords accessible to your app, without hardcoding. You can use secrets to authenticate third-party services/solutions.
+
+.. tip::
+	For non-sensitive configuration values, we recommend using :ref:`plain-text Environment Variables <environment_variables>`.
+
+*******************
+Overview of Secrets
+*******************
+
+The ``--secret`` option has been added to the **lightning run app** command. ``--secret`` can be used by itself or alongside ``--env``.
+
+When a Lightning App (App) **runs in the cloud**, the Secret can be exposed to the App using environment variables.
+The value of the Secret is encrypted in the Lightning.ai database, and is only decrypted and accessible to
+LightningFlow (Flow) or LightningWork (Work) processes in the cloud (when you use the ``--cloud`` option running your App).
+
+----
+
+*********************
+Use Encrypted Secrets
+*********************
+
+First, a Secret must be created using the admin web UI. Once you create a Secret, you can bind it to any of your Apps. You do not need to create a new Secret for each App if the Secret value is the same.
+
+.. note::
+    Secret names must start with a letter and can only contain letters, numbers, dashes, and periods. The Secret names must comply with `RFC1123 naming conventions <https://www.rfc-editor.org/rfc/rfc1123>`_. The Secret value has no restrictions.
+
+In the example below, we already used the admin UI to create a Secret named ``my-secret`` with the value ``some-value``` and will bind it to the environment variable ``MY_APP_SECRET`` within our App. The binding is accomplished by using the ``--secret`` option when running the App from the Lightning CLI.
+
+The ``--secret``` option works similar to ``--env``, but instead of providing a value, you provide the name of the Secret which will be replaced with with the value that you want to bind to the environment variable:
+
+.. code:: bash
+
+    lightning run app app.py --cloud --secret MY_APP_SECRET=my-secret
+
+The environment variables are available in all Flows and Works, and can be accessed as follows:
+
+.. code:: python
+
+    import os
+
+    print(os.environ["MY_APP_SECRET"])
+
+The code above will print out ``some-value``.
+
+The ``--secret`` option can be used for multiple Secrets, and alongside the ``--env`` option:
+
+.. code:: bash
+
+    lightning run app app.py --cloud --env FOO=bar --secret MY_APP_SECRET=my-secret --secret ANOTHER_SECRET=another-secret

docs/source-app/index.rst

@@ -270,6 +270,7 @@ Keep Learning
    DAG <glossary/dag>
    Event Loop <glossary/event_loop>
    Environment Variables <glossary/environment_variables>
+   Encrypted Secrets <glossary/secrets>
    Frontend <workflows/add_web_ui/glossary_front_end.rst>
    Apple and Android mobile devices with Lighting Apps <glossary/ios_and_android>
    REST API <glossary/restful_api/restful_api>

docs/source-lit/glossary/secrets.rst

@@ -0,0 +1 @@
+../../source-app/glossary/secrets.rst

docs/source-lit/index.rst

@@ -113,6 +113,7 @@ Welcome to ⚡ Lightning Apps
    DAG <glossary/dag>
    Event Loop <glossary/event_loop>
    Environment Variables <glossary/environment_variables>
+   Encrypted Secrets <glossary/secrets>
    Frontend <workflows/add_web_ui/glossary_front_end.rst>
    Sharing Components <glossary/sharing_components>
    Scheduling <glossary/scheduling.rst>

docs/source-lit/workflows/build_command_line_interface/commands/notebook/run.py

@@ -0,0 +1 @@
+../../../../../source-app/workflows/build_command_line_interface/commands/notebook/run.py

src/lightning_app/CHANGELOG.md

@@ -10,6 +10,7 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/).
 ### Added
 
 - Add `load_state_dict` and `state_dict` ([#14100](https://github.com/Lightning-AI/lightning/pull/14100))
+- Add `--secret` option to CLI to allow binding Secrets to app environment variables when running in the cloud ([#14612](https://github.com/Lightning-AI/lightning/pull/14612))
 
 
 ### Changed

src/lightning_app/cli/lightning_cli.py

@@ -305,6 +305,7 @@ def _run_app(
     blocking: bool,
     open_ui: bool,
     env: tuple,
+    secret: tuple,
 ):
     file = _prepare_file(file)
 
@@ -320,10 +321,17 @@ def _run_app(
                 "Caching is a property of apps running in cloud. "
                 "Using the flag --no-cache in local execution is not supported."
             )
+        if secret:
+            raise click.ClickException(
+                "Secrets can only be used for apps running in cloud. "
+                "Using the option --secret in local execution is not supported."
+            )
 
     env_vars = _format_input_env_variables(env)
     os.environ.update(env_vars)
 
+    secrets = _format_input_env_variables(secret)
+
     def on_before_run(*args, **kwargs):
         if open_ui and not without_server:
             click.launch(get_app_url(runtime_type, *args, **kwargs))
@@ -342,6 +350,7 @@ def on_before_run(*args, **kwargs):
         on_before_run=on_before_run,
         name=name,
         env_vars=env_vars,
+        secrets=secrets,
         cluster_id=cluster_id,
     )
     if runtime_type == RuntimeType.CLOUD:
@@ -377,7 +386,8 @@ def run():
     default=True,
     help="Decide whether to launch the app UI in a web browser",
 )
-@click.option("--env", type=str, default=[], multiple=True, help="Env variables to be set for the app.")
+@click.option("--env", type=str, default=[], multiple=True, help="Environment variables to be set for the app.")
+@click.option("--secret", type=str, default=[], multiple=True, help="Secret variables to be set for the app.")
 @click.option("--app_args", type=str, default=[], multiple=True, help="Collection of arguments for the app.")
 def run_app(
     file: str,
@@ -389,10 +399,11 @@ def run_app(
     blocking: bool,
     open_ui: bool,
     env: tuple,
-    app_args: List[str],
+    secret: tuple,
+    app_args: tuple,
 ):
     """Run an app from a file."""
-    _run_app(file, cloud, cluster_id, without_server, no_cache, name, blocking, open_ui, env)
+    _run_app(file, cloud, cluster_id, without_server, no_cache, name, blocking, open_ui, env, secret)
 
 
 @_main.group(hidden=True)

src/lightning_app/runners/cloud.py

@@ -50,6 +50,7 @@
 from lightning_app.utilities.dependency_caching import get_hash
 from lightning_app.utilities.packaging.app_config import AppConfig, find_config_file
 from lightning_app.utilities.packaging.lightning_utils import _prepare_lightning_wheels_and_requirements
+from lightning_app.utilities.secrets import _names_to_ids
 
 logger = Logger(__name__)
 
@@ -98,8 +99,16 @@ def dispatch(
 
         print(f"The name of the app is: {app_config.name}")
 
-        work_reqs: List[V1Work] = []
         v1_env_vars = [V1EnvVar(name=k, value=v) for k, v in self.env_vars.items()]
+
+        if len(self.secrets.values()) > 0:
+            secret_names_to_ids = _names_to_ids(self.secrets.values())
+            env_vars_from_secrets = [
+                V1EnvVar(name=k, from_secret=secret_names_to_ids[v]) for k, v in self.secrets.items()
+            ]
+            v1_env_vars.extend(env_vars_from_secrets)
+
+        work_reqs: List[V1Work] = []
         for flow in self.app.flows:
             for work in flow.works(recurse=False):
                 work_requirements = "\n".join(work.cloud_build_config.requirements)

src/lightning_app/runners/runtime.py

@@ -27,7 +27,8 @@ def dispatch(
     blocking: bool = True,
     on_before_run: Optional[Callable] = None,
     name: str = "",
-    env_vars: Dict[str, str] = {},
+    env_vars: Dict[str, str] = None,
+    secrets: Dict[str, str] = None,
     cluster_id: str = None,
 ) -> Optional[Any]:
     """Bootstrap and dispatch the application to the target.
@@ -43,6 +44,7 @@ def dispatch(
         on_before_run: Callable to be executed before run.
         name: Name of app execution
         env_vars: Dict of env variables to be set on the app
+        secrets: Dict of secrets to be passed as environment variables to the app
         cluster_id: the Lightning AI cluster to run the app on. Defaults to managed Lightning AI cloud
     """
     from lightning_app.runners.runtime_type import RuntimeType
@@ -54,11 +56,20 @@ def dispatch(
     runtime_cls: Type[Runtime] = runtime_type.get_runtime()
     app = load_app_from_file(str(entrypoint_file))
 
+    env_vars = {} if env_vars is None else env_vars
+    secrets = {} if secrets is None else secrets
+
     if blocking:
         app.stage = AppStage.BLOCKING
 
     runtime = runtime_cls(
-        app=app, entrypoint_file=entrypoint_file, start_server=start_server, host=host, port=port, env_vars=env_vars
+        app=app,
+        entrypoint_file=entrypoint_file,
+        start_server=start_server,
+        host=host,
+        port=port,
+        env_vars=env_vars,
+        secrets=secrets,
     )
     # a cloud dispatcher will return the result while local
     # dispatchers will be running the app in the main process
@@ -78,6 +89,7 @@ class Runtime:
     done: bool = False
     backend: Optional[Union[str, Backend]] = "multiprocessing"
     env_vars: Dict[str, str] = field(default_factory=dict)
+    secrets: Dict[str, str] = field(default_factory=dict)
 
     def __post_init__(self):
         if isinstance(self.backend, str):

src/lightning_app/utilities/secrets.py

@@ -0,0 +1,26 @@
+from typing import Dict, List
+
+from lightning_app.utilities.cloud import _get_project
+from lightning_app.utilities.network import LightningClient
+
+
+def _names_to_ids(secret_names: List[str]) -> Dict[str, str]:
+    """Returns the name/ID pair for each given Secret name.
+
+    Raises a `ValueError` if any of the given Secret names do not exist.
+    """
+    lightning_client = LightningClient()
+
+    project = _get_project(lightning_client)
+    secrets = lightning_client.secret_service_list_secrets(project_id=project.project_id)
+
+    secret_names_to_ids: Dict[str, str] = {}
+    for secret in secrets.secrets:
+        if secret.name in secret_names:
+            secret_names_to_ids[secret.name] = secret.id
+
+    for secret_name in secret_names:
+        if secret_name not in secret_names_to_ids.keys():
+            raise ValueError(f"Secret with name '{secret_name}' not found")
+
+    return secret_names_to_ids

tests/tests_app/cli/test_run_app.py

@@ -33,7 +33,6 @@ def _lightning_app_run_and_logging(self, *args, **kwargs):
 
     with caplog.at_level(logging.INFO):
         with mock.patch("lightning_app.LightningApp._run", _lightning_app_run_and_logging):
-
             runner = CliRunner()
             result = runner.invoke(
                 run_app,
@@ -70,6 +69,7 @@ def test_lightning_run_cluster_without_cloud(monkeypatch):
             open_ui=False,
             no_cache=True,
             env=("FOO=bar",),
+            secret=(),
         )
 
 
@@ -80,7 +80,7 @@ def test_lightning_run_app_cloud(mock_dispatch: mock.MagicMock, open_ui, caplog,
     """This test validates the command has ran properly when --cloud argument is passed.
 
     It tests it by checking if the click.launch is called with the right url if --open-ui was true and also checks the
-    call to `dispatch` for the right arguments
+    call to `dispatch` for the right arguments.
     """
     monkeypatch.setattr("lightning_app.runners.cloud.logger", logging.getLogger())
 
@@ -95,6 +95,7 @@ def test_lightning_run_app_cloud(mock_dispatch: mock.MagicMock, open_ui, caplog,
             open_ui=open_ui,
             no_cache=True,
             env=("FOO=bar",),
+            secret=("BAR=my-secret",),
         )
     # capture logs.
     # TODO(yurij): refactor the test, check if the actual HTTP request is being sent and that the proper admin
@@ -108,5 +109,25 @@ def test_lightning_run_app_cloud(mock_dispatch: mock.MagicMock, open_ui, caplog,
         name="",
         no_cache=True,
         env_vars={"FOO": "bar"},
+        secrets={"BAR": "my-secret"},
         cluster_id="",
     )
+
+
+def test_lightning_run_app_secrets(monkeypatch):
+    """Validates that running apps only supports the `--secrets` argument if the `--cloud` argument is passed."""
+    monkeypatch.setattr("lightning_app.runners.cloud.logger", logging.getLogger())
+
+    with pytest.raises(click.exceptions.ClickException):
+        _run_app(
+            file=os.path.join(_PROJECT_ROOT, "tests/tests_app/core/scripts/app_metadata.py"),
+            cloud=False,
+            cluster_id="test-cluster",
+            without_server=False,
+            name="",
+            blocking=False,
+            open_ui=False,
+            no_cache=True,
+            env=(),
+            secret=("FOO=my-secret"),
+        )

tests/tests_app/utilities/test_secrets.py

@@ -0,0 +1,49 @@
+from typing import Dict, List
+from unittest import mock
+from unittest.mock import MagicMock
+
+import pytest
+from lightning_cloud.openapi import V1ListMembershipsResponse, V1ListSecretsResponse, V1Membership, V1Secret
+
+from lightning_app.utilities.secrets import _names_to_ids
+
+
+@pytest.mark.parametrize(
+    "secret_names, secrets, expected, expected_exception",
+    [
+        ([], [], {}, False),
+        (
+            ["first-secret", "second-secret"],
+            [
+                V1Secret(name="first-secret", id="1234"),
+            ],
+            {},
+            True,
+        ),
+        (
+            ["first-secret", "second-secret"],
+            [V1Secret(name="first-secret", id="1234"), V1Secret(name="second-secret", id="5678")],
+            {"first-secret": "1234", "second-secret": "5678"},
+            False,
+        ),
+    ],
+)
+@mock.patch("lightning_cloud.login.Auth.authenticate", MagicMock())
+@mock.patch("lightning_app.utilities.network.LightningClient.secret_service_list_secrets")
+@mock.patch("lightning_app.utilities.network.LightningClient.projects_service_list_memberships")
+def test_names_to_ids(
+    list_memberships: MagicMock,
+    list_secrets: MagicMock,
+    secret_names: List[str],
+    secrets: List[V1Secret],
+    expected: Dict[str, str],
+    expected_exception: bool,
+):
+    list_memberships.return_value = V1ListMembershipsResponse(memberships=[V1Membership(project_id="default-project")])
+    list_secrets.return_value = V1ListSecretsResponse(secrets=secrets)
+
+    if expected_exception:
+        with pytest.raises(ValueError):
+            _names_to_ids(secret_names)
+    else:
+        assert _names_to_ids(secret_names) == expected