feat: add secrets concept to Jina(k8s) (#5557)

Co-authored-by: Jina Dev Bot <dev-bot@jina.ai>

docs/cloud-nativeness/docker-compose.md

@@ -45,12 +45,12 @@ If you change the Docker images in your Docker Compose generated file, ensure th
 the Gateway are built with the same Jina version to guarantee compatibility.
 ````
 
-## Example: Index and search images using CLIPEncoder and ANNLiteIndexer
+## Example: Index and search images using CLIPEncoder and AnnLiteIndexer
 
 Install [`Docker Compose`](https://docs.docker.com/compose/install/) locally to follow this how-to.
 
 This example shows how to build and deploy a Flow with Docker Compose, using [`CLIPImageEncoder`](https://cloud.jina.ai/executor/0hnlmu3q)
-as an image encoder and [`ANNLiteIndexer`](https://cloud.jina.ai/executor/7yypg8qk) as an indexer to perform fast nearest
+as an image encoder and [`AnnLiteIndexer`](https://cloud.jina.ai/executor/7yypg8qk) as an indexer to perform fast nearest
 neighbor retrieval on image embeddings.
 
 ### Deploy the Flow

docs/cloud-nativeness/k8s.md

@@ -58,7 +58,7 @@ Here are possible configuration options you may need to add or change
 - Add `requests` and `limits` for the resources of the different Pods 
 - Set up persistent volume storage to save your data on disk
 - Pass custom configuration to your Executor with `ConfigMap` 
-- Manage credentials of your Executor with secrets
+- Manage credentials of your Executor with Kubernetes secrets, you can use `f.add(..., env_from_secret={'SECRET_PASSWORD': {'name': 'mysecret', 'key': 'password'}})` to map them to Pod environment variables
 - Edit the default rolling update configuration
 
 

docs/cloud-nativeness/kubernetes.md

@@ -169,7 +169,7 @@ f = (
     .add(name='encoder', uses='jinaai+docker://jina-ai/CLIPEncoder', replicas=2)
     .add(
         name='indexer',
-        uses='jinaai+docker://jina-ai/ANNLiteIndexer',
+        uses='jinaai+docker://jina-ai/AnnLiteIndexer',
         uses_with={'dim': 512},
         shards=2,
     )
@@ -208,6 +208,70 @@ If you already have the simple Flow from the first example running on your clust
 kubectl apply -R -f ./k8s_flow
 ```
 
+### Deploy Flow with custom environment variables and secrets
+
+You can customize the environment variables that are available inside runtime, either defined directly or read from a [Kubernetes secret](https://kubernetes.io/docs/concepts/configuration/secret/):
+
+````{tab} with Python
+```python
+from jina import Flow
+
+f = (
+    Flow(port=8080)
+    .add(
+        name='indexer',
+        uses='jinaai+docker://jina-ai/AnnLiteIndexer',
+        uses_with={'dim': 512},
+        env={'k1': 'v1', 'k2': 'v2'},
+        env_from_secret={
+            'SECRET_USERNAME': {'name': 'mysecret', 'key': 'username'},
+            'SECRET_PASSWORD': {'name': 'mysecret', 'key': 'password'},
+        },
+    )
+)
+
+f.to_kubernetes_yaml('./k8s_flow', k8s_namespace='custom-namespace')
+```
+````
+````{tab} with flow YAML
+In a `flow.yml` file :
+```yaml
+jtype: Flow
+version: '1'
+with:
+  protocol: http
+executors:
+- name: indexer
+  uses: jinaai+docker://jina-ai/AnnLiteIndexer
+  uses_with:
+    dim: 512
+  env:
+    k1: v1
+    k2: v2
+  env_from_secret:
+    SECRET_USERNAME:
+      name: mysecret
+      key: username
+    SECRET_PASSWORD:
+      name: mysecret
+      key: password
+```
+
+You can generate Kubernetes YAML configs using `jina export`:
+```shell
+jina export kubernetes flow.yml ./k8s_flow --k8s-namespace custom-namespace
+```
+````
+
+After creating the namespace, you need to create the secrets mentioned above:
+
+```shell
+kubectl -n custom-namespace create secret generic mysecret --from-literal=username=jina --from-literal=password=123456
+```
+
+Then you can apply your configuration.
+
+
 
 (kubernetes-expose)=
 ## Exposing the Flow

docs/concepts/flow/executor-args.md

@@ -29,6 +29,7 @@
 | `runtime_cls` | The runtime class to run inside the Pod | `string` | `WorkerRuntime` |
 | `timeout_ready` | The timeout in milliseconds of a Pod waits for the runtime to be ready, -1 for waiting forever | `number` | `600000` |
 | `env` | The map of environment variables that are available inside runtime | `object` | `None` |
+| `env_from_secret` | The map of environment variables that are read from kubernetes cluster secrets | `object` | `None` |
 | `floating` | If set, the current Pod/Deployment can not be further chained, and the next `.add()` will chain after the last Pod/Deployment not this current one. | `boolean` | `False` |
 | `reload` | If set, the Executor will restart while serving if YAML configuration source or Executor modules are changed. If YAML configuration is changed, the whole deployment is reloaded and new processes will be restarted. If only Python modules of the Executor have changed, they will be reloaded to the interpreter without restarting process. | `boolean` | `False` |
 | `install_requirements` | If set, try to install `requirements.txt` from the local Executor if exists in the Executor folder. If using Hub, install `requirements.txt` in the Hub Executor bundle to local. | `boolean` | `False` |

docs/concepts/flow/gateway-args.md

@@ -36,6 +36,7 @@
 | `runtime_cls` | The runtime class to run inside the Pod | `string` | `GatewayRuntime` |
 | `timeout_ready` | The timeout in milliseconds of a Pod waits for the runtime to be ready, -1 for waiting forever | `number` | `600000` |
 | `env` | The map of environment variables that are available inside runtime | `object` | `None` |
+| `env_from_secret` | The map of environment variables that are read from kubernetes cluster secrets | `object` | `None` |
 | `floating` | If set, the current Pod/Deployment can not be further chained, and the next `.add()` will chain after the last Pod/Deployment not this current one. | `boolean` | `False` |
 | `reload` | If set, the Gateway will restart while serving if YAML configuration source is changed. | `boolean` | `False` |
 | `port` | The port for input data to bind the gateway server to, by default, random ports between range [49152, 65535] will be assigned. The port argument can be either 1 single value in case only 1 protocol is used or multiple values when many protocols are used. | `number` | `random in [49152, 65535]` |

jina/orchestrate/deployments/config/helper.py

@@ -92,6 +92,7 @@ def construct_runtime_container_args(cargs, uses_metas, uses_with, pod_type):
         'workspace_id',
         'noblock_on_start',
         'env',
+        'env_from_secret',
     }
 
     if pod_type == PodRoleType.HEAD:

jina/orchestrate/deployments/config/k8s.py

@@ -208,6 +208,7 @@ def get_runtime_yamls(
                 pod_type=self.pod_type,
                 shard_id=self.shard_id,
                 env=cargs.env,
+                env_from_secret=cargs.env_from_secret,
                 gpus=cargs.gpus if hasattr(cargs, 'gpus') else None,
                 monitoring=cargs.monitoring,
                 port_monitoring=cargs.port_monitoring,

jina/orchestrate/deployments/config/k8slib/kubernetes_deployment.py

@@ -20,6 +20,7 @@ def get_template_yamls(
     shard_id: Optional[int] = None,
     port: Optional[Union[int, List[int]]] = None,
     env: Optional[Dict] = None,
+    env_from_secret: Optional[Dict] = None,
     gpus: Optional[Union[int, str]] = None,
     image_name_uses_before: Optional[str] = None,
     image_name_uses_after: Optional[str] = None,
@@ -46,6 +47,7 @@ def get_template_yamls(
     :param shard_id: id of this shard, None if shards=1 or this is gateway/head
     :param port: port which will be exposed by the deployed containers
     :param env: environment variables to be passed into configmap.
+    :param env_from_secret: environment variables from secret to be passed to this pod
     :param gpus: number of gpus to use, for k8s requires you pass an int number, refers to the number of requested gpus.
     :param image_name_uses_before: image for uses_before container in the k8s deployment
     :param image_name_uses_after: image for uses_after container in the k8s deployment
@@ -97,6 +99,7 @@ def get_template_yamls(
         'jina_deployment_name': jina_deployment_name,
         'shard_id': f'\"{shard_id}\"' if shard_id is not None else '\"\"',
         'pod_type': pod_type,
+        'env_from_secret': env_from_secret,
         'protocol': str(protocols[0]).lower() if protocols[0] is not None else '',
         'volume_path': volumes[0] if volumes is not None else None,
     }

jina/orchestrate/deployments/config/k8slib/kubernetes_tools.py

@@ -25,9 +25,12 @@ def get_yaml(template: str, params: Dict) -> Dict:
     """
     if template == 'configmap':
         yaml = _get_configmap_yaml(template, params)
-    elif template in DEPLOYMENT_FILES and params.get('device_plugins'):
+    elif template in DEPLOYMENT_FILES:
         yaml = _get_yaml(template, params)
-        yaml = _get_deployment_with_device_plugins(yaml, params)
+        if params.get('device_plugins'):
+            yaml = _get_deployment_with_device_plugins(yaml, params)
+        if params.get('env_from_secret'):
+            yaml = _get_deployment_with_env_secret(yaml, params)
     else:
         yaml = _get_yaml(template, params)
 
@@ -77,3 +80,13 @@ def _get_deployment_with_device_plugins(deployment: Dict, params: Dict) -> Dict:
         'resources'
     ] = device_plugins
     return deployment
+
+def _get_deployment_with_env_secret(deployment: Dict, params: Dict) -> Dict:
+    for k,v in params['env_from_secret'].items():
+        env_var = {}
+        env_var['name'] = k
+        env_var['valueFrom'] = {'secretKeyRef': {'name': v['name'], 'key': v['key']}}
+
+        deployment['spec']['template']['spec']['containers'][0]['env'].append(env_var)
+
+    return deployment

jina/orchestrate/flow/base.py

@@ -168,6 +168,7 @@ def __init__(
         docker_kwargs: Optional[dict] = None, 
         entrypoint: Optional[str] = None, 
         env: Optional[dict] = None, 
+        env_from_secret: Optional[dict] = None, 
         expose_endpoints: Optional[str] = None, 
         expose_graphql_endpoint: Optional[bool] = False, 
         floating: Optional[bool] = False, 
@@ -222,6 +223,7 @@ def __init__(
           More details can be found in the Docker SDK docs:  https://docker-py.readthedocs.io/en/stable/
         :param entrypoint: The entrypoint command overrides the ENTRYPOINT in Docker image. when not set then the Docker image ENTRYPOINT takes effective.
         :param env: The map of environment variables that are available inside runtime
+        :param env_from_secret: The map of environment variables that are read from kubernetes cluster secrets
         :param expose_endpoints: A JSON string that represents a map from executor endpoints (`@requests(on=...)`) to HTTP endpoints.
         :param expose_graphql_endpoint: If set, /graphql endpoint is added to HTTP interface.
         :param floating: If set, the current Pod/Deployment can not be further chained, and the next `.add()` will chain after the last Pod/Deployment not this current one.
@@ -400,6 +402,7 @@ def __init__(
           More details can be found in the Docker SDK docs:  https://docker-py.readthedocs.io/en/stable/
         :param entrypoint: The entrypoint command overrides the ENTRYPOINT in Docker image. when not set then the Docker image ENTRYPOINT takes effective.
         :param env: The map of environment variables that are available inside runtime
+        :param env_from_secret: The map of environment variables that are read from kubernetes cluster secrets
         :param expose_endpoints: A JSON string that represents a map from executor endpoints (`@requests(on=...)`) to HTTP endpoints.
         :param expose_graphql_endpoint: If set, /graphql endpoint is added to HTTP interface.
         :param floating: If set, the current Pod/Deployment can not be further chained, and the next `.add()` will chain after the last Pod/Deployment not this current one.
@@ -847,6 +850,7 @@ def add(
         docker_kwargs: Optional[dict] = None, 
         entrypoint: Optional[str] = None, 
         env: Optional[dict] = None, 
+        env_from_secret: Optional[dict] = None, 
         exit_on_exceptions: Optional[List[str]] = [], 
         external: Optional[bool] = False, 
         floating: Optional[bool] = False, 
@@ -907,6 +911,7 @@ def add(
           More details can be found in the Docker SDK docs:  https://docker-py.readthedocs.io/en/stable/
         :param entrypoint: The entrypoint command overrides the ENTRYPOINT in Docker image. when not set then the Docker image ENTRYPOINT takes effective.
         :param env: The map of environment variables that are available inside runtime
+        :param env_from_secret: The map of environment variables that are read from kubernetes cluster secrets
         :param exit_on_exceptions: List of exceptions that will cause the Executor to shut down.
         :param external: The Deployment will be considered an external Deployment that has been started independently from the Flow.This Deployment will not be context managed by the Flow.
         :param floating: If set, the current Pod/Deployment can not be further chained, and the next `.add()` will chain after the last Pod/Deployment not this current one.
@@ -1053,6 +1058,7 @@ def add(
           More details can be found in the Docker SDK docs:  https://docker-py.readthedocs.io/en/stable/
         :param entrypoint: The entrypoint command overrides the ENTRYPOINT in Docker image. when not set then the Docker image ENTRYPOINT takes effective.
         :param env: The map of environment variables that are available inside runtime
+        :param env_from_secret: The map of environment variables that are read from kubernetes cluster secrets
         :param exit_on_exceptions: List of exceptions that will cause the Executor to shut down.
         :param external: The Deployment will be considered an external Deployment that has been started independently from the Flow.This Deployment will not be context managed by the Flow.
         :param floating: If set, the current Pod/Deployment can not be further chained, and the next `.add()` will chain after the last Pod/Deployment not this current one.
@@ -1246,6 +1252,7 @@ def config_gateway(
         docker_kwargs: Optional[dict] = None, 
         entrypoint: Optional[str] = None, 
         env: Optional[dict] = None, 
+        env_from_secret: Optional[dict] = None, 
         expose_endpoints: Optional[str] = None, 
         expose_graphql_endpoint: Optional[bool] = False, 
         floating: Optional[bool] = False, 
@@ -1300,6 +1307,7 @@ def config_gateway(
           More details can be found in the Docker SDK docs:  https://docker-py.readthedocs.io/en/stable/
         :param entrypoint: The entrypoint command overrides the ENTRYPOINT in Docker image. when not set then the Docker image ENTRYPOINT takes effective.
         :param env: The map of environment variables that are available inside runtime
+        :param env_from_secret: The map of environment variables that are read from kubernetes cluster secrets
         :param expose_endpoints: A JSON string that represents a map from executor endpoints (`@requests(on=...)`) to HTTP endpoints.
         :param expose_graphql_endpoint: If set, /graphql endpoint is added to HTTP interface.
         :param floating: If set, the current Pod/Deployment can not be further chained, and the next `.add()` will chain after the last Pod/Deployment not this current one.
@@ -1395,6 +1403,7 @@ def config_gateway(
           More details can be found in the Docker SDK docs:  https://docker-py.readthedocs.io/en/stable/
         :param entrypoint: The entrypoint command overrides the ENTRYPOINT in Docker image. when not set then the Docker image ENTRYPOINT takes effective.
         :param env: The map of environment variables that are available inside runtime
+        :param env_from_secret: The map of environment variables that are read from kubernetes cluster secrets
         :param expose_endpoints: A JSON string that represents a map from executor endpoints (`@requests(on=...)`) to HTTP endpoints.
         :param expose_graphql_endpoint: If set, /graphql endpoint is added to HTTP interface.
         :param floating: If set, the current Pod/Deployment can not be further chained, and the next `.add()` will chain after the last Pod/Deployment not this current one.

jina/parsers/orchestrate/pod.py

@@ -62,6 +62,14 @@ def mixin_pod_parser(parser, pod_type: str = 'worker'):
         help='The map of environment variables that are available inside runtime',
     )
 
+    gp.add_argument(
+        '--env-from-secret',
+        action=KVAppendAction,
+        metavar='KEY: VALUE',
+        nargs='*',
+        help='The map of environment variables that are read from kubernetes cluster secrets',
+    )
+
     # hidden CLI used for internal only
 
     gp.add_argument(

jina/serve/executors/__init__.py

@@ -532,6 +532,7 @@ def serve(
         docker_kwargs: Optional[dict] = None, 
         entrypoint: Optional[str] = None, 
         env: Optional[dict] = None, 
+        env_from_secret: Optional[dict] = None, 
         exit_on_exceptions: Optional[List[str]] = [], 
         external: Optional[bool] = False, 
         floating: Optional[bool] = False, 
@@ -592,6 +593,7 @@ def serve(
           More details can be found in the Docker SDK docs:  https://docker-py.readthedocs.io/en/stable/
         :param entrypoint: The entrypoint command overrides the ENTRYPOINT in Docker image. when not set then the Docker image ENTRYPOINT takes effective.
         :param env: The map of environment variables that are available inside runtime
+        :param env_from_secret: The map of environment variables that are read from kubernetes cluster secrets
         :param exit_on_exceptions: List of exceptions that will cause the Executor to shut down.
         :param external: The Deployment will be considered an external Deployment that has been started independently from the Flow.This Deployment will not be context managed by the Flow.
         :param floating: If set, the current Pod/Deployment can not be further chained, and the next `.add()` will chain after the last Pod/Deployment not this current one.

jina_cli/autocomplete.py

@@ -54,6 +54,7 @@
             '--runtime-cls',
             '--timeout-ready',
             '--env',
+            '--env-from-secret',
             '--shard-id',
             '--pod-role',
             '--noblock-on-start',
@@ -151,6 +152,7 @@
             '--runtime-cls',
             '--timeout-ready',
             '--env',
+            '--env-from-secret',
             '--shard-id',
             '--pod-role',
             '--noblock-on-start',
@@ -272,6 +274,7 @@
             '--runtime-cls',
             '--timeout-ready',
             '--env',
+            '--env-from-secret',
             '--shard-id',
             '--pod-role',
             '--noblock-on-start',
@@ -333,6 +336,7 @@
             '--runtime-cls',
             '--timeout-ready',
             '--env',
+            '--env-from-secret',
             '--shard-id',
             '--pod-role',
             '--noblock-on-start',

tests/k8s/test-executor/debug_executor.py

@@ -41,6 +41,8 @@ def env(self, docs: DocumentArray, **kwargs):
             doc.tags['k2'] = os.environ.get('k2')
             doc.tags['JINA_LOG_LEVEL'] = os.environ.get('JINA_LOG_LEVEL')
             doc.tags['env'] = {'k1': os.environ.get('k1'), 'k2': os.environ.get('k2')}
+            doc.tags['SECRET_USERNAME'] = os.environ.get('SECRET_USERNAME')
+            doc.tags['SECRET_PASSWORD'] = os.environ.get('SECRET_PASSWORD')
 
     @requests(on='/cuda')
     def cuda(self, docs: DocumentArray, **kwargs):