Obfuscate sensitive data in TaskConfig

[rambrus]

Motivation: Why do you think this is important?

Flyte dumps all TaskConfig to the UI and logs, including sensitive data (such as access tokens).
This is a critical security issue in enterprise environments.

Goal: What should the final outcome look like, ideally?

Flyte admins can mark certain fields as sensitive in the config, sensitive fields are obfuscated throughout the application.

Describe alternatives you've considered

If this approach does not fit the design, perhaps we should consider a more generic approach similar to the one implemented in Databricks. All the values retrieved from the secret store are treated as sensitive data and obfuscated throughout the application.

Propose: Link/Inline OR Additional context

https://docs.databricks.com/en/security/secrets/redaction.html

Are you sure this issue hasn't been raised already?

• Yes

Have you read the Code of Conduct?

• Yes

[runllm]

To obfuscate sensitive data in Flyte's TaskConfig, you can use Flyte's secret management feature. This allows you to securely pass sensitive information to your tasks without exposing it in the UI or logs. Here's an example of how to use Flyte's secret management:

1. Define Secrets: Use the Secret class to define the secrets required by your task.
2. Access Secrets: Retrieve the secrets within your task using flytekit.current_context().secrets.get.

import flytekit
from flytekit import Secret, task, workflow

SECRET_NAME = "user_secret"
SECRET_GROUP = "user-info"

@task(secret_requests=[Secret(group=SECRET_GROUP, key=SECRET_NAME)])
def secret_task() -> str:
    secret_val = flytekit.current_context().secrets.get(SECRET_GROUP, SECRET_NAME)
    # Use the secret value securely
    return secret_val

@workflow
def wf() -> str:
    return secret_task()

Would you like more detail on this approach?

Sources:

• Is there any way to mask secrets that are passed between tasks?
• Apologies if I’m missing something in the docs, but I’m tryin
• Question is it possible to use a Secret inside of spark_conf

---

This is an AI-generated response and your feedback is appreciated! Please leave a 👍 if this is helpful and 👎 if it is not.

Mention @RunLLM to ask follow-up questions. See runllm.com for more!

[kumare3]

We should not store sensitive information in task config. Secrets should be used for this

[rambrus]

@kumare3
When Databricks TaskConfig is defined, sensitive data (auth) needs to be passed in these fields:

• databricksConf / new_cluster / docker_image / basic_auth / password
• databricksToken

Do you mean these fields should not be exposed in TaskConfig, but the plugin / agent should be responsible to fill them from Secrets?

[kumare3]

Correct

[eapolinario]

@pingsutw , can you list the remaining work here?