[Feature] Support for per workflow DataReference (RawOutputDataPrefix) configuration

[kumare3]

Motivation: Why do you think this is important?
Today Flyte stores data in a preconfigured data output sandbox. It may be desirable to customize this per workflow execution. FlytePropeller should be able to configure this dynamically for the execution.

Goal: What should the final outcome look like, ideally?
A user could launch a workflow with their own data-sandbox paths, all data is created in that path.

Describe alternatives you've considered
Today Flyte supports custom data paths, but they are supported in flytekit. The user now has to take care of de-duplication, conflicts, uniqueness etc. This solution would make this simpler for the user.

Flyte component

• Overall
• Flyte Setup and Installation scripts
• Flyte Documentation
• Flyte communication (slack/email etc)
• FlytePropeller
• FlyteIDL (Flyte specification language)
• Flytekit (Python SDK)
• FlyteAdmin (Control Plane service)
• FlytePlugins
• DataCatalog
• FlyteStdlib (common libraries)
• FlyteConsole (UI)
• Other

[Optional] Propose: Link/Inline
This can be done easily in FlytePropeller. It already has support for DefaultDataSandbox. FlyteAdmin may decide to add a new sandbox per workflow.

Additional context
NA

Is this a blocker for you to adopt Flyte
Nope. But this will definitely enable usecases like fully managed workflows for sensitive data applications.

[wild-endeavor]

How do we want to do this?

Since Feb 2020, Flyte plugin machinery has supported the notion of a RawOutputPrefix, detailed in the code here. From a user's perspective, this is currently set in a workflow repo's flytekit config. This is fine for now, but for tasks like the Presto task, which does not involve running the user's container, there is currently no way to get at this information at runtime.

The thinking right now is that this seems pretty closely tied to the role that a workflow execution would ultimately run with, since where a workflow execution writes its output is closely tied with permissions and permissions are closely tied with IAM roles/service accounts. Currently, there is an entry for the AuthRole inside the LaunchPlanSpec. This is used shortly before actually creating the FlyteWorkflow CRD on the K8s cluster. From there, the metadata flows through to all objects created (like Pods) created for that CRD.

Unlike the auth role, which takes the form of a K8s metadata annotation, or a K8s entity ServiceAccount, the OutputPrefix has no parallel in the K8s domain, it's an entirely Flyte concept. How do other pure-Flyte settings flow from the control plane into a WF CRD's objects? Things like Flyte's cname address, oauth settings, statds hostnames, aws cli control settings, are all set via environment variables by propeller upon execution.

The proposed steps are:

1. Add an entry into the LaunchPlanSpec (output_data_prefix) for an output prefix (I think we can drop the 'raw' bit).
2. In flytekit, add a new configuration object and include it in the launchplanspec sent up to Admin. This effectively elevates this config from the user repo to the Flyte control plane.
3. When it comes time for execution, Admin should pass down this information to the WF CRD. This information will need to make it down to:
   1. User pods/containers, as created by various plugins.
   2. TaskExecutionContext objects provided by Propeller to plugins
4. We should probably pass this information by both
   1. As an explicit field on the WF CRD. This field will store the full string as specified by the user, but with no {} sharding characters.
   2. Fill in templated values like we do for {{ .Inputs.blah }} currently, only now we'll also fill in {{ .outputDataPrefix }}
5. Flytekit currently uses the S3 shard formatter option for three things: writing blobs, writing schemas, and generating Hive query output locations. We should change pyflyte-execute to pick up this new templated value and use it when doing these things, and change the default container args to have this template field. This change should be done as a separate PR from the above.

To preserve existing behavior, Propeller should not fill in the template, nor set the new WF CRD field if the launch plan does not have the OutputPrefix explicitly set.

[kumare3]

We should probably pass this information by both
As an explicit field on the WF CRD. This field will store the full string as specified by the user, but with no {} sharding characters.
Fill in templated values like we do for {{ .Inputs.blah }} currently, only now we'll also fill in {{ .outputDataPrefix }}
I do not follow this.

So IMO,
OutputPrefix should be part of the configuration, which can be overriden for a specific execution as well. We could create execution config as a meta-object, where OutputPrefix is one attribute

I agree, this is passed into the CRD as a top level attribute.
FlytePropeller should set this value as a command line option whenever it encounters {{ .ouputs.prefix }} or similar variable.

You will need to update the templating logic - https://github.com/lyft/flyteplugins/blob/master/go/tasks/pluginmachinery/utils/template.go#L20
And use
https://github.com/lyft/flyteplugins/blob/master/go/tasks/pluginmachinery/io/iface.go#L51

[wild-endeavor]

After some in person discussion, we have decided it's best to use an object to encapsulate the IDL change, in case future settings are also required (sharding behavior for instance). These can be added in the future as part of other tickets.

Also revamping execution time overrides is out of scope for this ticket, so for now, users will not be able to override the output prefix. If users want to write to a different location, they will have to create a new launch plan object.

[kumare3]

@EngHabu this seems to have been completed?

[wild-endeavor]

i'm not gonna have enough time to test everything by Friday.

[wild-endeavor]

Minimum versions for everything to work for this feature:
IDL: https://github.com/lyft/flyteidl/releases/tag/v0.18.1
Admin: https://github.com/lyft/flyteadmin/releases/tag/v0.3.1
Propeller: https://github.com/lyft/flytepropeller/releases/tag/v0.3.10
Plugins: https://github.com/lyft/flyteplugins/releases/tag/v0.4.5
Flytekit: https://github.com/lyft/flytekit/releases/tag/v0.12.1