[Core feature] Project-domain level settings

[wild-endeavor]

Motivation: Why do you think this is important?

Background info

Registration time settings

We've removed most of the deployment-specific settings from the compiled/serialized Flyte entity protobuf files (i.e. the assets packaged along with the flytesnacks releases). This makes those protobufs "portable" in the sense that anyone with any deployment of Flyte can use them.

However this means that when registering those or any other Flyte assets, users must not only specify the project/domain, they have to specify all the other settings, leading to a registration command that looks like the following

flytectl -c ~/.flyte/config.yaml register file /Users/ytong/go/src/github.com/flyteorg/flytesnacks/cookbook/case_studies/feature_engineering/feast_integration/_pb_output/* -p flytesnacks -d development --version legacyv1 --k8sServiceAccount default --outputLocationPrefix "s3://my-s3-bucket/raw-data"

The options exposed are:

• --k8sServiceAccount
• --outputLocationPrefix
• --assumableIamRole

Labels & Annotations

These are some settings on launch plans that might be pretty uniform across a project.
https://github.com/flyteorg/flyteidl/blob/master/protos/flyteidl/admin/launch_plan.proto#L98

• Labels
• Annotations
• QualityOfService

Matchable entities

Please take a look at the current documentation for the "matchable entities".
https://docs.flyte.org/en/latest/concepts/admin.html#divedeep-admin-matchable-resources
https://docs.flyte.org/en/latest/deployment/cluster_config/general.html#deployment-cluster-config-general

There are already some settings that are set along the project-domain dimension. Please reference https://docs.flyte.org/projects/flyteidl/en/latest/protos/docs/admin/admin.html#matchableresource
Some of these are not strictly user-facing though, like execution queue and cluster label, plugin override, etc. These are more platform admin facing. Default resources though probably make sense to expose to the user (at least non-writable).

Request

This issue requests that settings that are typically uniform across a project/domain combination, that make sense to be exposed to the user, to be so exposed, with configuration available to the user via flytectl and the UI. How this issue meshes with existing implementation of matchable resources, and which settings take priority, will be left to the implementer.

Goal: What should the final outcome look like, ideally?

After this feature is implemented, users should be able to

• Register files in flytectl to a project/domain and pick up things like iam role, service account, output location, by default, without the user specifying them.
• Be able to set labels/annotations on a project/domain level and have them take effect in K8s.
• Understand after registration which settings were specified by the entity and which settings are overrides provided at the project/domain level (and possibly the project level).

In the future, if we have time, this ticket should also unlock the ability for Admin to be able to "promote" Flyte entities from one domain in cases where no additional user-input is needed (but specification of that will be another ticket(s)).

Describe alternatives you've considered

None.

Propose: Link/Inline OR Additional context

Possibly related:
#2050
#211

Are you sure this issue hasn't been raised already?

• Yes

Have you read the Code of Conduct?

• Yes

[kumare3]

cc @jeevb / @cosmicBboy / @narape Some of you have indirectly asked for this feature in the past.

One of the benefits of this would be that, for every execution (including single task execution), this default ServiceAccount would be applied.

cc @katrogan can you help write this up?

[katrogan]

here's a proposal for what the matchable entity message structure could look like: flyteorg/flyteidl#253

much like we currently do max_parallelism, we can use these attributes at create execution time when a user's ExecutionSpec and corresponding LaunchPlanSpec omit these values:
https://github.com/flyteorg/flyteadmin/blob/352943f8a86f600510a2c23e4e730736730b42d1/pkg/manager/impl/execution_manager.go#L436,L467

[wild-endeavor]

@katrogan Is the WorkflowExecutionConfig message applicable on the project-domain-workflow/lp specificity level?

[katrogan]

@wild-endeavor yes it should be

[EngHabu]

For UX Comps development:

• Existing APIs for interacting with projects. (It's missing a GET api for a specific project, this should be added as part of the work here)
• The current structure for the project object.
• The new fields here

From UX standpoint, users should be able to specify project-level settings as well as per project per domain settings...
The project level settings remain as they are up there... one of the project fields is an array of Domains... for each of project-domain combination, we can retrieve/update settings through these APIs

Some mockup to demonstrate which API to use to populate which part:

For the Editing UI, we can offer some syntactic sugar to allow users to set the project-domain settings once and apply them to all domains (maybe a check box that says "apply these settings to all domains")... and under the hood that can make the separate API calls to update the settings in all domains. If users uncheck this box, they should be able to edit these settings for each domain separately...

[anrusina]

FE task - https://github.com/flyteorg/flyteconsole/issues/317 will be updated with more info and design

[EngHabu]

flytectl UX:

1. Retrieve [Not Hi Pri]:

   #punt: flytectl get settings -d development
   flytectl get settings -d development -p flytesnacks
   #punt: flytectl get settings -d development -p flytesnacks --workflow app.blah.bloh

   domain: development
   project: flytesnacks
   settings:
     task-resource:
       cpu: ...
     workflow-exec:
       maxParallelism: ...
       securityContext: ...
   workflow-settings:
     - name: app.blah.bloh
        settings:
         task-resource:
           cpu: ...
         workflow-exec:
           maxParallelism: ...
           securityContext: ...

2. Update Settings:

    flytectl get settings -d development -p flytesnacks --outputFile spec.yaml
    flytectl update settings -d development -p flytesnacks --spec spec.yaml

   domain: development
   project: flytesnacks
   settings:
     task-resource:
       cpu: ...
     workflow-exec:
       maxParallelism: ...
       securityContext: ...
   workflow-settings:
     - name: app.blah.bloh
        settings:
         task-resource:
           cpu: ...
         workflow-exec:
           maxParallelism: ...
           securityContext: ...

3. [Open Question] How to delete Settings?

[katrogan]

3. [Open Question] How to delete Settings?

there is a delete endpoint! https://github.com/flyteorg/flyteidl/blob/master/protos/flyteidl/service/admin.proto#L482

[pmahindrakar-oss]

Resolving this issue as done. The global setting would be taken up in separate ticket 2322