Umbrella issue: Implement pipeline-aware function run workflows

[droot]

There is significant change in "kpt fn run" workflows with introduction of pipeline construct. Following tasks need to be accomplished:

• Implement imperative function run (kpt fn run ---)
• Implement support for metafunctions
• Utility to generate pipeline from functionConfigs in a pkg
• Deprecate current workflow for declarative fn run

[bgrant0607]

Imperative function run is now eval: https://kpt.dev/book/04-using-functions/02-imperative-function-execution

Regarding metafunctions, modification of Kptfiles, and generation of pipelines:
https://kpt.dev/book/04-using-functions/02-imperative-function-execution?id=mutate-meta-resources

Per-function filters are discussed in #2364. krmignore is discussed in #1733 and #517.

If Kptfile is not passed as part of the ResourceList by default, is it essentially like one of these other types of filters except that it's applied by default and disabled by --include-meta-resources? If --include-meta-resources is applied to kpt fn source, does it also need to be applied to kpt fn eval invocations that read from stdin? Is there a way to specify it in a declarative pipeline?

Also, it's unclear whether ConfigMaps and pseudo-resources used for function inputs for particular functions via configPath can be passed in the ResourceList to other functions:
https://kpt.dev/book/04-using-functions/01-declarative-function-execution
https://kpt.dev/book/04-using-functions/02-imperative-function-execution?id=mutate-meta-resources

Use cases that have come up include:

• The ability for functions to generate inputs, such as ConfigMaps, to other functions, particularly of nested subpackages.
• The ability to transform inputs to other functions, such as in the case of setter replacement in a pseudo-resource that's an input to a generator function.
• The ability to generate Kptfiles for subpackages, which could be subsequently fetched.
• The ability to enable or disable pipeline steps. (Optional pipeline steps: enabled=false boolean #2306)
• The ability to arbitrarily transform any part of a package or even generate a package from scratch and write it via kpt fn sink.

At least the first two cases also have needed top-down preprocessing (preorder execution) prior to bottom-up rendering (post-order execution) during declarative execution.

[bgrant0607]

cc @justinsb

[droot]

Thanks for the detailed comment @bgrant0607 . This is helpful as we are beginning the design for supporting filtering in fn commands. (#2015 )

Will try to answer part of the comment here:

If Kptfile is not passed as part of the ResourceList by default, is it essentially like one of these other types of filters except that it's applied by default and disabled by --include-meta-resources?

this is fairly accurate in context of fn eval and fn source. We still need to rationalize this with fn render. Quick thought suggests that all the use-cases for processing meta-resources in render actually fits the use-cases for pre-processing step.

If --include-meta-resources is applied to kpt fn source, does it also need to be applied to kpt fn eval invocations that read from stdin?

I think both answers (yes, no) are possible in this case, but optimizing for the common CUJs should drive the answer. Quick thought suggests --include-meta-resources doesn't need to be applied to fn eval invocations reading from stdin because we can assume resources on stdin are already filtered (if one was using fn source or previous fn eval).

Is there a way to specify it in a declarative pipeline?

I think the need for this has a big overlap with pre-processing step requirements I think.

I need to think some more on the second part.

[bgrant0607]

Kind of related, I'd like kpt fn render to be able to take an entire package with a Kptfile with a pipeline from stdin.

Why? Because I always want to re-run render after other mutations in order to enforce invariants. I want to treat the pipeline like mutating + validating admission control.

[bgrant0607]

Every time we clone a package to create a variant of the package, we need to specialize the variant somehow.

The most basic thing that needs to be done in pretty much all cases is personalization: change the names of things:
https://github.com/kubernetes/community/blob/master/contributors/design-proposals/architecture/declarative-application-management.md#configuration-customization-and-variant-generation

We do this for the Kptfile's metadata.name field automatically. I would like to use that as a standard input for a variant constructor pattern. This pattern would use the first mutator in the Kptfile to do the basic variant specialization required by the package. We may also want to think about how to use the name as input to other standard name-changing functions, such as set-namespace, ensure-name-substring, and so on.

In general, for our use cases, I always want --include-meta-resources to be true, in source, eval, render, wherever. We need to be able to view and change the info in the Kptfile. It makes more sense to me to filter config objects in the SDKs or in function-specific filtering parameters.

[bgrant0607]

@phanimarupaka I described my use case above