PIP: Manifest based configuration of KRM function executions

[mozesl-nokia]

PIP: Manifest based configuration of KRM function executions

"Solution" part based on analysis by @kispaljr

Related Issues:

• Parallel KRM function execution #884
• EPIC: Function Runner Improvements #888

Related (non-implementing) PRs:

• Multiple parallel KRM function evaluator pods for the same KRM functions #336
• feat: Add parallel KRM function execution support #425

Problem

Currently, the configuration of individual function pipelines in Porch is done in 3 different places:

• a ConfigMap containing a YAML for the pod evaluator (which will now be referred to as "pod executor")
• a YAML file built into the function runner image for the executable evaluator (will be referred to as "binary executor")
• hard-coded map of images for the build-in runtime (will be referred to as "go executor")

None of these approaches are very "kubernetes-like", and the latter two are especially rigid.

Solution

FunctionConfig CRD

A new CRD placed in config.porch.kpt.dev named functionconfigs, which handles "mapping" between KRM function images and desired runner configuration.

Example CR:

apiVersion: config.porch.kpt.dev/v1alpha1
kind: FunctionConfig
metadata:
  name: apply-setters
spec:
  image: apply-setters                  # must be unique among FunctionConfigs, but we could consider merging configurations later
  prefixes:                             # a list of image prefixes to prepend the `image` name when checking for matches
    - my-ghcr-io-mirror.com/kptdev/krm-functions-catalog # the first entry will be used for pod cache warmup
    - ""                                # an empty string indicates the "default image prefix", so we will apply the config to that as well
    - ghcr.io/kptdev/krm-functions-catalog
  podExecutorConfig:                    # configuration for functions which will run in the pod executor/evaluator
    tags:                               # image tags which should use pod executor *and* the below config
      - v0.2.0                          # the first tag here will be used for cache warmup
      - v0.1.1                          # the goExecutorConfig will take precedence over this
      - v0.1
    serviceAccount: my-service-account  # possibility to give runner pod a service account (just for demonstration purposes, this can be implemented separately later)
    resources:                          # define runner pod resource constraints
      requests:
        cpu: 250m
        memory: 200M
      limits:
        cpu: 500m
        memory: 1G
    timeToLive: 20m
    maxParallelExecutions: 3            # see https://github.com/nephio-project/porch/pull/336
    preferredMaxQueueLength: 2          # see https://github.com/nephio-project/porch/pull/336
  binaryExecutorConfig:                 # configuration for functions which *can* run in the binary executor/executable evaluator
    tags:                               # the order doesn't matter here, if any of the tags match, the binary will be used
      - v0.2.3
      - v0.2
    relPath: apply-setters              # binary path relative to the dir defined in the function runner `--functions` arg
    # absPath: /functions/apply-setters # you can also give an absolute path instead, allows for better support of mounted volumes
                                        # relPath and absPath are mutually exclusive
  goExecutorConfig:                     # configuration for functions which *can* run as built-in go function calls
    tags:                               # the order doesn't matter here, if any of the tags match, the go executor will be used
      - v0.1.1
    # id: apply-setters                 # possibility to define a custom ID by which Porch can determine what method to run; if empty, we use `image`

FunctionConfig controller

Watches the FunctionConfig CRs and updates its internal cache of configurations whilst also doing additional validations.
This cache needs to efficiently match full image names to the stored configurations.

This cache must be accessible to the caller(s) of all 3 executors, which is why it might be necessary to move the built-in runtime to function runner
or proceed with merging the evaluators into porch server.
If we want to keep this PIP separate from (and implement it before) the function runner merging, we can read the configs in both function runner and porch server for now.

Updated Function Execution Request Handling

(This section assumes that there is a single multi-evaluator that handles all 3 evaluators.)

A single multi-evaluator would now handle dispatching all 3 types of evaluators, the precedence being goExecutor > binaryExecutor > podExecutor.
It should not be a failure if the configuration is valid, but the execution is not possible to start on that specific executor,
e.g.: we don't have a function stored in goExecutor, the binary is missing for binaryExecutor. In that case, we move onto the next executor.

Execution flow concept:

1. If we match the spec.image and one of spec.prefixes (if spec.prefixes is empty, assume the default image prefix):
   1. If the tag matches one of spec.goExecutorConfig.tags, then we try to match spec.goExecutorConfig.id (defaulted to spec.image if empty) to our available functions in goExecutor.
      If we can't find the appropriate function/method, we continue to the next step.
   2. If the tag matches one of spec.binaryExecutorConfig.tags, we try to run the specified binary with binaryExecutor.
      If the binary is missing, we continue to the next step.
   3. If the tag matches one of spec.podExecutorConfig.tags or spec.podExecutorConfig.tags is empty, we run the function in a pod with the specified configuration using podExecutor.
      If we cannot match any tag, we continue to the next step.
2. If all the previous steps failed to find a matching configuration, we run the function in a pod with the default configuration using podExecutor.

[mozesl-nokia]

Feedback and suggestions are welcome 😄

[nagygergo]

Would it make sense that for the serviceAccount/resources fields, instead of providing this as explicit parameters, this would be provided as part of a template configmap, similar to how it works today.
That would also allow for passing some additional sensitive data to the functions as mounted/envvar k8s secrets (kptdev/kpt#4337), and also support the current behavior of adding additional extensions to the running pods, like istio or prometheus annotations.

[mozesl-nokia]

@nagygergo It would certainly make sense, but it is both a lot of "fluff" and there are also a lot of things that the user can break IMO. Most use cases do not need "access" to the whole pod template, so I would prefer supporting them on a case-by-case basis instead.

If we really want to keep the flexibility, we could have some sort of configurable "field injection", basically running a bunch of search-replace functions on the pod manifest.

(Ofc. this is just my opinion, if we want to have full pod template support, that is very possible)

[thc1006]

Thanks for this design, @mozesl-nokia. The FunctionConfig CRD is a much cleaner approach than what we have now. I worked on PR nephio-project/nephio#425 for the parallel execution piece and reading through this PIP helped me see how that fits into the bigger picture.

One area I didn't see covered is the migration path from the current pod-cache-config ConfigMap. For existing deployments upgrading to a release with FunctionConfig, is the plan to support both during a transition period, or a clean cutover? If it helps, I can draft a migration guide based on
the current ConfigMap schema.

I'd also be happy to contribute test coverage for the FunctionConfig controller once implementation starts.

[mozesl-nokia]

@thc1006 Good point, I think the current format is simple enough that we can have a cutover. Although, if we merge the parallel execution changes first, it will be a little more nuanced, but the formats should then be similar. Nevertheless, it would be nice to have a short migration guide.

[thc1006]

Thanks @mozesl-nokia! Yeah clean cutover sounds good to me.

btw I've synced with @dgyorgy-nokia, we'll go with PR porch#336 as the base for parallel execution. I'll help with test coverage and review there.

For migration guide I'm thinking:

• mapping from current pod-cache-config ConfigMap to new FunctionConfig CR
• the new parallel execution params (maxParallelExecutions, preferredMaxQueueLength)
• any behavior differences

Will open a docs PR when draft is ready!