[WIP] Scaling with KEDA Kafka scaler

[aliok]

Summary

Adds KEDA Kafka consumer-lag scaling for the keda deployer, closes #3934.

• New scale.keda.triggers sub-key on func.yaml, supporting http, kafka, and cron trigger types. deployer: keda now requires at least one trigger to be declared explicitly (existing KEDA functions are migrated to an explicit http trigger to preserve current behavior).
• New scale.kpa sub-key for Knative-specific autoscaling config (metric/target/utilization), kept alongside the existing flat fields for backwards compatibility with older CLI versions.
• The keda deployer creates a ScaledObject + TriggerAuthentication (via the dynamic/unstructured client — no new dependency on kedacore/keda) when a kafka trigger is present, resolving SASL/TLS credentials from run.kafka and the function's secret-backed volumes.
• Fixed the raw deployer's readiness wait, which previously errored (could not find current pod-template-hash) on a deployment with 0 desired replicas — needed for scale.min: 0 to work with Kafka-triggered scale-to-zero.

func.yaml: old vs new structure

Before (implicit HTTP trigger, flat KPA fields only):

deployer: keda
options:
  scale:
    min: 1
    max: 10
    metric: concurrency
    target: 200
    utilization: 80

After — keda deployer now requires explicit triggers. HTTP-only (equivalent to the old implicit behavior):

deployer: keda
options:
  scale:
    min: 1
    max: 10
    keda:
      triggers:
        - type: http

After — Kafka consumer-lag trigger:

deployer: keda
options:
  scale:
    min: 0
    max: 10
    keda:
      triggers:
        - type: kafka
          lagThreshold: 5
          activationLagThreshold: 0
run:
  kafka:
    brokers: "my-cluster-kafka-bootstrap.kafka.svc.cluster.local:9093"
    topic: "my-topic"
    consumerGroup: "my-function-group"
    securityProtocol: "SASL_SSL"
    tls:
      caCert: "/etc/kafka/ca/ca.crt"
    sasl:
      mechanism: "SCRAM-SHA-512"
      user: "my-kafka-user"
      password: "{{ secret:my-kafka-user:password }}"
  volumes:
    - secret: my-cluster-cluster-ca-cert
      path: /etc/kafka/ca

After — knative deployer, explicit KPA sub-key (old flat metric/target/utilization fields still work and are migrated into kpa automatically on deploy):

deployer: knative
options:
  scale:
    min: 1
    max: 10
    kpa:
      metric: concurrency
      target: 200
      utilization: 80

A specVersion migration handles both transitions automatically: an old keda function with no triggers gets keda.triggers: [{type: http}] added, and an old knative function's flat KPA fields get copied into scale.kpa (the flat fields are left in place too, so older CLI versions can keep reading them).

Known limitations (tracked separately)

• A Deployment can only be owned by one ScaledObject, so http and kafka triggers cannot currently be combined on the same function — the HTTP trigger's HTTPScaledObject creates its own internal ScaledObject. Tracked in Migrate KEDA HTTP scaling from HTTPScaledObject to ScaledObject with external trigger #4043 (migrate the HTTP path to a ScaledObject with an external trigger, so triggers of any kind can be combined).
• Several KEDA/HPA tuning knobs (pollingInterval, cooldownPeriod, HTTP targetValue, etc.) are currently hardcoded rather than configurable via func.yaml. Tracked in Expose KEDA ScaledObject and HTTP scaling configuration options #4044.

Testing

• New unit tests: trigger resolution, secret-ref parsing, volume-path lookup, TriggerAuthentication/ScaledObject construction, SASL mechanism mapping, migration behavior, and cross-field validation.
• New integration test (TestInt_KafkaScaling, -tags integration): deploys a function with a Kafka-only trigger against a real cluster and verifies the ScaledObject/TriggerAuthentication are created with the expected spec and cleaned up on remove.
• Manually verified all {raw, knative, keda} x {no kafka, with kafka} deployer combinations end-to-end on Kind, including watching KEDA actually scale a Kafka consumer from 0 to N replicas under load, and the specVersion migration path from an old flat-scale func.yaml. Walkthroughs and captured resources for each scenario are under docs/testing-deployments/.

Docs

• docs/reference/func_yaml.md: documented deployer, run.kafka, scale.keda, scale.kpa.
• docs/reference/func_deploy.md / --deployer flag help: updated to mention KEDA's Kafka/cron triggers, not just HTTP.

[knative-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: aliok
Once this PR has been reviewed and has the lgtm label, please assign lkingland for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[knative-prow]

@aliok: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
unit-tests_func_main	aa0649a	link	true	/test unit-tests

Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.