feat: Implement ResourceIndexPolicy API and Controller Manager Deployment

[JoseSzycho]

Refactor Controller Manager, Implement ResourceIndexPolicy API with Webhooks, and Add CI Workflow

🚀 Summary

This PR deploys the controller manager deployment to align with project standards (milo), implements the ResourceIndexPolicy API with robust webhook validations, and establishes a GitHub Actions workflow for environment validation.

🛠️ Key Changes

1. Controller Manager Implementation

• Deployment Architecture: Implemented the controller manager using a modular Kustomize structure (config/controller-manager), establishing the base and overlays/core-control-plane pattern for environment-specific configurations.
• Runtime Configuration: Configured the deployment with production standards, including environment variable support (METRICS_BIND_ADDRESS, LOG_LEVEL) and strict security contexts (non-root execution, read-only filesystem).
• Identity & High Availability: Established the complete RBAC infrastructure, including ServiceAccounts and RoleBindings, to enable secure operation and leader election capability via coordination.k8s.io leases.

2. ResourceIndexPolicy API & Admission Controllers

• API Implementation: Added the ResourceIndexPolicy CRD and controller logic.
• Webhook Validation: Implemented a API validations that enforces:
  • CEL Expressions: Validates syntax and restricts operators to a safe subset (disallowing arithmetic/macros unless explicitly enabled).
  • JSONPath: Ensures field paths are valid.
• TLS & Cert Manager: Configured cert-manager CSI driver for automatic webhook certificate provisioning and CA bundle injection.

3. End-to-End Test Infrastructure & CI Workflow

• GitHub Actions CI: Established a new CI pipeline (test-environment-validation.yaml) that automates cluster setup and validation on every Pull Request.
• CI-Optimized Deployment: Created a ci:setup task and a dedicated ci Kustomize overlay to deploy a lightweight version of the search stack, explicitly excluding the observability stack (Prometheus/VictoriaMetrics) to optimize runner performance and avoid missing CRD errors.
• Portable E2E Tests: Refactored chainsaw tests (Meilisearch health and Webhook validation) to use portable shell constructs and correctly handle expected failure assertions, ensuring stability across local and CI execution environments.
• Infrastructure Tuning: Optimized NATS resource requests/limits and JetStream memory settings to provide a smaller footprint for CI environments.

[JoseSzycho]

@scotwells The documentations speaks on term of a [multicluster-runtime] for the resourceIndexer.

search/docs/components/resource-indexer.md

Line 266 in 587e955

The indexer uses the [multicluster-runtime] provider pattern to discover

I think that for the work of this PR it was not necessary to add a complexity layer of the multi cluster-runtime, for the admission controllers and the simple reconciler I implemented.

[scotwells]

@JoseSzycho you should be able to simplify this by removing the webhook and using an admission controller since we will be using the aggregated apiserver framework. That means the apiserver will do its own validation of the request without having to register a webhook with the core apiserver.

You also won't be able to use CRDs here because the core apiserver is going to proxy all traffic for the API group to this server instead of handling CRD requests. The search APIserver will need to connect to an etcd cluster for storage and use the standard etcd storage layer.

I think that for the work of this PR it was not necessary to add a complexity layer of the multi cluster-runtime, for the admission controllers and the simple reconciler I implemented.

@JoseSzycho correct, the controller manager component will not need to be integrated with the multi-cluster runtime library since it's only responsibility is managing the lifecycle of the index policy.

The resource indexer component (separate from the controller manager) is what will need to integrate with the multi-cluster runtime library because it will index resources from project control planes.

[JoseSzycho]

@scotwells I went into the Admission Control docs, and thought that implementing the webhooks was the intend approach. Sorry for the misunderstanding.

Just to confirm the plan: I will remove the External Controller + CRD + Webhook setup. Instead, I'll enable Etcd storage in the Search APIServer options and move the validation logic directly into the REST Strategy's Validate method (or an internal Admission Plugin if needed). The existing reconciliation logic in the controller manager will be updated to watch the policy.search.miloapis.com native API resource instead of the CRD.

Would that be correct?

[scotwells]

@JoseSzycho you'd keep the external controller for reconciling index policies. But correct, you'd remove the CRDs and replace them with an etcd storage backend and then use an admission plugin to replace the webhook.

I'd recommend looking at Milo's quota admission plugin for an example of creating an admission plugin. I don't see many resources available online for it.

[JoseSzycho]

@scotwells thanks the prompt review.

Will start working in fixing this.

I’m updating the enhancement issue with the delay I caused.

[JoseSzycho]

@scotwells everything should be okay now. Can you review the PR again?

[JoseSzycho]

@scotwells Thanks for the great feedback. All commits have been pushed.

[scotwells]

Good stuff! Nothing that can't be done in a follow up PR.

[scotwells]

Why was this changed? This should be creating the CA certificate that will be used to sign TLS certs.