OpenTelemetry Collector CRD (spec.config) changes not reflected in cluster — potential schema/diffing implications with GitOps tools?

[Iamkishore486]

Hi all,

I’m working with the OpenTelemetry Operator and managing the OpenTelemetryCollector resource via GitOps (Argo CD + Helm).

I’m observing unexpected behaviour where updates to nested fields under spec.config are not being reflected in the cluster, even though the rendered manifests clearly include the changes.

Problem Description

I updated a nested configuration value inside the collector spec:

spec:
  config:
    processors:
      memory_limiter:
        check_interval: 1s
        limit_percentage: 77

Observed Behavior

• Rendered manifests (Helm / Argo CD) show the updated value
• However, the live resource in the cluster still shows the old value:
  kubectl get opentelemetrycollector <name> -o yaml
• The update is not applied unless forced

Argo CD behavior:

- argocd app manifests <app-name> → shows updated value
- argocd app diff <app-name> → no output
- Application remains in Synced state
- No sync is triggered

Additional Observations

• Forcing a sync does apply the change
• Adding/changing annotations also triggers the update
• No errors are observed in the operator or controller logs

Important Clarifications

• spec.config is defined as structured YAML (not a multiline string)
• No GitOps-specific diff ignore rules are configured
• No validation or schema errors are reported

Questions

1. Is spec.config intentionally treated as an opaque field in the CRD schema?
2. Are there known limitations around:
3. Updating nested fields inside spec.config
4. Change detection for deeply nested configurations
5. Does the OpenTelemetry Operator:
6. Perform deep reconciliation on spec.config
7. Or rely on top-level changes to trigger updates?
8. Are there recommended patterns for ensuring updates are consistently applied, such as:
9. Using annotations to trigger rollout
10. Structuring config differently
11. Avoiding certain nested update patterns

Why This Matters

In GitOps-driven environments:

• Changes to nested config fields should ideally be:
• Detectable
• Reconciled automatically

Current behavior leads to:

• Drift between desired and actual state
• Manual intervention (force sync)

Thanks in advance for the support

[Cloud-Architect-Emma]

Hi @Iamkishore486! This is a known behaviour with how the OpenTelemetry Operator
handles spec.config.

The root cause is that spec.config in the OpenTelemetryCollector CRD is
stored as a raw/unstructured field (effectively treated as an opaque blob by
some versions of the operator). This means that when Argo CD performs its diff,
it may not detect nested field changes because the CRD schema marks spec.config
as x-kubernetes-preserve-unknown-fields: true, which causes Kubernetes (and
tools like Argo CD) to skip structural diffing of nested fields.

Why force-sync works: It bypasses the diff check and applies the full manifest
regardless.

Recommended workarounds:

1. Use an annotation to trigger reconciliation : add a rollout-timestamp
   annotation that you bump on every change:

   metadata:
     annotations:
       rollout-timestamp: "2026-05-10T10:00:00Z"

2. Use Argo CD's ignoreDifferences carefully : but in reverse: configure
   Argo CD to always apply (syncPolicy: automated with selfHeal: true).

3. Track this upstream issue : the OpenTelemetry Operator team is aware of
   this class of problem. You may want to file or follow an issue at
   https://github.com/open-telemetry/opentelemetry-operator/issues

The correct long-term fix would be for the operator's CRD to use a more
structured schema for spec.config so that GitOps tools can diff it properly.
In the meantime, the annotation bump approach is the most reliable workaround.

Hope this helps!

[Iamkishore486]

Hi @Cloud-Architect-Emma Thank you for the response. I performed some additional testing based on your observations and noticed the following behaviour:

When changes are made at the spec.config level, Argo CD is able to detect the differences and successfully sync the resource.
However, when changes are made deeper within the configuration, such as under spec.config.processors, spec.config.exporters, or similar nested fields, those changes appear to be ignored by Argo CD.

While reviewing the CRD, I also observed that the following fields are configured with x-kubernetes-preserve-unknown-fields: true.

Before I proceed with the suggested workaround recommendations, I would appreciate a clearer explanation of why changes at the spec.config level are still being detected correctly, even though this path/field itself also has x-kubernetes-preserve-unknown-fields: true configured.

spec.config
spec.config.connectors
spec.config.exporters
spec.config.extensions
spec.config.processors
spec.config.receivers
spec.config.service.pipelines
spec.config.service.telemetry
spec.targetAllocator.prometheusCR.scrapeClasses