Support configuring "target reload interval" via CRD

[bohub12]

Component(s)

target allocator, collector

Describe the issue you're reporting

Background

• Metric loss occurs after target re-allocation #4090 (comment)

Currently, the OpenTelemetry Operator does not expose a way to configure the "target reload interval" for the collector via the Custom Resource Definition (CRD). This value is useful for controlling how frequently the collector reloads its target list, especially in dynamic environments.

I’m planning to work on adding a new field to the CRD to allow configuring "target reload interval". This change will improve flexibility and simplify configuration for users.

I'm thinking of naming the new field CollectorTargetReloadInterval, but I'm open to suggestions if there's a more appropriate or consistent naming convention you'd recommend.

Please let me know if there are any concerns or guidelines I should follow before starting the implementation.

Thanks!

[jaronoff97]

@bohub12 you should be able to configure the target reload interval as you would any other field in the collector's configuration.

receivers:
  prometheus:
    target_allocator:
      endpoint: http://my-targetallocator-service
      interval: 30s # << Change this duration to your liking
      collector_id: collector-1

Is this not doing anything today? Or do you want to change the interval for the automatically generated target allocator configuration?

[bohub12]

Is this not doing anything today? Or do you want to change the interval for the automatically generated target allocator configuration?

@jaronoff97, In the current implementation, I don’t see a way to configure the interval field for the target allocator via the CRD. So, That’s why I’m proposing to expose this as a configurable field.

I'm planning to proceed with the field name CollectorTargetReloadInterval. If you have any suggestions or prefer a different naming convention, please feel free to let me know!

[jaronoff97]

@bohub12 Can you share the configuration you have today so I can understand the problem first? The config I shared before should available like this:

apiVersion: opentelemetry.io/v1beta1
kind: OpenTelemetryCollector
metadata:
  name: collector-with-ta
spec:
  mode: statefulset
  config:
    receivers:
      prometheus:
        target_allocator:
          endpoint: http://my-targetallocator-service
          interval: 30s # << Change this duration to your liking
          collector_id: collector-1
        config:
          scrape_configs:
          - job_name: 'otel-collector'
            scrape_interval: 10s
            static_configs:
            - targets: [ '0.0.0.0:8888' ]
    exporters:
      debug: {}

    service:
      pipelines:
        metrics:
          receivers: [prometheus]
          exporters: [debug]
---
apiVersion: opentelemetry.io/v1alpha1
kind: TargetAllocator
metadata:
  name: ta
spec:

[bohub12]

@jaronoff97, This is not about a problem with how the target reload interval currently works.

As mentioned in the “Background” section of this issue, I experienced a problem where some metrics were lost when collector pods scaled out or in. According to the response I received, the target allocator waits until a collector pod becomes healthy, while the Prometheus receiver attempts to scrape targets immediately. As a result, the target list may temporarily be returned as empty.

To avoid this problem, I was advised to either adjust the target reload interval or modify the Prometheus receiver. Since the current CRD spec does not allow configuring the target reload interval, I created this issue to propose making it configurable.

Are you suggesting that it might be sufficient to modify the target reload interval directly in the Prometheus receiver configuration, without adding a new field to the CRD spec?

[jaronoff97]

This depends on how you are installing the target allocator today. I believe we would need to add a field if you are using the automatic prometheus receiver configuration the operator provides. In this case, because you are installing the CRD separately, we would need the field. If you wanted to configure the target allocator portion of the collector's configuration yourself, you wouldn't need the field. All that to say, please continue with the change, I think it makes sense.