Tracking issue for confmap.strictlyTypedInput feature gate

[mx-psi]

This issue describes the confmap.strictlyTypedInput feature gate, intended to address #8565 and #9532.

What is changing?

1. Configurations relying on the implicit type casting behaviors listed on [confmap] Stricter types on configuration resolution #9532 will start to fail.
2. Configurations using URI expansion (i.e. field: ${env:ENV}) for string-typed fields will use the value passed in ENV verbatim without intermediate type casting.

What does it mean to me?

Starting on v0.105.0, you may have to adapt your configuration to use the correct type on some fields. This should be a straightforward. You are able to skip the error and update your configuration at your own pace by disabling the confmap.strictlyTypedInput feature gate.

Please reply below if:

• You are unsure what an error means or how to update your configuration
• An error tells you that you can disable the feature gate to skip it, but this does not work.

Changelog

Collector version	Change	PR
v0.103.0	Introduce feature gate as alpha	#10400
v0.105.0	Promote feature gate to beta (enabled by default)	#10554
TBD	Promote feature gate to stable (can't be disabled)
TBD	Remove feature gate (errors out when referenced)

[galecore]

I observe a breaking change with this newly introduced behavior.

After the feature gate was turned on, grafana_cloud integrations via basic_auth extension with confmap/provider/envprovider now report errors like:

'client_auth.username' expected type 'string', got unconvertible type 'int', value: '12345'

As grafana cloud usernames are often integers, seems like users running this setup with docker without hardcoding credentials might experience unexpected errors.

One of the workarounds I see could be using bearertokenauth and providing token as ${env:GRAFANA_CLOUD_USERNAME}:${env:GRAFANA_CLOUD_PASSWORD}.
Are there any other workarounds? Is there a correct way of providing integer-based values as strings via environment?

Here is a minimal example which allows reproducing the error:

docker-compose.yaml

version: '3.8'
services:
  otel-collector:
    image: otel/opentelemetry-collector-contrib:latest
    command: ["--config=/etc/otel-collector-config.yaml"]
    volumes:
      - ./otel-collector-config.yaml:/etc/otel-collector-config.yaml
    env_file:
      - .secrets

.secrets

GRAFANA_CLOUD_USERNAME="12345"
GRAFANA_CLOUD_PASSWORD="secret"
GRAFANA_CLOUD_OTLP_ENDPOINT="https://otlp-gateway-prod-eu-west-0.grafana.net/otlp"

otel-collector-config.yaml

extensions:
  basicauth/grafana_cloud:
    # https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/extension/basicauthextension
    client_auth:
      username: "${env:GRAFANA_CLOUD_USERNAME}"
      password: "${env:GRAFANA_CLOUD_PASSWORD}"

receivers:
  otlp:
    # https://github.com/open-telemetry/opentelemetry-collector/tree/main/receiver/otlpreceiver
    protocols:
      grpc:
        endpoint: "0.0.0.0:4317"
  hostmetrics:
    # Optional. Host Metrics Receiver added as an example of Infra Monitoring capabilities of the OpenTelemetry Collector
    # https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/hostmetricsreceiver
    scrapers:
      load:
      memory:

processors:
  batch:
  # https://github.com/open-telemetry/opentelemetry-collector/tree/main/processor/batchprocessor
  resourcedetection:
    # Enriches telemetry data with resource information from the host
    # https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/resourcedetectionprocessor
    detectors: ["env", "system"]
    override: false
  transform/add_resource_attributes_as_metric_attributes:
    # https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/transformprocessor
    error_mode: ignore
    metric_statements:
      - context: datapoint
        statements:
          - set(attributes["deployment.environment"], resource.attributes["deployment.environment"])
          - set(attributes["service.version"], resource.attributes["service.version"])

exporters:
  otlphttp/grafana_cloud:
    # https://github.com/open-telemetry/opentelemetry-collector/tree/main/exporter/otlpexporter
    endpoint: "${env:GRAFANA_CLOUD_OTLP_ENDPOINT}"
    auth:
      authenticator: basicauth/grafana_cloud

service:
  extensions: [basicauth/grafana_cloud]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [resourcedetection, batch]
      exporters: [otlphttp/grafana_cloud]
    metrics:
      receivers: [otlp, hostmetrics]
      processors: [
        resourcedetection,
        transform/add_resource_attributes_as_metric_attributes,
        batch,
      ]
      exporters: [otlphttp/grafana_cloud]
    logs:
      receivers: [otlp]
      processors: [resourcedetection, batch]
      exporters: [otlphttp/grafana_cloud]

[TylerHelmuth]

@galecore you're likely experiencing the same issue as #10659. I haven't been able to reproduce your use-case yet either.

[galecore]

Not sure if relevant, but would note that my setup for this is:

Operating System: Ubuntu 22.04.4 
Kernel: Linux 5.15.0-83-generic
Architecture: x86-64
Docker version: 24.0.7, build afdd53b
Docker Compose version: v2.20.2-desktop.1

An actual example that I'm running could be found here (just remove --feature-gates=-confmap.strictlyTypedInput) - https://github.com/galecore/telemetry-example/blob/master/examples/otel-collector/docker-compose.yml#L33

[mx-psi]

Is there a correct way of providing integer-based values as strings via environment?

#10618 will fix this, in the mean time you can revert the feature gate

[Kuckkuck]

Btw. I get the same error twice within this section:

    transform/containerd:
        log_statements:
          context: log
          statements:
            - merge_maps(cache,ExtractPatterns(body,"^(?P<time>[^Z]+Z) (?P<stream>stdout|stderr) (?P<logtag>[^\\s]*) ?(?P<log>.*)$"), "upsert") where body != nil
            - merge_maps(cache,ExtractPatterns(body,"^(?P<time>\\d+/\\d+/\\d+\\s+\\d+:\\d+\\d+) (?P<log>.*)$"), "upsert") where attributes["log_name"]!= "MeshAccessLog" and cache["log"]!= nil and not IsMap(cache["log"])
            - set(body,cache["log"]) where cache["log"] != nil
            - merge_maps(cache,ParseJSON(body), "upsert") where IsMap(body)
            - set(body,cache["message"]) where cache["message"] != nil
            - set(body,cache["msg"]) where cache["msg"] != nil
            - set(severity_text,cache["level"]) where cache["level"] != nil
            - set(severity_text,cache["severity"]) where cache["severity"] != nil
            - set(severity_number,SEVERITY_NUMBER_INFO) where cache["level"] == "INFO"
            - set(severity_number,SEVERITY_NUMBER_INFO) where cache["severity"] == "info"
            - set(attributes["loggerName"],cache["loggerName"]) where cache["loggerName"] != nil

Error: failed to get config: cannot unmarshal the configuration: 1 error(s) decoding:

* error decoding 'processors': error reading configuration for "transform/containerd": 1 error(s) decoding:

* 'log_statements': source data must be an array or slice, got map

Hint: Temporarily restore the previous behavior by disabling
      the `confmap.strictlyTypedInput` feature gate. More details at:
      https://github.com/open-telemetry/opentelemetry-collector/issues/10552
2024/07/25 14:02:24 collector server run finished with error: failed to get config: cannot unmarshal the configuration: 1 error(s) decoding:

* error decoding 'processors': error reading configuration for "transform/containerd": 1 error(s) decoding:

* 'log_statements': source data must be an array or slice, got map

Hint: Temporarily restore the previous behavior by disabling
      the `confmap.strictlyTypedInput` feature gate. More details at:
      https://github.com/open-telemetry/opentelemetry-collector/issues/10552

[TylerHelmuth]

@Kuckkuck your configuration is wrong.

transform/containerd:
  log_statements:
    - context: log
      statements:
        - ...

[mx-psi]

Btw. I get the same error twice within this section:

This is not specific to this error, it happens for all errors: you get the error once in the logs and another in stdout.