update error code if ES dial connection error

[rubvs]

• If ES is down, we want to return an error unavailable code, along with a user facing error: retryable server error.
• If elasticsearchErr is not ok, it implies the ES instance could not be reached, and therefore does not return a specific ES error, but rather a code.Internal error. The error happens on the TCP connection level.

Manual Test

# Console 1: Spin up a cluster
> tilt up

# Console 2: Make ES unavailable
> kubectl scale statefulset elasticsearch-es-default --replicas=0

# Console 3: Ensure a retryable error is returned to the user when trying to auth to an unavailable ES.
> TEST_COUNT=10 make run-otelbench mode=local

2025-10-05T15:56:21.572Z  error  internal/queue_sender.go:51  Exporting failed. Dropping data.
{
  "resource": {
    "service.instance.id": "645a4319-43c4-4f8b-b97b-27f5adf6dc0c",
    "service.name": "/ko-app/loadgen",
    "service.version": "0.0.1"
  },
  "otelcol.component.id": "otlp",
  "otelcol.component.kind": "exporter",
  "otelcol.signal": "logs",
  "error": "interrupted due to shutdown: rpc error: code = Unavailable desc = rpc error: code = Unavailable desc = retryable server error \"o1voipUB6v8S06vJAX5J\": an error happened during the HasPrivileges query execution: dial tcp 10.96.217.236:9200: connect: connection refused",
  "dropped_items": 325
}

[carsonip]

Can we get a unit test for the new branch that you're adding?

[carsonip]

If we are talking about serverless ES that is unavailable, we might get 502 from proxy instead. Have you considered handling 502 and surfacing that as a retryable error to otel client?

[isaacaflores2]

Changes look good. I still think we need to add a separate test case when we get a non elasticsearch err like @carson mentioned here

[rubvs]

@carsonip any non-elasticsearch error will return a retryable error. So if the proxy - sitting between ingest and ES - returns any error, a retryable error should be returned.

@isaacaflores2 I've added a basic unit test to mock the proxy returning a 502.

[isaacaflores2]

Thanks for the updates!

[carsonip]

1 nit and a non-blocking question on the e2e behavior in the collector, whether the error is sanitized enough for user consumption.

[rubvs]

How did this merge without passing CI first. Creating a fix quickly.