[exporter/elasticsearch] Clarify and decide retry.max_requests behavior

[carsonip]

Component(s)

exporter/elasticsearch

What happened?

Description

Discovered while working on #32323

Docs (code comments, README) do not clearly reflect the meaning of retry.max_requests, as in practice it counts the initial publishing attempt as well. Therefore, max_requests=1 means retry is disabled, which is a little counter intuitive.

We should either fix the documentation to align with implementation, or fix the implementation to make it more intuitive. The issue with fixing the implementation is that it is a breaking change for existing users.

Either way, docs need to be updated such that it is clear that the same retry value is used for both HTTP retries and per-document retries, and as a result the total attempts can be greater than max_requests.

Steps to Reproduce

set retry.max_requests=1, get a per-document 429 from ES

Expected Result

the document is retried once

Actual Result

the document is not retried

Collector version

0.97.0

Environment information

No response

OpenTelemetry Collector configuration

No response

Log output

No response

Additional context

No response

[github-actions]

Pinging code owners:

• exporter/elasticsearch: @JaredTan95 @ycombinator

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[JaredTan95]

Nice catch. Would you be interested in trying to clarify it?

[carsonip]

The scope of this issue is larger than a documentation fix.

Ideally I would like to either

• modify the existing retry.max_requests behavior, which is a "breaking change" if existing users rely on the behavior. But if it is seen as a bug given the unintuitive behavior and inconsistency with docs, then it can be called a bugfix.
• (the complete solution) deprecate retry.max_requests and introduce e.g. retry.max_retries with a more intuitive behavior

Either way, docs will need to be updated.

Would like the community's and the maintainers' @JaredTan95 @ycombinator views on this

[ycombinator]

(the complete solution) deprecate retry.max_requests and introduce e.g. retry.max_retries with a more intuitive behavior

I'm in favor of this option. @JaredTan95 WDYT?