Resource detectors should not log at ERROR level when info not found

[cheempz]

Description of the bug

Per the resource SDK spec, failure to detect resource information must not be considered an error.

Currently when AWS and Container detectors are included, on startup they log ERROR level messages:

E, [2025-05-15T11:53:45.078373 #62040] ERROR -- : OpenTelemetry error: EC2 metadata service request failed - Failed to open TCP connection to 169.254.169.254:80 (No route to host - connect(2) for 169.254.169.254:80) - /.rbenv/versions/3.1.2/lib/ruby/3.1.0/socket.rb:1214:in `__connect_nonblock'
E, [2025-05-15T11:53:45.079090 #62040] ERROR -- : OpenTelemetry error: EC2 metadata service request failed - Failed to open TCP connection to 169.254.169.254:80 (Host is down - connect(2) for 169.254.169.254:80) - /.rbenv/versions/3.1.2/lib/ruby/3.1.0/socket.rb:1214:in `__connect_nonblock'
E, [2025-05-15T11:53:45.080134 #62040] ERROR -- : OpenTelemetry error: Container resource detector - /proc/self/mountinfo could not be read.
E, [2025-05-15T11:53:45.080195 #62040] ERROR -- : OpenTelemetry error: Container resource detector - /proc/self/cgroup could not be read.

Could these be DEBUG level logs instead?

Share details about your runtime

Operating system details: MacOS
RUBY_ENGINE: "ruby"
RUBY_VERSION: "3.1.2"
RUBY_DESCRIPTION: "ruby 3.1.2p20 (2022-04-12 revision 4491bb740a) [arm64-darwin24]"

[kaylareopelle]

Great call-out @cheempz. I missed that part of the spec and mistakenly recommended we use handle_error in these scenarios.

@yiyuan-he, can you update the detectors to address this?

[yiyuan-he]

Thanks @cheempz and @kaylareopelle - I'll raise a PR to change the AWS resource detectors to use debug logs for detection failure by end of week.

[arielvalentin]

I am reading the spec and trying to come to the same undrestanding that you all have around this.  

Resource detection logic is expected to complete quickly since this code will be run during application initialization. Errors should be handled as specified in the Error Handling principles.

I believe we got this right. Errors detecting resources should not result in errors trying to run the SDK or impacting the application in any way.

Note the failure to detect any resource information MUST NOT be considered an error, whereas an error that occurs during an attempt to detect resource information SHOULD be considered an error.

I do not know what the difference is between these use cases.

"MUST NOT be considered an error" - does that mean the error cannot be reported to the error handler and
"SHOULD be considered an error" does not?

If the resource detector isn't working are we saying we want folks to set their log levels to debug vs warn?

[cheempz]

I am reading the spec and trying to come to the same undrestanding that you all have around this.  

Resource detection logic is expected to complete quickly since this code will be run during application initialization. Errors should be handled as specified in the Error Handling principles.

I believe we got this right. Errors detecting resources should not result in errors trying to run the SDK or impacting the application in any way.

Note the failure to detect any resource information MUST NOT be considered an error, whereas an error that occurs during an attempt to detect resource information SHOULD be considered an error.

I do not know what the difference is between these use cases.

"MUST NOT be considered an error" - does that mean the error cannot be reported to the error handler and "SHOULD be considered an error" does not?

It is kind of ambiguous. My take is that an error level message should only be logged if the resource detector determined, for example, that it is actually running on EC2 but somehow failed to retrieve the instance ID.

If the resource detector isn't working are we saying we want folks to set their log levels to debug vs warn?

Yes, from a user perspective I'd find it annoying to get a lot of error log messages that end up being false alarm. But open to other log levels if people think debug is too low.

[arielvalentin]

I've written up some use cases that I think are relevant. What should be the behavior/log level that is used in each scenario?

A resource detector requires a remote API to discover relevant facts about a workload.

It fails due to a API authentication issue, e.g. bad credentials

If fails due to a authorization issue, e.g. it doesn't have permission to too resource details

It fails due network connectivity issues.

It fails because it's unable to interpret the response (e.g. incompatible API version)

It fails because the detector is irrelevant, e.g. GCP is activated in a AWS deployed app

[cheempz]

Hmm, to me it would be:

It fails because the detector is irrelevant, e.g. GCP is activated in a AWS deployed app

Not an error. I've put this one first because ideally the detector then shortcuts out of below.

It fails due to a API authentication issue, e.g. bad credentials

Error.

If fails due to a authorization issue, e.g. it doesn't have permission to too resource details

Not an error, maybe the application really shouldn't have permission to said resource detail.

It fails due network connectivity issues.

Error.

It fails because it's unable to interpret the response (e.g. incompatible API version)

Error.

Thanks for making us think more about this. General feeling is it may be up to the maintainer of the detector to strike that balance.