[otelcol] Allow confmap to write logs using configured logger #10008

TylerHelmuth · 2024-04-19T20:41:42Z

Description

This PR allows confmap to create logs, and then actually writes the logs out after the collector's real logger is instantiated.

Example of the logger in action

receivers:
  nop:

exporters:
  otlphttp:
    endpoint: http://0.0.0.0:4317
    headers:
      # Not set
      x-test: ${env:TEMP3}
  debug:
    # set to "detailed"
    verbosity: $TEMP

service:
  pipelines:
    traces:
      receivers:
        - nop
      exporters:
        - debug

Alternative to #10007

Link to tracking issue

Related to #9162
Related to #5615

Testing

If we like this approach I'll add tests

TylerHelmuth · 2024-04-19T20:42:06Z

A downside to this approach is that the logs come in a weird order. This is highlighted in the description's screenshot.

evan-bradley · 2024-04-19T21:09:28Z

@TylerHelmuth I think this is a more scalable solution given that providers may want to do additional logging in the future (e.g. an HTTP provider that polls an endpoint can't reach the endpoint during runtime) and it would be nice if these logging settings are in line with the rest of the Collector.

Regarding the log ordering, this highlights the fact that, in my opinion, otelcol.Collector and service have a lot of weird overlap. We could realistically do various odd things to fix the ordering (e.g. passing the logs into the service), but I don't think it's such a big deal.

mx-psi · 2024-04-22T07:52:42Z

A downside to this approach is that the logs come in a weird order. This is highlighted in the description's screenshot.

I don't feel like this is a big deal. As part of #4970 we could explore moving the logger initialization outside of service, that would give us more flexibility with how to do this.

Note that service.Logger() has this comment:

This is a temporary API that may be removed soon after investigating how the collector should record different events.

I am fine with using it here for now, but I guess we'll have to work on refactoring this part at some point (it does feel a bit weird that that method exists).

cc @ankitpatel96, this is a rework of #9908

TylerHelmuth · 2024-04-22T15:31:12Z

Another potential downside I need to investigate: if there is an error during confmap resolution will the logs still be written

TylerHelmuth · 2024-04-22T15:54:06Z

if there is an error during confmap resolution will the logs still be written

The answer is no, the logs will not be written. So if we wanted to add debug logs to help users while troubleshooting configuration resolution that wouldn't work - they'd need to entirely depend on the errors returned.

With this solution any logs from confmap will only be written on a successful collector configuration resolution AND service creation.

evan-bradley · 2024-04-23T00:07:00Z

With this solution any logs from confmap will only be written on a successful collector configuration resolution AND service creation.

Could we circumvent this by creating an "error only" logger in otelcol.Collector that replays the logs from the observer with a configuration similar to the one in #10007?

mx-psi · 2024-04-23T09:27:52Z

The answer is no, the logs will not be written. So if we wanted to add debug logs to help users while troubleshooting configuration resolution that wouldn't work - they'd need to entirely depend on the errors returned.

The alternative (without changing public API) would be to create the logger twice: once before creating the service to log these, and once inside the server.

I am okay with dealing with this problem later if we don't like that though, this is a net improvement IMO and we could merge after adding some tests

otelcol/collector_test.go

codecov · 2024-04-23T17:31:31Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 91.56%. Comparing base (31528ce) to head (8e9bce4).

Additional details and impacted files

@@           Coverage Diff           @@
##             main   #10008   +/-   ##
=======================================
  Coverage   91.56%   91.56%           
=======================================
  Files         360      360           
  Lines       16698    16703    +5     
=======================================
+ Hits        15289    15294    +5     
  Misses       1073     1073           
  Partials      336      336

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

otelcol/collector_test.go

Co-authored-by: Evan Bradley <11745660+evan-bradley@users.noreply.github.com>

mx-psi · 2024-04-29T09:55:20Z

otelcol/collector.go

@@ -202,6 +207,12 @@ func (col *Collector) setupConfigurationComponents(ctx context.Context) error {
 		return err
 	}

+	if col.ol != nil {
+		for _, log := range col.ol.All() {
+			col.service.Logger().Log(log.Level, log.Message)


I am wondering if we should use the core directly here via its Check and Write methods. That way we would preserve the exact logged message, including the timestamp and stack trace. I think it can be done, but it's not trivial since there is some logic we would have to copy from here.

Do you think it's worth the effort?

Would having out-of-order timestamps be a problem? I dont think stacktraces is a valid scenario bc currently if the confmap errors its logs will not get written.

I dont think stacktraces is a valid scenario bc currently if the confmap errors its logs will not get written.

I don't get what you mean by this. What I meant is that instead of having otelcol@v0.98.0/collector:XX we would have the actual line where this log came from.

Looking at #10056 this is actually necessary since we are not logging the fields (if you compare the screenshots, with this PR you don't know what env var is unset)

I really don't like the idea of duplicating functionality that exists in zapcore already

Also I can fix the field issue by passing in log.Context... as the last param.

I really don't like the idea of duplicating functionality that exists in zapcore already

To be clear: it does not exist in zapcore, it exists in *zap.Logger

otelcol/collector.go

codeboten · 2024-04-29T19:17:38Z

otelcol/collector.go

@@ -15,6 +15,7 @@ import (

 	"go.uber.org/multierr"
 	"go.uber.org/zap"
+	"go.uber.org/zap/zaptest/observer"


bringing in a test import here seems a bit strange. from the observer docs:

It's useful for applications that want to unit test their log output without tying their tests to a particular output encoding.

I worry that this package may change in the future causing issue, but maybe it's not a problem?

We're certainly using the package in a use case I think they didn't intend. I still think the simplicity of #10007 is most appropriate as it is the collector making logging decisions before being told how to log. It keeps the logs in proper order as well

I worry that this package may change in the future causing issue, but maybe it's not a problem?

I don't think that should be a problem since (i) the fact that we use this package is an implementation detail and (ii) even if the package disappeared we could rebuild it ourselves (it does not depend on any internal bits of zap)

TylerHelmuth · 2024-04-29T19:32:25Z

I think this is a more scalable solution given that providers may want to do additional logging in the future (e.g. an HTTP provider that polls an endpoint can't reach the endpoint during runtime) and it would be nice if these logging settings are in line with the rest of the Collector.

@evan-bradley to address this concern in #10007 could the confmap provider's logger be updated with a new logger once the actual one is instantiated?

evan-bradley · 2024-04-29T21:00:08Z

@evan-bradley to address this concern in #10007 could the confmap provider's logger be updated with a new logger once the actual one is instantiated?

I would prefer not to mess around with the logger instance if we can get away with it. Additionally, that may be a challenge: once we pass the logger in and the providers store the reference, I don't think we have any real ability to update it.

If we want to avoid using the observer, I would say we should do one of these:

Refactor the otelcol<->service interface to move config resolution responsibilities to the service. This needs a lot of discussion and may not be the right move, but I can bring this one up at the next SIG if there aren't any major objections to it.
Have otelcol.Collector instantiate the logger and pass it to the service.

In both of these cases we would ideally find a way to delay actually outputting any logs until we've resolved a log level. Otherwise we would just have to use a default log level until we get the real one.

Actually, looking at this a second time, I think we may have to move away from the observer regardless if we want logs at runtime. It doesn't look like it has any methods that would support "streaming" the logs to a real logger.

evan-bradley · 2024-04-29T21:06:30Z

once we pass the logger in and the providers store the reference, I don't think we have any real ability to update it.

We could get around this by passing a wrapper struct that allows us to update the instance reference so long as providers access the logger only through the wrapper. It would be nice to see if we could solve this some other way first.

TylerHelmuth · 2024-04-29T21:18:55Z

@evan-bradley I don't think either option 1 or 2 will solve the problem of trying to have an the actual user-configured logger during config resolution since they don't solve the problem that the user-configured logger depends on config resolution.

TylerHelmuth · 2024-04-29T21:25:40Z

We could get around this by passing a wrapper struct that allows us to update the instance reference so long as providers access the logger only through the wrapper. It would be nice to see if we could solve this some other way first.

This would work and would be enforceable if ProviderSettings.Logger used that wrapper instead of *zap.Logger. I believe we'd also need a new method on ConfigProvider that would allow updating the logger, or we'd need to keep around a reference to the original wrapper passed into ProviderSettings that could be updated.

evan-bradley · 2024-04-29T21:48:04Z

@evan-bradley I don't think either option 1 or 2 will solve the problem of trying to have an the actual user-configured logger during config resolution since they don't solve the problem that the user-configured logger depends on config resolution.

Both of those options are intended to keep a single instance of the logger and just update it in-place after the config is resolved. I think you may be right that they don't solve the problem though, since it appears the zap APIs all return a new instance when you update a logger.

This would work and would be enforceable if ProviderSettings.Logger used that wrapper instead of *zap.Logger.

Providers could still store the *zap.Logger reference and just use that; any updates to the wrapper would then go unused. I think to make it fully enforceable we would need to fully wrap *zap.Logger and keep the instance reference hidden. Realistically we could also rely on documentation and just say to only store references to the wrapper since the instance may change.

TylerHelmuth · 2024-04-29T22:39:31Z

I think to make it fully enforceable we would need to fully wrap *zap.Logger and keep the instance reference hidden.

Agreed, ProviderSettings would need to look something like

// ProviderSettings are the settings to initialize a Provider.

type OurCustomLoggingInterface {
  # expose the zap.Logger functions here and the ability to update the underlying zap logger.
}

type ProviderSettings struct {
	Logger OurCustomLoggingInterface
}

I don't like it.

mx-psi · 2024-04-30T08:53:40Z

Re: how to update the logger at runtime. I don't get why we need to wrap *zap.Logger into something else. Why not just build a custom zapcore.Core that we can update at runtime? It's an interface already so we can do whatever we want there, including replacing its internals after the fact, can't we?

TylerHelmuth · 2024-04-30T15:12:01Z

Why not just build a custom zapcore.Core that we can update at runtime?

I can investigate this idea

TylerHelmuth · 2024-04-30T18:11:12Z

Tried out the idea here: #10056

mx-psi · 2024-04-30T18:56:47Z

otelcol/collector.go

@@ -202,6 +207,12 @@ func (col *Collector) setupConfigurationComponents(ctx context.Context) error {
 		return err
 	}

+	if col.ol != nil {
+		for _, log := range col.ol.All() {
+			col.service.Logger().Log(log.Level, log.Message)


Looking at #10056 this is actually necessary since we are not logging the fields (if you compare the screenshots, with this PR you don't know what env var is unset)

TylerHelmuth · 2024-05-02T16:44:00Z

Closing this for now in favor of #10056

#### Description Provides a logger to confmap that buffers logs in memory until the primary logger can be used. Once the primary logger exists, places that used the original logger are given the updated Core. If an error occurs that would shut down the collector before the primary logger could be created, the logs are written to stdout/err using a fallback logger. Alternative to #10008 I've pushed the testing I did to show how the logger successfully updates. Before config resolution the debug log in confmap is not printed, but afterwards it is. test config: ```yaml receivers: nop: exporters: otlphttp: endpoint: http://0.0.0.0:4317 headers: # Not set x-test: ${env:TEMP3} debug: # set to "detailed" verbosity: $TEMP service: telemetry: logs: level: debug pipelines: traces: receivers: - nop exporters: - debug ``` ![image](https://github.com/open-telemetry/opentelemetry-collector/assets/12352919/6a17993f-1f97-4c54-9165-5c34dd58d108)  #### Link to tracking issue Related to #9162 Related to #5615  #### Testing If we like this approach I'll add tests  #### Documentation --------- Co-authored-by: Dan Jaglowski <jaglows3@gmail.com> Co-authored-by: Pablo Baeyens <pbaeyens31+github@gmail.com>

…ry#10056)  #### Description Provides a logger to confmap that buffers logs in memory until the primary logger can be used. Once the primary logger exists, places that used the original logger are given the updated Core. If an error occurs that would shut down the collector before the primary logger could be created, the logs are written to stdout/err using a fallback logger. Alternative to open-telemetry#10008 I've pushed the testing I did to show how the logger successfully updates. Before config resolution the debug log in confmap is not printed, but afterwards it is. test config: ```yaml receivers: nop: exporters: otlphttp: endpoint: http://0.0.0.0:4317 headers: # Not set x-test: ${env:TEMP3} debug: # set to "detailed" verbosity: $TEMP service: telemetry: logs: level: debug pipelines: traces: receivers: - nop exporters: - debug ``` ![image](https://github.com/open-telemetry/opentelemetry-collector/assets/12352919/6a17993f-1f97-4c54-9165-5c34dd58d108)  #### Link to tracking issue Related to open-telemetry#9162 Related to open-telemetry#5615  #### Testing If we like this approach I'll add tests  #### Documentation --------- Co-authored-by: Dan Jaglowski <jaglows3@gmail.com> Co-authored-by: Pablo Baeyens <pbaeyens31+github@gmail.com>

Write logs once we have a logger

562baaf

TylerHelmuth requested a review from a team as a code owner April 19, 2024 20:41

TylerHelmuth requested a review from codeboten April 19, 2024 20:41

TylerHelmuth added the Skip Changelog PRs that do not require a CHANGELOG.md entry label Apr 19, 2024

TylerHelmuth mentioned this pull request Apr 19, 2024

[otelcol] Pass a real logger to ProviderSettings #10007

Closed

TylerHelmuth requested review from evan-bradley and mx-psi April 19, 2024 20:50

mx-psi approved these changes Apr 22, 2024

View reviewed changes

TylerHelmuth changed the title ~~[otelcol] Write logs once we have a logger~~ [otelcol] Allow confmap to write logs using configured logger Apr 22, 2024

Add a test

31b4ef9

TylerHelmuth commented Apr 23, 2024

View reviewed changes

otelcol/collector_test.go Outdated Show resolved Hide resolved

Use Greater assertion

d23b50a

TylerHelmuth added 3 commits April 25, 2024 11:30

Update test to use Provider that logs expected message

47b5f0c

Improve test

8f3211e

Merge branch 'main' into pass-logger-to-providersettings-save-logs

8f25cb3

evan-bradley approved these changes Apr 26, 2024

View reviewed changes

otelcol/collector_test.go Outdated Show resolved Hide resolved

Update otelcol/collector_test.go

8e9bce4

Co-authored-by: Evan Bradley <11745660+evan-bradley@users.noreply.github.com>

mx-psi reviewed Apr 29, 2024

View reviewed changes

codeboten reviewed Apr 29, 2024

View reviewed changes

TylerHelmuth mentioned this pull request Apr 30, 2024

[otelcol] Add a custom zapcore.Core for confmap logging #10056

Merged

mx-psi requested changes Apr 30, 2024

View reviewed changes

TylerHelmuth mentioned this pull request May 1, 2024

[otelcol] rfc for how to log during startup #10066

Merged

TylerHelmuth closed this May 2, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[otelcol] Allow confmap to write logs using configured logger #10008

[otelcol] Allow confmap to write logs using configured logger #10008

TylerHelmuth commented Apr 19, 2024 •

edited

TylerHelmuth commented Apr 19, 2024

evan-bradley commented Apr 19, 2024 •

edited

mx-psi commented Apr 22, 2024

TylerHelmuth commented Apr 22, 2024

TylerHelmuth commented Apr 22, 2024 •

edited

evan-bradley commented Apr 23, 2024

mx-psi commented Apr 23, 2024

codecov bot commented Apr 23, 2024 •

edited

mx-psi Apr 29, 2024

TylerHelmuth Apr 29, 2024

mx-psi Apr 30, 2024

mx-psi Apr 30, 2024

TylerHelmuth Apr 30, 2024

TylerHelmuth Apr 30, 2024

mx-psi May 2, 2024

codeboten Apr 29, 2024

TylerHelmuth Apr 29, 2024

mx-psi Apr 30, 2024

TylerHelmuth commented Apr 29, 2024

evan-bradley commented Apr 29, 2024 •

edited

evan-bradley commented Apr 29, 2024

TylerHelmuth commented Apr 29, 2024

TylerHelmuth commented Apr 29, 2024

evan-bradley commented Apr 29, 2024

TylerHelmuth commented Apr 29, 2024

mx-psi commented Apr 30, 2024

TylerHelmuth commented Apr 30, 2024

TylerHelmuth commented Apr 30, 2024

mx-psi Apr 30, 2024

TylerHelmuth commented May 2, 2024

[otelcol] Allow confmap to write logs using configured logger #10008

[otelcol] Allow confmap to write logs using configured logger #10008

Conversation

TylerHelmuth commented Apr 19, 2024 • edited

Description

Link to tracking issue

Testing

TylerHelmuth commented Apr 19, 2024

evan-bradley commented Apr 19, 2024 • edited

mx-psi commented Apr 22, 2024

TylerHelmuth commented Apr 22, 2024

TylerHelmuth commented Apr 22, 2024 • edited

evan-bradley commented Apr 23, 2024

mx-psi commented Apr 23, 2024

codecov bot commented Apr 23, 2024 • edited

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

TylerHelmuth commented Apr 29, 2024

evan-bradley commented Apr 29, 2024 • edited

evan-bradley commented Apr 29, 2024

TylerHelmuth commented Apr 29, 2024

TylerHelmuth commented Apr 29, 2024

evan-bradley commented Apr 29, 2024

TylerHelmuth commented Apr 29, 2024

mx-psi commented Apr 30, 2024

TylerHelmuth commented Apr 30, 2024

TylerHelmuth commented Apr 30, 2024

Choose a reason for hiding this comment

TylerHelmuth commented May 2, 2024

TylerHelmuth commented Apr 19, 2024 •

edited

evan-bradley commented Apr 19, 2024 •

edited

TylerHelmuth commented Apr 22, 2024 •

edited

codecov bot commented Apr 23, 2024 •

edited

evan-bradley commented Apr 29, 2024 •

edited