[otelcol] Add a custom zapcore.Core for confmap logging

[TylerHelmuth]

Description

Provides a logger to confmap that buffers logs in memory until the primary logger can be used. Once the primary logger exists, places that used the original logger are given the updated Core.

If an error occurs that would shut down the collector before the primary logger could be created, the logs are written to stdout/err using a fallback logger.

Alternative to #10008

I've pushed the testing I did to show how the logger successfully updates. Before config resolution the debug log in confmap is not printed, but afterwards it is.

test config:

receivers:
  nop:

exporters:
  otlphttp:
    endpoint: http://0.0.0.0:4317
    headers:
      # Not set
      x-test: ${env:TEMP3}
  debug:
    # set to "detailed"
    verbosity: $TEMP

service:
  telemetry:
    logs:
      level: debug
  pipelines:
    traces:
      receivers:
        - nop
      exporters:
        - debug

Link to tracking issue

Related to #9162
Related to #5615

Testing

If we like this approach I'll add tests

Documentation

[codecov]

Codecov Report

Attention: Patch coverage is 89.79592% with 10 lines in your changes are missing coverage. Please review.

Project coverage is 91.67%. Comparing base (c6b70a7) to head (c3b7488).
Report is 3 commits behind head on main.

Files	Patch %	Lines
otelcol/collector.go	72.22%	7 Missing and 3 partials ⚠️

Additional details and impacted files

@@           Coverage Diff            @@
##             main   #10056    +/-   ##
========================================
  Coverage   91.67%   91.67%            
========================================
  Files         362      367     +5     
  Lines       16754    16878   +124     
========================================
+ Hits        15359    15473   +114     
- Misses       1056     1063     +7     
- Partials      339      342     +3     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[djaglowski]

Generally looks good to me

[TylerHelmuth]

@mx-psi I believe I've address all feedback

[mx-psi]

@TylerHelmuth can you address the lint error?

Error: collector.go:289:10: indent-error-flow: if block ends with a return statement, so drop this else and outdent its block (revive)

I can merge after that

[mx-psi]

The Windows test failure seems to be legit:

    collector_windows_service_test.go:146: 
        	Error Trace:	D:/a/opentelemetry-collector/opentelemetry-collector/otelcol/collector_windows_service_test.go:146
        	Error:      	Should be true
        	Test:       	TestCollectorAsService/ConfigFileNotFound

It's failing here

opentelemetry-collector/otelcol/collector_windows_service_test.go

Line 146 in 7b63bfc

require.True(t, eventTime.After(startTime.In(time.UTC)))

with this test

opentelemetry-collector/otelcol/collector_windows_service_test.go

Lines 54 to 58 in 7b63bfc

	{
	name: "ConfigFileNotFound",
	configFile: filepath.Join(".", "non", "existent", "otel-config.yaml"),
	expectStartFailure: true,
	},

The Windows service uses a special zap core

opentelemetry-collector/otelcol/collector_windows.go

Line 203 in 7b63bfc

func withWindowsCore(elog *eventlog.Log, serviceConfig **service.Config) func(zapcore.Core) zapcore.Core {

I am wondering if the fallback logger is somehow messing with this

[TylerHelmuth]

Agreed that the failure is legit, looking into it. I'll be using CI as the runner lol

[TylerHelmuth]

@pjanotti I need some help here, why can the windows service not handle instantiating a fallback logger before returning the error on start? If I use zap.NewNOP it works, but something about the builder causes problems.

[TylerHelmuth]

@mx-psi fixed the windows test. The issue was that when the config could not resolve then the underlying serviceConfig was nil, but this check was only checking the pointer to the pointer not being nil. When we passed the options to the fallback logger we ended up in this method and tried to do (*serviceConfig).Telemetry.Logs.OutputPaths on nil. Added an extra nil check to make sure the underlying serviceConfig is also not nil.

[codeboten]

Thanks for getting to the bottom of this Windows test, can we make you the official Windows reviewer? 😄

[TylerHelmuth]

No thank you

[mx-psi]

Thanks for digging into this, we'll officially name you the Windows guy in a separate PR 😄 🪟