Set grpc logger to collector #4501

pavankrish123 · 2021-12-01T21:43:45Z

Description:

Adding a feature - This feature sets the grpc logger to one derived from collector logger configuration and enables users to control grpc logs.

In the case of collector default configuration, gRPC logs minimum level is set to WARN (to reduce the log traffic) but shared rest of collector's logger configuration.

$ ./bin/cmd-otelcol --config ./bin/config.yaml
2021-11-30T16:47:38.293-0800	info	service/collector.go:201	Applying configuration...
2021-11-30T16:47:38.294-0800	info	builder/exporters_builder.go:254	Exporter was built.	{"kind": "exporter", "name": "otlp"}
2021-11-30T16:47:38.294-0800	info	builder/exporters_builder.go:254	Exporter was built.	{"kind": "exporter", "name": "logging"}
2021-11-30T16:47:38.295-0800	info	builder/pipelines_builder.go:222	Pipeline was built.	{"name": "pipeline", "name": "metrics"}
2021-11-30T16:47:38.295-0800	info	builder/pipelines_builder.go:222	Pipeline was built.	{"name": "pipeline", "name": "traces"}
2021-11-30T16:47:38.295-0800	info	builder/receivers_builder.go:224	Receiver was built.	{"kind": "receiver", "name": "otlp", "datatype": "traces"}
2021-11-30T16:47:38.295-0800	info	builder/receivers_builder.go:224	Receiver was built.	{"kind": "receiver", "name": "otlp", "datatype": "metrics"}
2021-11-30T16:47:38.295-0800	info	service/service.go:86	Starting extensions...
2021-11-30T16:47:38.295-0800	info	service/service.go:91	Starting exporters...
2021-11-30T16:47:38.295-0800	info	builder/exporters_builder.go:40	Exporter is starting...	{"kind": "exporter", "name": "logging"}
2021-11-30T16:47:38.295-0800	info	builder/exporters_builder.go:48	Exporter started.	{"kind": "exporter", "name": "logging"}
2021-11-30T16:47:38.295-0800	info	builder/exporters_builder.go:40	Exporter is starting...	{"kind": "exporter", "name": "otlp"}
2021-11-30T16:47:38.295-0800	info	builder/exporters_builder.go:48	Exporter started.	{"kind": "exporter", "name": "otlp"}
2021-11-30T16:47:38.295-0800	info	service/service.go:96	Starting processors...
2021-11-30T16:47:38.295-0800	info	builder/pipelines_builder.go:54	Pipeline is starting...	{"name": "pipeline", "name": "metrics"}
2021-11-30T16:47:38.295-0800	info	builder/pipelines_builder.go:65	Pipeline is started.	{"name": "pipeline", "name": "metrics"}
2021-11-30T16:47:38.295-0800	info	builder/pipelines_builder.go:54	Pipeline is starting...	{"name": "pipeline", "name": "traces"}
2021-11-30T16:47:38.295-0800	info	builder/pipelines_builder.go:65	Pipeline is started.	{"name": "pipeline", "name": "traces"}
2021-11-30T16:47:38.295-0800	info	service/service.go:101	Starting receivers...
2021-11-30T16:47:38.295-0800	info	builder/receivers_builder.go:68	Receiver is starting...	{"kind": "receiver", "name": "otlp"}
2021-11-30T16:47:38.295-0800	info	otlpreceiver/otlp.go:68	Starting GRPC server on endpoint 0.0.0.0:4317	{"kind": "receiver", "name": "otlp"}
2021-11-30T16:47:38.302-0800	info	builder/receivers_builder.go:73	Receiver started.	{"kind": "receiver", "name": "otlp"}
2021-11-30T16:47:38.302-0800	info	service/telemetry.go:92	Setting up own telemetry...
2021-11-30T16:47:38.310-0800	info	service/telemetry.go:116	Serving Prometheus metrics	{"address": ":8888", "level": "basic", "service.instance.id": "a8caf36a-00df-407e-b356-23569252e525", "service.version": "latest"}
2021-11-30T16:47:38.310-0800	info	service/collector.go:250	Starting cmd-otelcol...	{"Version": "0.40.0-dev", "NumCPU": 12}
2021-11-30T16:47:38.310-0800	info	service/collector.go:138	Everything is ready. Begin running and processing data.
2021-11-30T16:47:38.384-0800	warn	zapgrpc/zapgrpc.go:191	[core] grpc: addrConn.createTransport failed to connect to {otelcol:4317 otelcol:4317 <nil> <nil> 0 <nil>}. Err: connection error: desc = "transport: Error while dialing dial tcp: lookup otelcol: no such host"
2021-11-30T16:47:42.912-0800	INFO	loggingexporter/logging_exporter.go:40	TracesExporter	{"#spans": 200}

for all other levels - gRPC logs min log level takes whatever is set for collector log sharing rest of collector's logger configuration. for example

service:
  telemetry:
    logs:
      level: "DEBUG"
      encoding: "console"
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [logging, otlp]

in non development made

$ ./bin/cmd-otelcol --config ./bin/config.yaml
2021-11-30T17:35:09.442-0800	info	builder/exporters_builder.go:40	Exporter is starting...	{"kind": "exporter", "name": "otlp"}
2021-11-30T17:35:09.442-0800	info	zapgrpc/zapgrpc.go:174	[core] original dial target is: "otelcol:4317"
2021-11-30T17:35:09.443-0800	info	zapgrpc/zapgrpc.go:174	[core] parsed dial target is: {Scheme:otelcol Authority: Endpoint:4317 URL:{Scheme:otelcol Opaque:4317 User: Host: Path: RawPath: ForceQuery:false RawQuery: Fragment: RawFragment:}}
2021-11-30T17:35:09.443-0800	info	zapgrpc/zapgrpc.go:174	[core] fallback to scheme "passthrough"
2021-11-30T17:35:09.443-0800	info	zapgrpc/zapgrpc.go:174	[core] parsed dial target is: {Scheme:passthrough Authority: Endpoint:otelcol:4317 URL:{Scheme:passthrough Opaque: User: Host: Path:/otelcol:4317 RawPath: ForceQuery:false RawQuery: Fragment: RawFragment:}}
2021-11-30T17:35:09.443-0800	info	zapgrpc/zapgrpc.go:174	[core] Channel authority set to "otelcol:4317"
2021-11-30T17:35:09.443-0800	info	zapgrpc/zapgrpc.go:174	[core] ccResolverWrapper: sending update to cc: {[{otelcol:4317  <nil> <nil> 0 <nil>}] <nil> <nil>}
2021-11-30T17:35:09.443-0800	info	zapgrpc/zapgrpc.go:174	[core] ClientConn switching balancer to "pick_first"
2021-11-30T17:35:09.443-0800	info	zapgrpc/zapgrpc.go:174	[core] Channel switches to new LB policy "pick_first"
2021-11-30T17:35:09.443-0800	info	builder/exporters_builder.go:48	Exporter started.	{"kind": "exporter", "name": "otlp"}
2021-11-30T17:35:09.443-0800	info	zapgrpc/zapgrpc.go:174	[core] Subchannel Connectivity change to CONNECTING
2021-11-30T17:35:09.443-0800	info	builder/exporters_builder.go:40	Exporter is starting...	{"kind": "exporter", "name": "logging"}
2021-11-30T17:35:09.443-0800	info	builder/exporters_builder.go:48	Exporter started.	{"kind": "exporter", "name": "logging"}
2021-11-30T17:35:09.443-0800	info	service/service.go:96	Starting processors...
2021-11-30T17:35:09.443-0800	info	builder/pipelines_builder.go:54	Pipeline is starting...	{"name": "pipeline", "name": "traces"}
2021-11-30T17:35:09.443-0800	info	zapgrpc/zapgrpc.go:174	[core] Subchannel picks a new address "otelcol:4317" to connect
2021-11-30T17:35:09.443-0800	info	builder/pipelines_builder.go:65	Pipeline is started.	{"name": "pipeline", "name": "traces"}
2021-11-30T17:35:09.443-0800	info	zapgrpc/zapgrpc.go:174	[core] pickfirstBalancer: UpdateSubConnState: 0xc0000f4440, {CONNECTING <nil>}
2021-11-30T17:35:09.443-0800	info	builder/pipelines_builder.go:54	Pipeline is starting...	{"name": "pipeline", "name": "metrics"}
2021-11-30T17:35:09.443-0800	info	zapgrpc/zapgrpc.go:174	[core] Channel Connectivity change to CONNECTING
2021-11-30T17:35:09.443-0800	info	builder/pipelines_builder.go:65	Pipeline is started.	{"name": "pipeline", "name": "metrics"}
2021-11-30T17:35:09.443-0800	info	service/service.go:101	Starting receivers...
2021-11-30T17:35:09.444-0800	info	builder/receivers_builder.go:68	Receiver is starting...	{"kind": "receiver", "name": "otlp"}
2021-11-30T17:35:09.444-0800	info	otlpreceiver/otlp.go:68	Starting GRPC server on endpoint 0.0.0.0:4317	{"kind": "receiver", "name": "otlp"}
2021-11-30T17:35:09.445-0800	info	builder/receivers_builder.go:73	Receiver started.	{"kind": "receiver", "name": "otlp"}
2021-11-30T17:35:09.445-0800	info	service/telemetry.go:92	Setting up own telemetry...
2021-11-30T17:35:09.449-0800	warn	zapgrpc/zapgrpc.go:191	[core] grpc: addrConn.createTransport failed to connect to {otelcol:4317 otelcol:4317 <nil> <nil> 0 <nil>}. Err: connection error: desc = "transport: Error while dialing dial tcp: lookup otelcol: no such host"
2021-11-30T17:35:09.449-0800	info	zapgrpc/zapgrpc.go:174	[core] Subchannel Connectivity change to TRANSIENT_FAILURE
2021-11-30T17:35:09.449-0800	info	zapgrpc/zapgrpc.go:174	[core] pickfirstBalancer: UpdateSubConnState: 0xc0000f4440, {TRANSIENT_FAILURE connection error: desc = "transport: Error while dialing dial tcp: lookup otelcol: no such host"}
2021-11-30T17:35:09.449-0800	info	zapgrpc/zapgrpc.go:174	[core] Channel Connectivity change to TRANSIENT_FAILURE
2021-11-30T17:35:09.455-0800	info	service/telemetry.go:116	Serving Prometheus metrics	{"address": ":8888", "level": "basic", "service.instance.id": "a579c69f-edb3-4fca-8911-e6fd9a5f5344", "service.version": "latest"}
2021-11-30T17:35:09.456-0800	info	service/collector.go:243	Starting cmd-otelcol...	{"Version": "0.40.0-dev", "NumCPU": 12}
2021-11-30T17:35:09.456-0800	info	service/collector.go:136	Everything is ready. Begin running and processing data.
2021-11-30T17:35:10.449-0800	info	zapgrpc/zapgrpc.go:174	[core] Subchannel Connectivity change to IDLE
2021-11-30T17:35:10.449-0800	info	zapgrpc/zapgrpc.go:174	[core] pickfirstBalancer: UpdateSubConnState: 0xc0000f4440, {IDLE connection error: desc = "transport: Error while dialing dial tcp: lookup otelcol: no such host"}
2021-11-30T17:35:10.449-0800	info	zapgrpc/zapgrpc.go:174	[core] Channel Connectivity change to IDLE
2021-11-30T17:35:12.072-0800	info	zapgrpc/zapgrpc.go:174	[transport] transport: loopyWriter.run returning. connection error: desc = "transport is closing"
2021-11-30T17:35:12.255-0800	INFO	loggingexporter/logging_exporter.go:40	TracesExporter	{"#spans": 2}
2021-11-30T17:35:28.595-0800	info	zapgrpc/zapgrpc.go:174	[core] Subchannel Connectivity change to TRANSIENT_FAILURE
2021-11-30T17:35:28.595-0800	info	zapgrpc/zapgrpc.go:174	[core] pickfirstBalancer: UpdateSubConnState: 0xc0000f4440, {TRANSIENT_FAILURE connection error: desc = "transport: Error while dialing dial tcp: lookup otelcol: no such host"}
2021-11-30T17:35:28.595-0800	info	zapgrpc/zapgrpc.go:174	[core] Channel Connectivity change to TRANSIENT_FAILURE
2021-11-30T17:35:28.595-0800	info	exporterhelper/queued_retry.go:215	Exporting failed. Will retry the request after interval.	{"kind": "exporter", "name": "otlp", "error": "rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp: lookup otelcol: no such host\"", "interval": "13.101300599s"}
2021-11-30T17:35:32.049-0800	info	zapgrpc/zapgrpc.go:174	[core] Subchannel Connectivity change to IDLE
2021-11-30T17:35:32.049-0800	info	zapgrpc/zapgrpc.go:174	[core] pickfirstBalancer: UpdateSubConnState: 0xc0000f4440, {IDLE connection error: desc = "transport: Error while dialing dial tcp: lookup otelcol: no such host"}
2021-11-30T17:35:32.049-0800	info	zapgrpc/zapgrpc.go:174	[core] Channel Connectivity change to IDLE

DEBUG in development mode

service:
  telemetry:
    logs:
      level: "DEBUG"
      development: true
      encoding: "console"
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [logging, otlp]

produces

2021-11-30T17:30:55.091-0800	info	builder/pipelines_builder.go:65	Pipeline is started.	{"name": "pipeline", "name": "metrics"}
2021-11-30T17:30:55.091-0800	info	builder/pipelines_builder.go:54	Pipeline is starting...	{"name": "pipeline", "name": "traces"}
2021-11-30T17:30:55.091-0800	info	zapgrpc/zapgrpc.go:174	[core] Subchannel picks a new address "otelcol:4317" to connect
2021-11-30T17:30:55.091-0800	info	builder/pipelines_builder.go:65	Pipeline is started.	{"name": "pipeline", "name": "traces"}
2021-11-30T17:30:55.091-0800	info	zapgrpc/zapgrpc.go:174	[core] pickfirstBalancer: UpdateSubConnState: 0xc00011c8f0, {CONNECTING <nil>}
2021-11-30T17:30:55.091-0800	info	service/service.go:101	Starting receivers...
2021-11-30T17:30:55.091-0800	info	zapgrpc/zapgrpc.go:174	[core] Channel Connectivity change to CONNECTING
2021-11-30T17:30:55.091-0800	info	builder/receivers_builder.go:68	Receiver is starting...	{"kind": "receiver", "name": "otlp"}
2021-11-30T17:30:55.091-0800	info	otlpreceiver/otlp.go:68	Starting GRPC server on endpoint 0.0.0.0:4317	{"kind": "receiver", "name": "otlp"}
2021-11-30T17:30:55.091-0800	info	builder/receivers_builder.go:73	Receiver started.	{"kind": "receiver", "name": "otlp"}
2021-11-30T17:30:55.091-0800	info	service/telemetry.go:92	Setting up own telemetry...
2021-11-30T17:30:55.101-0800	info	service/telemetry.go:116	Serving Prometheus metrics	{"address": ":8888", "level": "basic", "service.instance.id": "38ece474-2cdc-42a3-b442-46477936c2b7", "service.version": "latest"}
2021-11-30T17:30:55.101-0800	info	service/collector.go:243	Starting cmd-otelcol...	{"Version": "0.40.0-dev", "NumCPU": 12}
2021-11-30T17:30:55.101-0800	info	service/collector.go:136	Everything is ready. Begin running and processing data.
2021-11-30T17:30:55.236-0800	warn	zapgrpc/zapgrpc.go:191	[core] grpc: addrConn.createTransport failed to connect to {otelcol:4317 otelcol:4317 <nil> <nil> 0 <nil>}. Err: connection error: desc = "transport: Error while dialing dial tcp: lookup otelcol: no such host"
go.uber.org/zap/zapgrpc.(*Logger).Warningln
	go.uber.org/zap@v1.19.1/zapgrpc/zapgrpc.go:191
google.golang.org/grpc/internal/grpclog.WarningDepth
	google.golang.org/grpc@v1.42.0/internal/grpclog/grpclog.go:46
google.golang.org/grpc/grpclog.(*componentData).WarningDepth
	google.golang.org/grpc@v1.42.0/grpclog/component.go:41
google.golang.org/grpc/internal/channelz.Warningf
	google.golang.org/grpc@v1.42.0/internal/channelz/logging.go:75
google.golang.org/grpc.(*addrConn).createTransport
	google.golang.org/grpc@v1.42.0/clientconn.go:1315
google.golang.org/grpc.(*addrConn).tryAllAddrs
	google.golang.org/grpc@v1.42.0/clientconn.go:1249
google.golang.org/grpc.(*addrConn).resetTransport
	google.golang.org/grpc@v1.42.0/clientconn.go:1184
google.golang.org/grpc.(*addrConn).connect
	google.golang.org/grpc@v1.42.0/clientconn.go:845
2021-11-30T17:30:55.237-0800	info	zapgrpc/zapgrpc.go:174	[core] Subchannel Connectivity change to TRANSIENT_FAILURE
2021-11-30T17:30:55.237-0800	info	zapgrpc/zapgrpc.go:174	[core] pickfirstBalancer: UpdateSubConnState: 0xc00011c8f0, {TRANSIENT_FAILURE connection error: desc = "transport: Error while dialing dial tcp: lookup otelcol: no such host"}
2021-11-30T17:30:55.237-0800	info	zapgrpc/zapgrpc.go:174	[core] Channel Connectivity change to TRANSIENT_FAILURE
2021-11-30T17:30:56.238-0800	info	zapgrpc/zapgrpc.go:174	[core] Subchannel Connectivity change to IDLE
2021-11-30T17:30:56.238-0800	info	zapgrpc/zapgrpc.go:174	[core] pickfirstBalancer: UpdateSubConnState: 0xc00011c8f0, {IDLE connection error: desc = "transport: Error while dialing dial tcp: lookup otelcol: no such host"}
2021-11-30T17:30:56.238-0800	info	zapgrpc/zapgrpc.go:174	[core] Channel Connectivity change to IDLE
^C2021-11-30T17:30:58.448-0800	info	service/collector.go:167	Received signal from OS	{"signal": "interrupt"}
2021-11-30T17:30:58.448-0800	info	service/collector.go:259	Starting shutdown...
2021-11-30

Link to tracking Issue: #2237

Testing: Added unit tests and manual testing

- used grpc_settable instead of grpc_zap ReplaceV2 directly as it supports multiple sets in thread safe fashoin as opposed to grpc_zap.ReplaceV2

codecov · 2021-12-01T21:52:07Z

Codecov Report

Merging #4501 (c2f49c0) into main (9309a1f) will increase coverage by 0.02%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##             main    #4501      +/-   ##
==========================================
+ Coverage   90.67%   90.70%   +0.02%     
==========================================
  Files         179      179              
  Lines       10414    10445      +31     
==========================================
+ Hits         9443     9474      +31     
  Misses        755      755              
  Partials      216      216

Impacted Files	Coverage Δ
service/collector.go	`74.21% <100.00%> (+0.32%)`	⬆️
service/internal/telemetrylogs/logger.go	`92.85% <100.00%> (+5.35%)`	⬆️
exporter/otlphttpexporter/otlp.go	`82.92% <0.00%> (+0.87%)`	⬆️
exporter/otlpexporter/otlp.go	`74.41% <0.00%> (+1.57%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9309a1f...c2f49c0. Read the comment docs.

go.sum

jpkrohling

This looks like the right direction to me, we just need to settle on the opentracing dependency that is brought by go-grpc-middleware.

service/internal/telemetrylogs/logger.go

examples/local/otel-config.bak

service/internal/telemetrylogs/logger.go

service/collector.go

service/internal/telemetrylogs/logger.go

pavankrish123 · 2021-12-10T21:29:06Z

@bogdandrutu @tigrannajaryan can we please get either of your blessing on this PR and let me know if anything else is needed. Please and Thank you.

@jpkrohling Also I do see release-after-ga label on the tracking feature request #2237. Let me know if that label is still valid and we want to merge according to that label.

I can update our internal teams using collector with a workaround solution involving environment variables accordingly.

Thanks you

cc: @svrnm

jpkrohling · 2021-12-13T08:56:44Z

I don't think that label has any meaning by now, don't worry about it.

@bogdandrutu, @tigrannajaryan, we need a maintainer to merge this, possibly doing a final review (despite having three approvals).

tigrannajaryan · 2021-12-13T14:52:19Z

In the case of collector default configuration, gRPC logs minimum level is set to WARN (to reduce the log traffic) but shared rest of collector's logger configuration.

Before merging this I would like to understand how chatty is gRPC in WARN level. For example if there is no connection to the destination does it periodically print warnings? Can this flood the logs?

pavankrish123 · 2021-12-13T21:06:06Z

Thank you @tigrannajaryan - In case of default collector setting - which sets gRPC logging to WARN - We will have 1 message per connection failure per retry for exporter.

2021-11-30T16:47:38.384-0800    warn    zapgrpc/zapgrpc.go:191  [core] grpc: addrConn.createTransport failed to connect to {otelcol:4317 otelcol:4317 <nil> <nil> 0 <nil>}. Err: connection error: desc = "transport: Error while dialing dial tcp: lookup otelcol: no such host"

I have not tested for all other failures - but quickly glancing the gRPC code. The error specific verbose logs only happen to appear in INFO level.

On the other hand - once we set debug logging on collector - we will have more chatty messages about gRPC transport
these are 1 time start up messages per grpc exporter

2021-11-30T17:35:09.442-0800    info    zapgrpc/zapgrpc.go:174  [core] original dial target is: "otelcol:4317"
2021-11-30T17:35:09.443-0800    info    zapgrpc/zapgrpc.go:174  [core] parsed dial target is: {Scheme:otelcol Authority: Endpoint:4317 URL:{Scheme:otelcol Opaque:4317 User: Host: Path: RawPath: ForceQuery:false RawQuery: Fragment: RawFragment:}}
2021-11-30T17:35:09.443-0800    info    zapgrpc/zapgrpc.go:174  [core] fallback to scheme "passthrough"
2021-11-30T17:35:09.443-0800    info    zapgrpc/zapgrpc.go:174  [core] parsed dial target is: {Scheme:passthrough Authority: Endpoint:otelcol:4317 URL:{Scheme:passthrough Opaque: User: Host: Path:/otelcol:4317 RawPath: ForceQuery:false RawQuery: Fragment: RawFragment:}}
2021-11-30T17:35:09.443-0800    info    zapgrpc/zapgrpc.go:174  [core] Channel authority set to "otelcol:4317"
2021-11-30T17:35:09.443-0800    info    zapgrpc/zapgrpc.go:174  [core] ccResolverWrapper: sending update to cc: {[{otelcol:4317  <nil> <nil> 0 <nil>}] <nil> <nil>}
2021-11-30T17:35:09.443-0800    info    zapgrpc/zapgrpc.go:174  [core] ClientConn switching balancer to "pick_first"
2021-11-30T17:35:09.443-0800    info    zapgrpc/zapgrpc.go:174  [core] Channel switches to new LB policy "pick_first"

plus in case of failure (for no host error) we will have these set of error specific messages per retry cycle of the exporter.

2021-11-30T17:35:09.445-0800    info    service/telemetry.go:92 Setting up own telemetry...
2021-11-30T17:35:09.449-0800    warn    zapgrpc/zapgrpc.go:191  [core] grpc: addrConn.createTransport failed to connect to {otelcol:4317 otelcol:4317 <nil> <nil> 0 <nil>}. Err: connection error: desc = "transport: Error while dialing dial tcp: lookup otelcol: no such host"
2021-11-30T17:35:09.449-0800    info    zapgrpc/zapgrpc.go:174  [core] Subchannel Connectivity change to TRANSIENT_FAILURE
2021-11-30T17:35:09.449-0800    info    zapgrpc/zapgrpc.go:174  [core] pickfirstBalancer: UpdateSubConnState: 0xc0000f4440, {TRANSIENT_FAILURE connection error: desc = "transport: Error while dialing dial tcp: lookup otelcol: no such host"}
2021-11-30T17:35:09.449-0800    info    zapgrpc/zapgrpc.go:174  [core] Channel Connectivity change to TRANSIENT_FAILURE
2021-11-30T17:35:10.449-0800    info    zapgrpc/zapgrpc.go:174  [core] Subchannel Connectivity change to IDLE
2021-11-30T17:35:10.449-0800    info    zapgrpc/zapgrpc.go:174  [core] pickfirstBalancer: UpdateSubConnState: 0xc0000f4440, {IDLE connection error: desc = "transport: Error while dialing dial tcp: lookup otelcol: no such host"}
2021-11-30T17:35:10.449-0800    info    zapgrpc/zapgrpc.go:174  [core] Channel Connectivity change to IDLE
2021-11-30T17:35:12.072-0800    info    zapgrpc/zapgrpc.go:174  [transport] transport: loopyWriter.run returning. connection error: desc = "transport is closing"
2021-11-30T17:35:12.255-0800    INFO    loggingexporter/logging_exporter.go:40  TracesExporter  {"#spans": 2}
2021-11-30T17:35:28.595-0800    info    zapgrpc/zapgrpc.go:174  [core] Subchannel Connectivity change to TRANSIENT_FAILURE
2021-11-30T17:35:28.595-0800    info    zapgrpc/zapgrpc.go:174  [core] pickfirstBalancer: UpdateSubConnState: 0xc0000f4440, {TRANSIENT_FAILURE connection error: desc = "transport: Error while dialing dial tcp: lookup otelcol: no such host"}
2021-11-30T17:35:28.595-0800    info    zapgrpc/zapgrpc.go:174  [core] Channel Connectivity change to TRANSIENT_FAILURE

The default behavior of gRPC framework itself is to set logging level to ERR and write logs to `ioutil.DIscard.

tigrannajaryan · 2021-12-13T22:01:44Z

This sounds ok to me. Let's merge this but be prepared to adjust the default behavior if it turns out to be more noisy than we would want.

pavankrish123 · 2021-12-13T22:48:00Z

Thanks @tigrannajaryan . Please let me know if you need anything else from my end.

Set grpc logger to collector

8b2c69c

- used grpc_settable instead of grpc_zap ReplaceV2 directly as it supports multiple sets in thread safe fashoin as opposed to grpc_zap.ReplaceV2

pavankrish123 requested a review from a team as a code owner December 1, 2021 21:43

pavankrish123 requested a review from bogdandrutu December 1, 2021 21:43

added commit message

5bbcd41

mx-psi reviewed Dec 2, 2021

View reviewed changes

go.sum Outdated Show resolved Hide resolved

jpkrohling self-requested a review December 2, 2021 08:50

jpkrohling reviewed Dec 2, 2021

View reviewed changes

service/internal/telemetrylogs/logger.go Outdated Show resolved Hide resolved

pavankrish123 added 2 commits December 2, 2021 01:23

added comment

9b0cf8b

added comment

4e19b87

jpkrohling reviewed Dec 2, 2021

View reviewed changes

examples/local/otel-config.bak Outdated Show resolved Hide resolved

pavankrish123 added 2 commits December 2, 2021 09:05

removed temorary file

bb15c20

removed usage of settable from grpc middleware and work around

9210d3e

jpkrohling reviewed Dec 3, 2021

View reviewed changes

service/internal/telemetrylogs/logger.go Outdated Show resolved Hide resolved

service/collector.go Outdated Show resolved Hide resolved

pavankrish123 added 2 commits December 3, 2021 02:01

fixed comments

c44808e

fixed comments

542b4c9

jpkrohling reviewed Dec 3, 2021

View reviewed changes

service/collector.go Outdated Show resolved Hide resolved

pavankrish123 requested a review from jpkrohling December 3, 2021 10:13

fixed comments

8199c1d

jpkrohling approved these changes Dec 3, 2021

View reviewed changes

Merge branch 'main' into new/grpclogging

7e625e1

mx-psi reviewed Dec 3, 2021

View reviewed changes

service/internal/telemetrylogs/logger.go Show resolved Hide resolved

Aneurysm9 approved these changes Dec 3, 2021

View reviewed changes

added comment explaining ignore error

c2f49c0

mx-psi approved these changes Dec 6, 2021

View reviewed changes

jpkrohling approved these changes Dec 6, 2021

View reviewed changes

tigrannajaryan merged commit 2ba858c into open-telemetry:main Dec 13, 2021

pavankrish123 deleted the new/grpclogging branch December 16, 2021 00:57

jpkrohling mentioned this pull request Dec 16, 2021

[exporter/coralogix] Initial implementation open-telemetry/opentelemetry-collector-contrib#6816

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Set grpc logger to collector #4501

Set grpc logger to collector #4501

pavankrish123 commented Dec 1, 2021 •

edited

codecov bot commented Dec 1, 2021 •

edited

jpkrohling left a comment

pavankrish123 commented Dec 10, 2021 •

edited

jpkrohling commented Dec 13, 2021

tigrannajaryan commented Dec 13, 2021 •

edited

pavankrish123 commented Dec 13, 2021 •

edited

tigrannajaryan commented Dec 13, 2021

pavankrish123 commented Dec 13, 2021

Set grpc logger to collector #4501

Set grpc logger to collector #4501

Conversation

pavankrish123 commented Dec 1, 2021 • edited

codecov bot commented Dec 1, 2021 • edited

Codecov Report

jpkrohling left a comment

Choose a reason for hiding this comment

pavankrish123 commented Dec 10, 2021 • edited

jpkrohling commented Dec 13, 2021

tigrannajaryan commented Dec 13, 2021 • edited

pavankrish123 commented Dec 13, 2021 • edited

tigrannajaryan commented Dec 13, 2021

pavankrish123 commented Dec 13, 2021

pavankrish123 commented Dec 1, 2021 •

edited

codecov bot commented Dec 1, 2021 •

edited

pavankrish123 commented Dec 10, 2021 •

edited

tigrannajaryan commented Dec 13, 2021 •

edited

pavankrish123 commented Dec 13, 2021 •

edited