Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

graceful shutdown shows internal error instead of context canceled #160

Closed
jaronoff97 opened this issue Mar 21, 2024 · 1 comment · Fixed by #170
Closed

graceful shutdown shows internal error instead of context canceled #160

jaronoff97 opened this issue Mar 21, 2024 · 1 comment · Fixed by #170

Comments

@jaronoff97
Copy link
Contributor

I'm running a local build of an arrow collector in docker and when i go to shutdown the container the context gets canceled which then makes all of the streams error like:

2024-03-21T17:19:29.220Z	info	otelcol@v0.96.0/collector.go:272	Received signal from OS	{"signal": "interrupt"}
2024-03-21T17:19:29.220Z	info	service@v0.96.0/service.go:206	Starting shutdown...
2024-03-21T17:19:29.220Z	error	arrow/stream.go:256	arrow stream internal error	{"kind": "exporter", "data_type": "metrics", "name": "otelarrow", "error": "context canceled"}
github.com/open-telemetry/otel-arrow/collector/exporter/otelarrowexporter/internal/arrow.(*Stream).run
	github.com/open-telemetry/otel-arrow/collector/exporter/otelarrowexporter@v0.18.0/internal/arrow/stream.go:256
github.com/open-telemetry/otel-arrow/collector/exporter/otelarrowexporter/internal/arrow.(*Exporter).runArrowStream
	github.com/open-telemetry/otel-arrow/collector/exporter/otelarrowexporter@v0.18.0/internal/arrow/exporter.go:214
2024-03-21T17:19:29.220Z	error	arrow/stream.go:256	arrow stream internal error	{"kind": "exporter", "data_type": "metrics", "name": "otelarrow", "error": "context canceled"}
github.com/open-telemetry/otel-arrow/collector/exporter/otelarrowexporter/internal/arrow.(*Stream).run
	github.com/open-telemetry/otel-arrow/collector/exporter/otelarrowexporter@v0.18.0/internal/arrow/stream.go:256
github.com/open-telemetry/otel-arrow/collector/exporter/otelarrowexporter/internal/arrow.(*Exporter).runArrowStream
	github.com/open-telemetry/otel-arrow/collector/exporter/otelarrowexporter@v0.18.0/internal/arrow/exporter.go:214
2024-03-21T17:19:29.220Z	error	arrow/stream.go:256	arrow stream internal error	{"kind": "exporter", "data_type": "metrics", "name": "otelarrow", "error": "context canceled"}
github.com/open-telemetry/otel-arrow/collector/exporter/otelarrowexporter/internal/arrow.(*Stream).run
	github.com/open-telemetry/otel-arrow/collector/exporter/otelarrowexporter@v0.18.0/internal/arrow/stream.go:256
github.com/open-telemetry/otel-arrow/collector/exporter/otelarrowexporter/internal/arrow.(*Exporter).runArrowStream
	github.com/open-telemetry/otel-arrow/collector/exporter/otelarrowexporter@v0.18.0/internal/arrow/exporter.go:214
2024-03-21T17:19:29.220Z	error	arrow/stream.go:256	arrow stream internal error	{"kind": "exporter", "data_type": "metrics", "name": "otelarrow", "error": "context canceled"}
github.com/open-telemetry/otel-arrow/collector/exporter/otelarrowexporter/internal/arrow.(*Stream).run
	github.com/open-telemetry/otel-arrow/collector/exporter/otelarrowexporter@v0.18.0/internal/arrow/stream.go:256
github.com/open-telemetry/otel-arrow/collector/exporter/otelarrowexporter/internal/arrow.(*Exporter).runArrowStream
	github.com/open-telemetry/otel-arrow/collector/exporter/otelarrowexporter@v0.18.0/internal/arrow/exporter.go:214
2024-03-21T17:19:29.220Z	error	arrow/stream.go:256	arrow stream internal error	{"kind": "exporter", "data_type": "metrics", "name": "otelarrow", "error": "context canceled"}
github.com/open-telemetry/otel-arrow/collector/exporter/otelarrowexporter/internal/arrow.(*Stream).run
	github.com/open-telemetry/otel-arrow/collector/exporter/otelarrowexporter@v0.18.0/internal/arrow/stream.go:256
github.com/open-telemetry/otel-arrow/collector/exporter/otelarrowexporter/internal/arrow.(*Exporter).runArrowStream
	github.com/open-telemetry/otel-arrow/collector/exporter/otelarrowexporter@v0.18.0/internal/arrow/exporter.go:214
2024-03-21T17:19:29.220Z	error	arrow/stream.go:256	arrow stream internal error	{"kind": "exporter", "data_type": "metrics", "name": "otelarrow", "error": "context canceled"}
github.com/open-telemetry/otel-arrow/collector/exporter/otelarrowexporter/internal/arrow.(*Stream).run
	github.com/open-telemetry/otel-arrow/collector/exporter/otelarrowexporter@v0.18.0/internal/arrow/stream.go:256
github.com/open-telemetry/otel-arrow/collector/exporter/otelarrowexporter/internal/arrow.(*Exporter).runArrowStream
	github.com/open-telemetry/otel-arrow/collector/exporter/otelarrowexporter@v0.18.0/internal/arrow/exporter.go:214
2024-03-21T17:19:29.221Z	error	arrow/stream.go:256	arrow stream internal error	{"kind": "exporter", "data_type": "traces", "name": "otelarrow", "error": "context canceled"}
github.com/open-telemetry/otel-arrow/collector/exporter/otelarrowexporter/internal/arrow.(*Stream).run
	github.com/open-telemetry/otel-arrow/collector/exporter/otelarrowexporter@v0.18.0/internal/arrow/stream.go:256
github.com/open-telemetry/otel-arrow/collector/exporter/otelarrowexporter/internal/arrow.(*Exporter).runArrowStream
	github.com/open-telemetry/otel-arrow/collector/exporter/otelarrowexporter@v0.18.0/internal/arrow/exporter.go:214
2024-03-21T17:19:29.221Z	error	arrow/stream.go:256	arrow stream internal error	{"kind": "exporter", "data_type": "traces", "name": "otelarrow", "error": "context canceled"}
github.com/open-telemetry/otel-arrow/collector/exporter/otelarrowexporter/internal/arrow.(*Stream).run
	github.com/open-telemetry/otel-arrow/collector/exporter/otelarrowexporter@v0.18.0/internal/arrow/stream.go:256
github.com/open-telemetry/otel-arrow/collector/exporter/otelarrowexporter/internal/arrow.(*Exporter).runArrowStream
	github.com/open-telemetry/otel-arrow/collector/exporter/otelarrowexporter@v0.18.0/internal/arrow/exporter.go:214
2024-03-21T17:19:29.221Z	error	arrow/stream.go:256	arrow stream internal error	{"kind": "exporter", "data_type": "traces", "name": "otelarrow", "error": "context canceled"}
github.com/open-telemetry/otel-arrow/collector/exporter/otelarrowexporter/internal/arrow.(*Stream).run
	github.com/open-telemetry/otel-arrow/collector/exporter/otelarrowexporter@v0.18.0/internal/arrow/stream.go:256
github.com/open-telemetry/otel-arrow/collector/exporter/otelarrowexporter/internal/arrow.(*Exporter).runArrowStream
	github.com/open-telemetry/otel-arrow/collector/exporter/otelarrowexporter@v0.18.0/internal/arrow/exporter.go:214
2024-03-21T17:19:29.221Z	error	arrow/stream.go:256	arrow stream internal error	{"kind": "exporter", "data_type": "traces", "name": "otelarrow", "error": "context canceled"}
github.com/open-telemetry/otel-arrow/collector/exporter/otelarrowexporter/internal/arrow.(*Stream).run
	github.com/open-telemetry/otel-arrow/collector/exporter/otelarrowexporter@v0.18.0/internal/arrow/stream.go:256
github.com/open-telemetry/otel-arrow/collector/exporter/otelarrowexporter/internal/arrow.(*Exporter).runArrowStream
	github.com/open-telemetry/otel-arrow/collector/exporter/otelarrowexporter@v0.18.0/internal/arrow/exporter.go:214
2024-03-21T17:19:29.221Z	info	extensions/extensions.go:59	Stopping extensions...

Given we know its a cancel, i wonder if it needs to be so loud in the logs in what is otherwise a graceful shutdown.
@jmacd
Copy link
Contributor

jmacd commented Mar 29, 2024

As I read the code, the "internal error" is actually a canceled status from the writer thread, which it's printing because the reader received a canceled. Action is to avoid showing canceled from both places.

@jmacd jmacd mentioned this issue Apr 1, 2024
Merged
@jmacd jmacd closed this as completed in 9551a08 Apr 3, 2024
@jmacd jmacd closed this as completed in #170 Apr 3, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
2 participants
@jmacd @jaronoff97