Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[receiver/prometheusreceiver] Tests hitting timeouts intermittently on actuated-arm64 runs #32298

Closed
crobert-1 opened this issue Apr 10, 2024 · 11 comments · Fixed by #32341
Closed

[receiver/prometheusreceiver] Tests hitting timeouts intermittently on actuated-arm64 runs #32298

crobert-1 opened this issue Apr 10, 2024 · 11 comments · Fixed by #32341
Labels
flaky test a test is flaky receiver/prometheus Prometheus receiver

Comments

@crobert-1
Copy link
Member

Component(s)

receiver/prometheus

Describe the issue you're reporting

Failed run

Output:

Running target 'test-with-cover' in module 'receiver/prometheusreceiver' as part of group 'receiver-2'
make --no-print-directory -C receiver/prometheusreceiver test-with-cover
mkdir -p /home/runner/actions-runner/_work/opentelemetry-collector-contrib/opentelemetry-collector-contrib/coverage/unit
/home/runner/actions-runner/_work/opentelemetry-collector-contrib/opentelemetry-collector-contrib/.tools/gotestsum --rerun-fails=1 --packages="./..." -- -race -timeout 300s -parallel 4 --tags="" -cover -covermode=atomic -args -test.gocoverdir="/home/runner/actions-runner/_work/opentelemetry-collector-contrib/opentelemetry-collector-contrib/coverage/unit"
∅  internal/metadata
✓  internal (8.33s) (coverage: 85.0% of statements)
✖  . (5m0.187s)

=== Skipped
=== SKIP: . TestOpenMetricsFail/bad_metadata_in_wrong_place_2 (0.00s)
ERROR rerun aborted because previous run had a suspected panic and some test may not have run
    metrics_receiver_open_metrics_test.go:85: skipping failing negative OpenMetrics parser tests
make[2]: *** [../../Makefile.Common:131: test-with-cover] Error 3
make[1]: *** [Makefile:165: receiver/prometheusreceiver] Error 2
make: *** [Makefile:117: gotest-with-cover] Error 2

=== SKIP: . TestOpenMetricsFail/bad_metadata_in_wrong_place_0 (0.00s)
    metrics_receiver_open_metrics_test.go:85: skipping failing negative OpenMetrics parser tests

=== SKIP: . TestOpenMetricsFail/bad_exemplars_on_unallowed_samples_2 (0.00s)
    metrics_receiver_open_metrics_test.go:85: skipping failing negative OpenMetrics parser tests

=== SKIP: . TestOpenMetricsFail/bad_grouping_or_ordering_8 (0.00s)
    metrics_receiver_open_metrics_test.go:85: skipping failing negative OpenMetrics parser tests

=== SKIP: . TestOpenMetricsFail/bad_stateset_info_values_0 (0.00s)
    metrics_receiver_open_metrics_test.go:85: skipping failing negative OpenMetrics parser tests

=== SKIP: . TestOpenMetricsFail/bad_histograms_6 (0.00s)
    metrics_receiver_open_metrics_test.go:85: skipping failing negative OpenMetrics parser tests

=== SKIP: . TestOpenMetricsFail/bad_grouping_or_ordering_0 (0.00s)
    metrics_receiver_open_metrics_test.go:85: skipping failing negative OpenMetrics parser tests

=== SKIP: . TestOpenMetricsFail/bad_counter_values_5 (0.00s)
    metrics_receiver_open_metrics_test.go:85: skipping failing negative OpenMetrics parser tests

=== SKIP: . TestOpenMetricsFail/bad_counter_values_18 (0.00s)
    metrics_receiver_open_metrics_test.go:85: skipping failing negative OpenMetrics parser tests

=== SKIP: . TestOpenMetricsFail/bad_counter_values_16 (0.00s)
    metrics_receiver_open_metrics_test.go:85: skipping failing negative OpenMetrics parser tests

=== SKIP: . TestOpenMetricsFail/bad_missing_or_invalid_labels_for_a_type_6 (0.00s)
    metrics_receiver_open_metrics_test.go:85: skipping failing negative OpenMetrics parser tests

=== SKIP: . TestOpenMetricsFail/bad_exemplar_timestamp_0 (0.00s)
    metrics_receiver_open_metrics_test.go:85: skipping failing negative OpenMetrics parser tests

=== SKIP: . TestOpenMetricsFail/bad_grouping_or_ordering_3 (0.00s)
    metrics_receiver_open_metrics_test.go:85: skipping failing negative OpenMetrics parser tests

=== SKIP: . TestOpenMetricsFail/bad_clashing_names_2 (0.00s)
    metrics_receiver_open_metrics_test.go:85: skipping failing negative OpenMetrics parser tests

=== SKIP: . TestOpenMetricsFail/bad_histograms_1 (0.00s)
    metrics_receiver_open_metrics_test.go:85: skipping failing negative OpenMetrics parser tests

=== SKIP: . TestOpenMetricsFail/bad_metadata_in_wrong_place_1 (0.00s)
    metrics_receiver_open_metrics_test.go:85: skipping failing negative OpenMetrics parser tests

=== SKIP: . TestOpenMetricsFail/bad_counter_values_11 (0.00s)
    metrics_receiver_open_metrics_test.go:85: skipping failing negative OpenMetrics parser tests

=== SKIP: . TestOpenMetricsFail/bad_counter_values_0 (0.00s)
    metrics_receiver_open_metrics_test.go:85: skipping failing negative OpenMetrics parser tests

=== SKIP: . TestOpenMetricsFail/bad_grouping_or_ordering_4 (0.00s)
    metrics_receiver_open_metrics_test.go:85: skipping failing negative OpenMetrics parser tests

=== SKIP: . TestOpenMetricsFail/bad_exemplar_timestamp_1 (0.00s)
    metrics_receiver_open_metrics_test.go:85: skipping failing negative OpenMetrics parser tests

=== SKIP: . TestOpenMetricsFail/bad_counter_values_3 (0.00s)
    metrics_receiver_open_metrics_test.go:85: skipping failing negative OpenMetrics parser tests

=== SKIP: . TestOpenMetricsFail/bad_missing_or_invalid_labels_for_a_type_7 (0.00s)
    metrics_receiver_open_metrics_test.go:85: skipping failing negative OpenMetrics parser tests

=== SKIP: . TestOpenMetricsFail/bad_exemplars_on_unallowed_samples_1 (0.00s)
    metrics_receiver_open_metrics_test.go:85: skipping failing negative OpenMetrics parser tests

=== SKIP: . TestOpenMetricsFail/bad_counter_values_17 (0.00s)
    metrics_receiver_open_metrics_test.go:85: skipping failing negative OpenMetrics parser tests

=== SKIP: . TestOpenMetricsFail/bad_histograms_7 (0.00s)
    metrics_receiver_open_metrics_test.go:85: skipping failing negative OpenMetrics parser tests

=== SKIP: . TestOpenMetricsFail/bad_missing_or_invalid_labels_for_a_type_3 (0.00s)
    metrics_receiver_open_metrics_test.go:85: skipping failing negative OpenMetrics parser tests

=== SKIP: . TestOpenMetricsFail/bad_repeated_metadata_3 (0.00s)
    metrics_receiver_open_metrics_test.go:85: skipping failing negative OpenMetrics parser tests

=== SKIP: . TestOpenMetricsFail/bad_missing_or_invalid_labels_for_a_type_4 (0.00s)
    metrics_receiver_open_metrics_test.go:85: skipping failing negative OpenMetrics parser tests

=== SKIP: . TestOpenMetricsFail/bad_timestamp_7 (0.00s)
    metrics_receiver_open_metrics_test.go:85: skipping failing negative OpenMetrics parser tests

=== SKIP: . TestOpenMetricsFail/bad_stateset_info_values_2 (0.00s)
    metrics_receiver_open_metrics_test.go:85: skipping failing negative OpenMetrics parser tests

=== SKIP: . TestOpenMetricsFail/bad_counter_values_6 (0.00s)
    metrics_receiver_open_metrics_test.go:85: skipping failing negative OpenMetrics parser tests

=== SKIP: . TestOpenMetricsFail/bad_repeated_metadata_1 (0.00s)
    metrics_receiver_open_metrics_test.go:85: skipping failing negative OpenMetrics parser tests

=== SKIP: . TestOpenMetricsFail/bad_histograms_3 (0.00s)
    metrics_receiver_open_metrics_test.go:85: skipping failing negative OpenMetrics parser tests

=== SKIP: . TestOpenMetricsFail/bad_unit_7 (0.00s)
    metrics_receiver_open_metrics_test.go:85: skipping failing negative OpenMetrics parser tests

=== SKIP: . TestOpenMetricsFail/bad_stateset_info_values_3 (0.00s)
    metrics_receiver_open_metrics_test.go:85: skipping failing negative OpenMetrics parser tests

=== SKIP: . TestOpenMetricsFail/bad_repeated_metadata_0 (0.00s)
    metrics_receiver_open_metrics_test.go:85: skipping failing negative OpenMetrics parser tests

=== SKIP: . TestOpenMetricsFail/bad_grouping_or_ordering_7 (0.00s)
    metrics_receiver_open_metrics_test.go:85: skipping failing negative OpenMetrics parser tests

=== SKIP: . TestOpenMetricsFail/bad_stateset_info_values_1 (0.00s)
    metrics_receiver_open_metrics_test.go:85: skipping failing negative OpenMetrics parser tests

=== SKIP: . TestOpenMetricsFail/bad_grouping_or_ordering_10 (0.00s)
    metrics_receiver_open_metrics_test.go:85: skipping failing negative OpenMetrics parser tests

=== SKIP: . TestOpenMetricsFail/bad_counter_values_2 (0.00s)
    metrics_receiver_open_metrics_test.go:85: skipping failing negative OpenMetrics parser tests

=== SKIP: . TestOpenMetricsFail/bad_exemplar_timestamp_2 (0.00s)
    metrics_receiver_open_metrics_test.go:85: skipping failing negative OpenMetrics parser tests

=== SKIP: . TestOpenMetricsFail/bad_exemplars_on_unallowed_metric_types_2 (0.00s)
    metrics_receiver_open_metrics_test.go:85: skipping failing negative OpenMetrics parser tests

=== SKIP: . TestOpenMetricsFail/bad_clashing_names_1 (0.00s)
    metrics_receiver_open_metrics_test.go:85: skipping failing negative OpenMetrics parser tests

=== SKIP: . TestOpenMetricsFail/bad_grouping_or_ordering_5 (0.00s)
    metrics_receiver_open_metrics_test.go:85: skipping failing negative OpenMetrics parser tests

=== SKIP: . TestOpenMetricsFail/bad_histograms_0 (0.00s)
    metrics_receiver_open_metrics_test.go:85: skipping failing negative OpenMetrics parser tests

=== SKIP: . TestOpenMetricsFail/bad_clashing_names_0 (0.00s)
    metrics_receiver_open_metrics_test.go:85: skipping failing negative OpenMetrics parser tests

=== SKIP: . TestOpenMetricsFail/bad_counter_values_1 (0.00s)
    metrics_receiver_open_metrics_test.go:85: skipping failing negative OpenMetrics parser tests

=== SKIP: . TestOpenMetricsFail/bad_grouping_or_ordering_6 (0.00s)
    metrics_receiver_open_metrics_test.go:85: skipping failing negative OpenMetrics parser tests

=== SKIP: . TestOpenMetricsFail/bad_timestamp_4 (0.00s)
    metrics_receiver_open_metrics_test.go:85: skipping failing negative OpenMetrics parser tests

=== SKIP: . TestOpenMetricsFail/bad_exemplars_on_unallowed_metric_types_1 (0.00s)
    metrics_receiver_open_metrics_test.go:85: skipping failing negative OpenMetrics parser tests

=== SKIP: . TestOpenMetricsFail/bad_counter_values_19 (0.00s)
    metrics_receiver_open_metrics_test.go:85: skipping failing negative OpenMetrics parser tests

=== SKIP: . TestOpenMetricsFail/bad_info_and_stateset_values_0 (0.00s)
    metrics_receiver_open_metrics_test.go:85: skipping failing negative OpenMetrics parser tests

=== SKIP: . TestOpenMetricsFail/bad_grouping_or_ordering_9 (0.00s)
    metrics_receiver_open_metrics_test.go:85: skipping failing negative OpenMetrics parser tests

=== SKIP: . TestOpenMetricsFail/bad_exemplars_on_unallowed_metric_types_0 (0.00s)
    metrics_receiver_open_metrics_test.go:85: skipping failing negative OpenMetrics parser tests

=== SKIP: . TestOpenMetricsFail/bad_histograms_2 (0.00s)
    metrics_receiver_open_metrics_test.go:85: skipping failing negative OpenMetrics parser tests

=== SKIP: . TestOpenMetricsFail/bad_exemplars_on_unallowed_samples_0 (0.00s)
    metrics_receiver_open_metrics_test.go:85: skipping failing negative OpenMetrics parser tests

=== SKIP: . TestOpenMetricsFail/bad_counter_values_15 (0.00s)
    metrics_receiver_open_metrics_test.go:85: skipping failing negative OpenMetrics parser tests

=== SKIP: . TestOpenMetricsFail/bad_counter_values_14 (0.00s)
    metrics_receiver_open_metrics_test.go:85: skipping failing negative OpenMetrics parser tests

=== SKIP: . TestOpenMetricsFail/bad_counter_values_12 (0.00s)
    metrics_receiver_open_metrics_test.go:85: skipping failing negative OpenMetrics parser tests

=== SKIP: . TestOpenMetricsFail/bad_missing_or_invalid_labels_for_a_type_1 (0.00s)
    metrics_receiver_open_metrics_test.go:85: skipping failing negative OpenMetrics parser tests

=== SKIP: . TestOpenMetricsFail/bad_counter_values_10 (0.00s)
    metrics_receiver_open_metrics_test.go:85: skipping failing negative OpenMetrics parser tests

=== SKIP: . TestOpenMetricsFail/bad_timestamp_5 (0.00s)
    metrics_receiver_open_metrics_test.go:85: skipping failing negative OpenMetrics parser tests

=== SKIP: . TestOpenMetricsFail/bad_exemplars_on_unallowed_samples_3 (0.00s)
    metrics_receiver_open_metrics_test.go:85: skipping failing negative OpenMetrics parser tests

=== SKIP: . TestOpenMetricsFail/bad_info_and_stateset_values_1 (0.00s)
    metrics_receiver_open_metrics_test.go:85: skipping failing negative OpenMetrics parser tests

=== SKIP: . TestOpenMetricsFail/bad_histograms_8 (0.00s)
    metrics_receiver_open_metrics_test.go:85: skipping failing negative OpenMetrics parser tests

=== SKIP: . TestOpenMetricsFail/bad_counter_values_13 (0.00s)
    metrics_receiver_open_metrics_test.go:85: skipping failing negative OpenMetrics parser tests

=== SKIP: . TestOpenMetricsFail/bad_unit_6 (0.00s)
    metrics_receiver_open_metrics_test.go:85: skipping failing negative OpenMetrics parser tests

=== SKIP: . TestOpenMetricsFail/bad_grouping_or_ordering_2 (0.00s)
    metrics_receiver_open_metrics_test.go:85: skipping failing negative OpenMetrics parser tests

=== Failed
=== FAIL: .  (0.00s)
panic: test timed out after 5m0s
running tests:
	TestMetricRenaming (7s)

goroutine 10885 [running]:
testing.(*M).startAlarm.func1()
	/home/runner/actions-runner/_work/_tool/go/1.21.9/arm64/src/testing/testing.go:2259 +0x1e0
created by time.goFunc
	/home/runner/actions-runner/_work/_tool/go/1.21.9/arm64/src/time/sleep.go:176 +0x48

goroutine 1 [chan receive]:
testing.(*T).Run(0xc00050b860, {0x413f146, 0x12}, 0x42a73[28](https://github.com/open-telemetry/opentelemetry-collector-contrib/actions/runs/8634182428/job/23669067311#step:12:29))
	/home/runner/actions-runner/_work/_tool/go/1.21.9/arm64/src/testing/testing.go:1649 +0x608
testing.runTests.func1(0x0?)
	/home/runner/actions-runner/_work/_tool/go/1.21.9/arm64/src/testing/testing.go:2054 +0x84
testing.tRunner(0xc00050b860, 0xc00083f928)
	/home/runner/actions-runner/_work/_tool/go/1.21.9/arm64/src/testing/testing.go:1595 +0x1b4
testing.runTests(0xc000652780?, {0x73c76e0, 0x39, 0x39}, {0xc00083f988?, 0xb5a18?, 0x73e2320?})
	/home/runner/actions-runner/_work/_tool/go/1.21.9/arm64/src/testing/testing.go:2052 +0x6e8
testing.(*M).Run(0xc000652780)
	/home/runner/actions-runner/_work/_tool/go/1.21.9/arm64/src/testing/testing.go:1925 +0x9f0
go.uber.org/goleak.VerifyTestMain({0x56b1520, 0xc000652780}, {0xc000837e28, 0x1, 0x1})
	/home/runner/go/pkg/mod/go.uber.org/goleak@v1.3.0/testmain.go:53 +0x4c
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/prometheusreceiver.TestMain(0x41228ab?)
	/home/runner/actions-runner/_work/opentelemetry-collector-contrib/opentelemetry-collector-contrib/receiver/prometheusreceiver/package_test.go:16 +0x11c
main.main()
	_testmain.go:195 +0x2b8

goroutine 7 [select]:
go.opencensus.io/stats/view.(*worker).start(0xc0001b1400)
	/home/runner/go/pkg/mod/go.opencensus.io@v0.24.0/stats/view/worker.go:[29](https://github.com/open-telemetry/opentelemetry-collector-contrib/actions/runs/8634182428/job/23669067311#step:12:30)2 +0x128
created by go.opencensus.io/stats/view.init.0 in goroutine 1
	/home/runner/go/pkg/mod/go.opencensus.io@v0.24.0/stats/view/worker.go:34 +0xf4

goroutine 10769 [select]:
github.com/stretchr/testify/assert.Eventually({0x56b1540, 0xc000f77a00}, 0xc000be0e40, 0x0?, 0x0?, {0x0, 0x0, 0x0})
	/home/runner/go/pkg/mod/github.com/stretchr/testify@v1.9.0/assert/assertions.go:1897 +0x1f4
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/prometheusreceiver.waitForScrapeResults(...)
	/home/runner/actions-runner/_work/opentelemetry-collector-contrib/opentelemetry-collector-contrib/receiver/prometheusreceiver/metrics_receiver_helper_test.go:170
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/prometheusreceiver.testComponent(0xc000f77a00, {0xc0026a8968?, 0x1, 0x1}, 0x0, {0xc0009cbe58, 0x1, 0x23d984?})
	/home/runner/actions-runner/_work/opentelemetry-collector-contrib/opentelemetry-collector-contrib/receiver/prometheusreceiver/metrics_receiver_helper_test.go:680 +0x8b8
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/prometheusreceiver.TestMetricRenaming(0x0?)
	/home/runner/actions-runner/_work/opentelemetry-collector-contrib/opentelemetry-collector-contrib/receiver/prometheusreceiver/metrics_reciever_metric_rename_test.go:49 +0x240
testing.tRunner(0xc000f77a00, 0x42a7328)
	/home/runner/actions-runner/_work/_tool/go/1.21.9/arm64/src/testing/testing.go:1595 +0x1b4
created by testing.(*T).Run in goroutine 1
	/home/runner/actions-runner/_work/_tool/go/1.21.9/arm64/src/testing/testing.go:1648 +0x5ec

goroutine 10824 [select]:
github.com/prometheus/prometheus/discovery.(*Manager).sender(0xc0004de0d0)
	/home/runner/go/pkg/mod/github.com/prometheus/prometheus@v0.51.2-0.20240405174432-b4a973753c6e/discovery/manager.go:345 +0x128
created by github.com/prometheus/prometheus/discovery.(*Manager).Run in goroutine 10821
	/home/runner/go/pkg/mod/github.com/prometheus/prometheus@v0.51.2-0.20240405174432-b4a973753c6e/discovery/manager.go:181 +0x88

goroutine 10561 [IO wait]:
internal/poll.runtime_pollWait(0xffff543c01f0, 0x72)
	/home/runner/actions-runner/_work/_tool/go/1.21.9/arm64/src/runtime/netpoll.go:343 +0xa0
internal/poll.(*pollDesc).wait(0xc000a78820, 0xc00052e000?, 0x0)
	/home/runner/actions-runner/_work/_tool/go/1.21.9/arm64/src/internal/poll/fd_poll_runtime.go:84 +0xb8
internal/poll.(*pollDesc).waitRead(...)
	/home/runner/actions-runner/_work/_tool/go/1.21.9/arm64/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc000a78800, {0xc00052e000, 0x1000, 0x1000})
	/home/runner/actions-runner/_work/_tool/go/1.21.9/arm64/src/internal/poll/fd_unix.go:164 +0x2e0
net.(*netFD).Read(0xc000a78800, {0xc00052e000, 0x1000, 0x1000})
	/home/runner/actions-runner/_work/_tool/go/1.21.9/arm64/src/net/fd_posix.go:55 +0x48
net.(*conn).Read(0xc00262c328, {0xc00052e000, 0x1000, 0x1000})
	/home/runner/actions-runner/_work/_tool/go/1.21.9/arm64/src/net/net.go:179 +0x8c
net/http.(*connReader).Read(0xc0017538f0, {0xc00052e000, 0x1000, 0x1000})
	/home/runner/actions-runner/_work/_tool/go/1.21.9/arm64/src/net/http/server.go:791 +0x224
bufio.(*Reader).fill(0xc001b360c0)
	/home/runner/actions-runner/_work/_tool/go/1.21.9/arm64/src/bufio/bufio.go:113 +0x21c
bufio.(*Reader).Peek(0xc001b360c0, 0x4)
	/home/runner/actions-runner/_work/_tool/go/1.21.9/arm64/src/bufio/bufio.go:151 +0xb4
net/http.(*conn).serve(0xc000aed8[30](https://github.com/open-telemetry/opentelemetry-collector-contrib/actions/runs/8634182428/job/23669067311#step:12:31), {0x56e9228, 0xc000c357a0})
	/home/runner/actions-runner/_work/_tool/go/1.21.9/arm64/src/net/http/server.go:2044 +0xab0
created by net/http.(*Server).Serve in goroutine 10770
	/home/runner/actions-runner/_work/_tool/go/1.21.9/arm64/src/net/http/server.go:3086 +0x63c

goroutine 10872 [chan receive]:
github.com/prometheus/prometheus/discovery.(*Manager).updater(0xc0004de0d0, {0x56e9260, 0xc0006b2ff0}, 0xc0005ae150, 0xc0007990e0)
	/home/runner/go/pkg/mod/github.com/prometheus/prometheus@v0.51.2-0.20240405174432-b4a973753c6e/discovery/manager.go:322 +0x418
created by github.com/prometheus/prometheus/discovery.(*Manager).startProvider in goroutine 10769
	/home/runner/go/pkg/mod/github.com/prometheus/prometheus@v0.51.2-0.20240405174432-b4a973753c6e/discovery/manager.go:293 +0x5d8

goroutine 10770 [IO wait]:
internal/poll.runtime_pollWait(0xffff543af670, 0x72)
	/home/runner/actions-runner/_work/_tool/go/1.21.9/arm64/src/runtime/netpoll.go:343 +0xa0
internal/poll.(*pollDesc).wait(0xc0006c92a0, 0x4d501?, 0x0)
	/home/runner/actions-runner/_work/_tool/go/1.21.9/arm64/src/internal/poll/fd_poll_runtime.go:84 +0xb8
internal/poll.(*pollDesc).waitRead(...)
	/home/runner/actions-runner/_work/_tool/go/1.21.9/arm64/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Accept(0xc0006c9280)
	/home/runner/actions-runner/_work/_tool/go/1.21.9/arm64/src/internal/poll/fd_unix.go:611 +0x[31](https://github.com/open-telemetry/opentelemetry-collector-contrib/actions/runs/8634182428/job/23669067311#step:12:32)4
net.(*netFD).accept(0xc0006c9280)
	/home/runner/actions-runner/_work/_tool/go/1.21.9/arm64/src/net/fd_unix.go:172 +0x38
net.(*TCPListener).accept(0xc000914d20)
	/home/runner/actions-runner/_work/_tool/go/1.21.9/arm64/src/net/tcpsock_posix.go:152 +0x3c
net.(*TCPListener).Accept(0xc000914d20)
	/home/runner/actions-runner/_work/_tool/go/1.21.9/arm64/src/net/tcpsock.go:315 +0x68
net/http.(*Server).Serve(0xc001a262d0, {0x56d6508, 0xc000914d20})
	/home/runner/actions-runner/_work/_tool/go/1.21.9/arm64/src/net/http/server.go:3056 +0x44c
net/http/httptest.(*Server).goServe.func1()
	/home/runner/actions-runner/_work/_tool/go/1.21.9/arm64/src/net/http/httptest/server.go:310 +0xbc
created by net/http/httptest.(*Server).goServe in goroutine 10769
	/home/runner/actions-runner/_work/_tool/go/1.21.9/arm64/src/net/http/httptest/server.go:308 +0x9c

goroutine 10884 [select]:
net/http.(*persistConn).writeLoop(0xc00055f440)
	/home/runner/actions-runner/_work/_tool/go/1.21.9/arm64/src/net/http/transport.go:2421 +0x120
created by net/http.(*Transport).dialConn in goroutine 10559
	/home/runner/actions-runner/_work/_tool/go/1.21.9/arm64/src/net/http/transport.go:1777 +0x1bb4

goroutine 10867 [select]:
github.com/prometheus/prometheus/scrape.(*Manager).Run(0xc0005ae000, 0xc000798300)
	/home/runner/go/pkg/mod/github.com/prometheus/prometheus@v0.51.2-0.202404051744[32](https://github.com/open-telemetry/opentelemetry-collector-contrib/actions/runs/8634182428/job/23669067311#step:12:33)-b4a973753c6e/scrape/manager.go:117 +0x128
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/prometheusreceiver.(*pReceiver).initPrometheusComponents.func3()
	/home/runner/actions-runner/_work/opentelemetry-collector-contrib/opentelemetry-collector-contrib/receiver/prometheusreceiver/metrics_receiver.go:327 +0xe8
created by github.com/open-telemetry/opentelemetry-collector-contrib/receiver/prometheusreceiver.(*pReceiver).initPrometheusComponents in goroutine 10769
	/home/runner/actions-runner/_work/opentelemetry-collector-contrib/opentelemetry-collector-contrib/receiver/prometheusreceiver/metrics_receiver.go:323 +0xd30

goroutine 10821 [chan receive]:
github.com/prometheus/prometheus/discovery.(*Manager).Run(0xc0004de0d0)
	/home/runner/go/pkg/mod/github.com/prometheus/prometheus@v0.51.2-0.20240405174432-b4a973753c6e/discovery/manager.go:182 +0xb8
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/prometheusreceiver.(*pReceiver).initPrometheusComponents.func1()
	/home/runner/actions-runner/_work/opentelemetry-collector-contrib/opentelemetry-collector-contrib/receiver/prometheusreceiver/metrics_receiver.go:273 +0x9c
created by github.com/open-telemetry/opentelemetry-collector-contrib/receiver/prometheusreceiver.(*pReceiver).initPrometheusComponents in goroutine 10769
	/home/runner/actions-runner/_work/opentelemetry-collector-contrib/opentelemetry-collector-contrib/receiver/prometheusreceiver/metrics_receiver.go:271 +0x[33](https://github.com/open-telemetry/opentelemetry-collector-contrib/actions/runs/8634182428/job/23669067311#step:12:34)4

goroutine 10883 [IO wait]:
internal/poll.runtime_pollWait(0xffff547ba9e0, 0x72)
	/home/runner/actions-runner/_work/_tool/go/1.21.9/arm64/src/runtime/netpoll.go:[34](https://github.com/open-telemetry/opentelemetry-collector-contrib/actions/runs/8634182428/job/23669067311#step:12:35)3 +0xa0
internal/poll.(*pollDesc).wait(0xc000a787a0, 0xc00067f000?, 0x0)
	/home/runner/actions-runner/_work/_tool/go/1.21.9/arm64/src/internal/poll/fd_poll_runtime.go:84 +0xb8
internal/poll.(*pollDesc).waitRead(...)
	/home/runner/actions-runner/_work/_tool/go/1.21.9/arm64/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc000a78780, {0xc00067f000, 0x1000, 0x1000})
	/home/runner/actions-runner/_work/_tool/go/1.21.9/arm64/src/internal/poll/fd_unix.go:164 +0x2e0
net.(*netFD).Read(0xc000a78780, {0xc00067f000, 0x1000, 0x1000})
	/home/runner/actions-runner/_work/_tool/go/1.21.9/arm64/src/net/fd_posix.go:55 +0x48
net.(*conn).Read(0xc0027dc3a8, {0xc00067f000, 0x1000, 0x1000})
	/home/runner/actions-runner/_work/_tool/go/1.21.9/arm64/src/net/net.go:179 +0x8c
net/http.(*persistConn).Read(0xc00055f440, {0xc00067f000, 0x1000, 0x1000})
	/home/runner/actions-runner/_work/_tool/go/1.21.9/arm64/src/net/http/transport.go:1954 +0xc8
bufio.(*Reader).fill(0xc0007edd40)
	/home/runner/actions-runner/_work/_tool/go/1.21.9/arm64/src/bufio/bufio.go:113 +0x21c
bufio.(*Reader).Peek(0xc0007edd40, 0x1)
	/home/runner/actions-runner/_work/_tool/go/1.21.9/arm64/src/bufio/bufio.go:151 +0xb4
net/http.(*persistConn).readLoop(0xc00055f440)
	/home/runner/actions-runner/_work/_tool/go/1.21.9/arm64/src/net/http/transport.go:2118 +0x268
created by net/http.(*Transport).dialConn in goroutine 10559
	/home/runner/actions-runner/_work/_tool/go/1.21.9/arm64/src/net/http/transport.go:1776 +0x1b48

goroutine 10882 [select]:
github.com/prometheus/prometheus/scrape.(*scrapeLoop).run(0xc000b578c0, 0xc00262c308?)
	/home/runner/go/pkg/mod/github.com/prometheus/prometheus@v0.51.2-0.20240405174432-b4a97[37](https://github.com/open-telemetry/opentelemetry-collector-contrib/actions/runs/8634182428/job/23669067311#step:12:38)53c6e/scrape/scrape.go:1239 +0x4c4
created by github.com/prometheus/prometheus/scrape.(*scrapePool).sync in goroutine 10481
	/home/runner/go/pkg/mod/github.com/prometheus/prometheus@v0.51.2-0.20240405174432-b4a973753c6e/scrape/scrape.go:523 +0xe70

goroutine 10873 [select]:
github.com/prometheus/prometheus/scrape.(*Manager).reloader(0xc0005ae000)
	/home/runner/go/pkg/mod/github.com/prometheus/prometheus@v0.51.2-0.20240405174432-b4a973753c6e/scrape/manager.go:148 +0x158
created by github.com/prometheus/prometheus/scrape.(*Manager).Run in goroutine 10867
	/home/runner/go/pkg/mod/github.com/prometheus/prometheus@v0.51.2-0.20240405174432-b4a973753c6e/scrape/manager.go:115 +0x90
FAIL	github.com/open-telemetry/opentelemetry-collector-contrib/receiver/prometheusreceiver	300.187s

=== FAIL: . TestMetricRenaming (unknown)
coverage: 57.0% of statements

DONE 648 tests, 68 skipped, 2 failures in [38](https://github.com/open-telemetry/opentelemetry-collector-contrib/actions/runs/8634182428/job/23669067311#step:12:39)8.639s
make[1]: Leaving directory '/home/runner/actions-runner/_work/opentelemetry-collector-contrib/opentelemetry-collector-contrib'
Error: Process completed with exit code 2.
@crobert-1 crobert-1 added the needs triage New item requiring triage label Apr 10, 2024
@github-actions GitHub Actions
Copy link
Contributor

Pinging code owners:

See Adding Labels via Comments if you do not have permissions to add labels yourself.

@github-actions github-actions bot added the receiver/prometheus Prometheus receiver label Apr 10, 2024
@crobert-1 crobert-1 added the flaky test a test is flaky label Apr 10, 2024
@dashpole
Copy link
Contributor

Looks like a timeout. Is there a way to increase the timeout?

@dashpole
Copy link
Contributor

I'm not sure if the actuated-arm64 runs will take longer or not

@crobert-1
Copy link
Member Author

My understanding is that test timeouts (other than integration tests) are all governed by the central GOTEST_TIMEOUT makefile variable, so we'd have to increase that if it is only a timeout issue.

It could be tested by opening a draft PR, increase the timeout, and run a number of times to check to make sure it just takes a little longer sometimes. I don't have much context for how strongly the timeout should be held to 5m vs. slightly longer.

The alternative would be seeing if the test itself could be broken up or modified to not take as long to workaround the timeout.

@dashpole
Copy link
Contributor

It looks like the overall test timed out, not the specific promethues test. I think this means the prometheus test had been running for 7 seconds? TestMetricRenaming (7s)

@crobert-1
Copy link
Member Author

+1 frequency: https://github.com/open-telemetry/opentelemetry-collector-contrib/actions/runs/8635682417/job/23674045361?pr=32300

This one shows a different test failing, but the same error message:

	TestLabelRenamingKeepAction (3s)

It looks like this is more of a suite issue than individual test. 👍

Out of curiosity I checked the Ubuntu runner's runtime for these tests:

Running target 'test' in module 'receiver/prometheusreceiver' as part of group 'receiver-2'
make --no-print-directory -C receiver/prometheusreceiver test
/home/runner/work/opentelemetry-collector-contrib/opentelemetry-collector-contrib/.tools/gotestsum --rerun-fails=1 --packages="./..." -- -race -timeout 300s -parallel 4 --tags=""
∅  internal/metadata
✓  internal (8.198s)
✓  . (4m52.755s)

It looks like it's "barely" finishing in time, so it looks like more a timing issue than anything else to me, at this point.

@crobert-1 crobert-1 changed the title [receiver/prometheusreceiver] TestMetricRenaming: Test failing intermittently on actuated-arm64 runs [receiver/prometheusreceiver] Tests hitting timeouts intermittently on actuated-arm64 runs Apr 10, 2024
@crobert-1
Copy link
Member Author

+1 freq: https://github.com/open-telemetry/opentelemetry-collector-contrib/actions/runs/8651025237/job/23720954940

@crobert-1
Copy link
Member Author

+1 freq: https://github.com/open-telemetry/opentelemetry-collector-contrib/actions/runs/8649558220/job/23716228773

@crobert-1
Copy link
Member Author

+1 freq: https://github.com/open-telemetry/opentelemetry-collector-contrib/actions/runs/8648996990/job/23714484932

@crobert-1
Copy link
Member Author

+1 freq: https://github.com/open-telemetry/opentelemetry-collector-contrib/actions/runs/8653744183/job/23729625407?pr=32333

@crobert-1 crobert-1 mentioned this issue Apr 12, 2024
Merged
@crobert-1 crobert-1 removed the needs triage New item requiring triage label Apr 15, 2024
@crobert-1 crobert-1 mentioned this issue Apr 15, 2024
Open
@crobert-1
Copy link
Member Author

+1 freq: https://github.com/open-telemetry/opentelemetry-collector-contrib/actions/runs/8694366585/job/23843290078?pr=32176

@github-actions github-actions bot mentioned this issue Apr 16, 2024
Closed
codeboten added a commit that referenced this issue Apr 17, 2024
@atoulme @codeboten
@dashpole @evan-bradley
[receiver/prometheus] speed up tests by setting skipOffsetting (#32341)
8012101 
Fixes #32298

This PR modifies prometheusreceiver to allow to set the skipOffsetting
option on its scrape option config. This option is private, so it can
only be set by reflection.

This removes the random offset to start added to scraping prometheus
metrics, so we may get faster CI builds.

---------

Co-authored-by: Alex Boten <223565+codeboten@users.noreply.github.com>
Co-authored-by: David Ashpole <dashpole@google.com>
Co-authored-by: Evan Bradley <11745660+evan-bradley@users.noreply.github.com>
rimitchell pushed a commit to rimitchell/opentelemetry-collector-contrib that referenced this issue May 8, 2024
@atoulme @codeboten
@dashpole @evan-bradley @rimitchell
[receiver/prometheus] speed up tests by setting skipOffsetting (open-…
200d5f8 
…telemetry#32341)

Fixes open-telemetry#32298

This PR modifies prometheusreceiver to allow to set the skipOffsetting
option on its scrape option config. This option is private, so it can
only be set by reflection.

This removes the random offset to start added to scraping prometheus
metrics, so we may get faster CI builds.

---------

Co-authored-by: Alex Boten <223565+codeboten@users.noreply.github.com>
Co-authored-by: David Ashpole <dashpole@google.com>
Co-authored-by: Evan Bradley <11745660+evan-bradley@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
flaky test a test is flaky receiver/prometheus Prometheus receiver
Projects
None yet
Milestone
No milestone
2 participants
@dashpole @crobert-1