Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Flaky test - prometheusremotewriteexporter - Test_PushMetrics/WAL #9124

Closed
jpkrohling opened this issue Apr 8, 2022 · 13 comments
Closed

Flaky test - prometheusremotewriteexporter - Test_PushMetrics/WAL #9124

jpkrohling opened this issue Apr 8, 2022 · 13 comments
Assignees
@kovrus
Labels
bug Something isn't working closed as inactive comp:prometheus Prometheus related issues exporter/prometheusremotewrite flaky test a test is flaky Stale

Comments

@jpkrohling
Copy link
Member

Seen here: https://github.com/open-telemetry/opentelemetry-collector-contrib/runs/5884764908?check_suite_focus=true

2022-04-08T12:16:18.3480658Z === RUN   Test_PushMetrics
2022-04-08T12:16:18.3480983Z === RUN   Test_PushMetrics/WAL
2022-04-08T12:16:18.3481310Z === RUN   Test_PushMetrics/WAL/invalid_type_case
2022-04-08T12:16:18.3481690Z === PAUSE Test_PushMetrics/WAL/invalid_type_case
2022-04-08T12:16:18.3482052Z === RUN   Test_PushMetrics/WAL/intSum_case
2022-04-08T12:16:18.3507241Z === PAUSE Test_PushMetrics/WAL/intSum_case
2022-04-08T12:16:18.3507709Z === RUN   Test_PushMetrics/WAL/doubleSum_case
2022-04-08T12:16:18.3508092Z === PAUSE Test_PushMetrics/WAL/doubleSum_case
2022-04-08T12:16:18.3508475Z === RUN   Test_PushMetrics/WAL/doubleGauge_case
2022-04-08T12:16:18.3508824Z === PAUSE Test_PushMetrics/WAL/doubleGauge_case
2022-04-08T12:16:18.3509195Z === RUN   Test_PushMetrics/WAL/intGauge_case
2022-04-08T12:16:18.3509564Z === PAUSE Test_PushMetrics/WAL/intGauge_case
2022-04-08T12:16:18.3509924Z === RUN   Test_PushMetrics/WAL/histogram_case
2022-04-08T12:16:18.3510285Z === PAUSE Test_PushMetrics/WAL/histogram_case
2022-04-08T12:16:18.3510646Z === RUN   Test_PushMetrics/WAL/summary_case
2022-04-08T12:16:18.3511005Z === PAUSE Test_PushMetrics/WAL/summary_case
2022-04-08T12:16:18.3511383Z === RUN   Test_PushMetrics/WAL/unmatchedBoundBucketHist_case
2022-04-08T12:16:18.3511809Z === PAUSE Test_PushMetrics/WAL/unmatchedBoundBucketHist_case
2022-04-08T12:16:18.3512190Z === RUN   Test_PushMetrics/WAL/5xx_case
2022-04-08T12:16:18.3512531Z === PAUSE Test_PushMetrics/WAL/5xx_case
2022-04-08T12:16:18.3512888Z === RUN   Test_PushMetrics/WAL/emptyGauge_case
2022-04-08T12:16:18.3513251Z === PAUSE Test_PushMetrics/WAL/emptyGauge_case
2022-04-08T12:16:18.3513641Z === RUN   Test_PushMetrics/WAL/emptyCumulativeSum_case
2022-04-08T12:16:18.3514014Z === PAUSE Test_PushMetrics/WAL/emptyCumulativeSum_case
2022-04-08T12:16:18.3514429Z === RUN   Test_PushMetrics/WAL/emptyCumulativeHistogram_case
2022-04-08T12:16:18.3516746Z === PAUSE Test_PushMetrics/WAL/emptyCumulativeHistogram_case
2022-04-08T12:16:18.3518089Z === RUN   Test_PushMetrics/WAL/emptySummary_case
2022-04-08T12:16:18.3520411Z === PAUSE Test_PushMetrics/WAL/emptySummary_case
2022-04-08T12:16:18.3521538Z === RUN   Test_PushMetrics/WAL/staleNaNIntGauge_case
2022-04-08T12:16:18.3523534Z === PAUSE Test_PushMetrics/WAL/staleNaNIntGauge_case
2022-04-08T12:16:18.3524517Z === RUN   Test_PushMetrics/WAL/staleNaNDoubleGauge_case
2022-04-08T12:16:18.3527591Z === PAUSE Test_PushMetrics/WAL/staleNaNDoubleGauge_case
2022-04-08T12:16:18.3527994Z === RUN   Test_PushMetrics/WAL/staleNaNIntSum_case
2022-04-08T12:16:18.3529621Z === PAUSE Test_PushMetrics/WAL/staleNaNIntSum_case
2022-04-08T12:16:18.3531011Z === RUN   Test_PushMetrics/WAL/staleNaNSum_case
2022-04-08T12:16:18.3532812Z === PAUSE Test_PushMetrics/WAL/staleNaNSum_case
2022-04-08T12:16:18.3554785Z === RUN   Test_PushMetrics/WAL/staleNaNHistogram_case
2022-04-08T12:16:18.3555277Z === PAUSE Test_PushMetrics/WAL/staleNaNHistogram_case
2022-04-08T12:16:18.3555705Z === RUN   Test_PushMetrics/WAL/staleNaNEmptyHistogram_case
2022-04-08T12:16:18.3556163Z === PAUSE Test_PushMetrics/WAL/staleNaNEmptyHistogram_case
2022-04-08T12:16:18.3556564Z === RUN   Test_PushMetrics/WAL/staleNaNSummary_case
2022-04-08T12:16:18.3556950Z === PAUSE Test_PushMetrics/WAL/staleNaNSummary_case
2022-04-08T12:16:18.3557893Z === CONT  Test_PushMetrics/WAL/invalid_type_case
2022-04-08T12:16:18.3558316Z === CONT  Test_PushMetrics/WAL/emptyCumulativeSum_case
2022-04-08T12:16:22.4700997Z === CONT  Test_PushMetrics/WAL/5xx_case
2022-04-08T12:16:26.5700241Z === CONT  Test_PushMetrics/WAL/unmatchedBoundBucketHist_case
2022-04-08T12:16:26.5734642Z === CONT  Test_PushMetrics/WAL/summary_case
2022-04-08T12:16:30.6756277Z === CONT  Test_PushMetrics/WAL/unmatchedBoundBucketHist_case
2022-04-08T12:16:30.6757417Z     testing.go:967: TempDir RemoveAll cleanup: unlinkat /tmp/Test_PushMetricsWALunmatchedBoundBucketHist_case1954787114/001: directory not empty
2022-04-08T12:16:30.6758030Z === CONT  Test_PushMetrics/WAL/histogram_case
2022-04-08T12:16:34.7776159Z === CONT  Test_PushMetrics/WAL/intGauge_case
2022-04-08T12:16:38.8782752Z === CONT  Test_PushMetrics/WAL/emptyGauge_case
2022-04-08T12:16:42.9812311Z === CONT  Test_PushMetrics/WAL/doubleGauge_case
2022-04-08T12:16:47.0825569Z === CONT  Test_PushMetrics/WAL/intSum_case
2022-04-08T12:16:51.1866769Z === CONT  Test_PushMetrics/WAL/doubleSum_case
2022-04-08T12:16:55.2890986Z === CONT  Test_PushMetrics/WAL/staleNaNSummary_case
2022-04-08T12:16:59.3915275Z === CONT  Test_PushMetrics/WAL/staleNaNEmptyHistogram_case
2022-04-08T12:17:03.4936243Z === CONT  Test_PushMetrics/WAL/staleNaNHistogram_case
2022-04-08T12:17:07.5961395Z === CONT  Test_PushMetrics/WAL/staleNaNIntSum_case
2022-04-08T12:17:11.7010754Z === CONT  Test_PushMetrics/WAL/staleNaNSum_case
2022-04-08T12:17:15.8029494Z === CONT  Test_PushMetrics/WAL/staleNaNIntGauge_case
2022-04-08T12:17:19.9059623Z === CONT  Test_PushMetrics/WAL/staleNaNDoubleGauge_case
2022-04-08T12:17:24.0073006Z === CONT  Test_PushMetrics/WAL/emptySummary_case
2022-04-08T12:17:28.1103885Z === CONT  Test_PushMetrics/WAL/emptyCumulativeHistogram_case
2022-04-08T12:17:36.3139815Z === RUN   Test_PushMetrics/NoWAL
2022-04-08T12:17:36.3140832Z === RUN   Test_PushMetrics/NoWAL/invalid_type_case
2022-04-08T12:17:36.3141793Z === PAUSE Test_PushMetrics/NoWAL/invalid_type_case
2022-04-08T12:17:36.3142703Z === RUN   Test_PushMetrics/NoWAL/intSum_case
2022-04-08T12:17:36.3143569Z === PAUSE Test_PushMetrics/NoWAL/intSum_case
2022-04-08T12:17:36.3144462Z === RUN   Test_PushMetrics/NoWAL/doubleSum_case
2022-04-08T12:17:36.3145366Z === PAUSE Test_PushMetrics/NoWAL/doubleSum_case
2022-04-08T12:17:36.3179731Z === RUN   Test_PushMetrics/NoWAL/doubleGauge_case
2022-04-08T12:17:36.3180230Z === PAUSE Test_PushMetrics/NoWAL/doubleGauge_case
2022-04-08T12:17:36.3180642Z === RUN   Test_PushMetrics/NoWAL/intGauge_case
2022-04-08T12:17:36.3181010Z === PAUSE Test_PushMetrics/NoWAL/intGauge_case
2022-04-08T12:17:36.3181397Z === RUN   Test_PushMetrics/NoWAL/histogram_case
2022-04-08T12:17:36.3181791Z === PAUSE Test_PushMetrics/NoWAL/histogram_case
2022-04-08T12:17:36.3182532Z === RUN   Test_PushMetrics/NoWAL/summary_case
2022-04-08T12:17:36.3182924Z === PAUSE Test_PushMetrics/NoWAL/summary_case
2022-04-08T12:17:36.3183369Z === RUN   Test_PushMetrics/NoWAL/unmatchedBoundBucketHist_case
2022-04-08T12:17:36.3183835Z === PAUSE Test_PushMetrics/NoWAL/unmatchedBoundBucketHist_case
2022-04-08T12:17:36.3184253Z === RUN   Test_PushMetrics/NoWAL/5xx_case
2022-04-08T12:17:36.3184616Z === PAUSE Test_PushMetrics/NoWAL/5xx_case
2022-04-08T12:17:36.3185004Z === RUN   Test_PushMetrics/NoWAL/emptyGauge_case
2022-04-08T12:17:36.3185405Z === PAUSE Test_PushMetrics/NoWAL/emptyGauge_case
2022-04-08T12:17:36.3185931Z === RUN   Test_PushMetrics/NoWAL/emptyCumulativeSum_case
2022-04-08T12:17:36.3186375Z === PAUSE Test_PushMetrics/NoWAL/emptyCumulativeSum_case
2022-04-08T12:17:36.3186835Z === RUN   Test_PushMetrics/NoWAL/emptyCumulativeHistogram_case
2022-04-08T12:17:36.3187321Z === PAUSE Test_PushMetrics/NoWAL/emptyCumulativeHistogram_case
2022-04-08T12:17:36.3187764Z === RUN   Test_PushMetrics/NoWAL/emptySummary_case
2022-04-08T12:17:36.3188180Z === PAUSE Test_PushMetrics/NoWAL/emptySummary_case
2022-04-08T12:17:36.3188577Z === RUN   Test_PushMetrics/NoWAL/staleNaNIntGauge_case
2022-04-08T12:17:36.3189005Z === PAUSE Test_PushMetrics/NoWAL/staleNaNIntGauge_case
2022-04-08T12:17:36.3189443Z === RUN   Test_PushMetrics/NoWAL/staleNaNDoubleGauge_case
2022-04-08T12:17:36.3189893Z === PAUSE Test_PushMetrics/NoWAL/staleNaNDoubleGauge_case
2022-04-08T12:17:36.3190316Z === RUN   Test_PushMetrics/NoWAL/staleNaNIntSum_case
2022-04-08T12:17:36.3190730Z === PAUSE Test_PushMetrics/NoWAL/staleNaNIntSum_case
2022-04-08T12:17:36.3191118Z === RUN   Test_PushMetrics/NoWAL/staleNaNSum_case
2022-04-08T12:17:36.3191519Z === PAUSE Test_PushMetrics/NoWAL/staleNaNSum_case
2022-04-08T12:17:36.3191933Z === RUN   Test_PushMetrics/NoWAL/staleNaNHistogram_case
2022-04-08T12:17:36.3192358Z === PAUSE Test_PushMetrics/NoWAL/staleNaNHistogram_case
2022-04-08T12:17:36.3192802Z === RUN   Test_PushMetrics/NoWAL/staleNaNEmptyHistogram_case
2022-04-08T12:17:36.3193252Z === PAUSE Test_PushMetrics/NoWAL/staleNaNEmptyHistogram_case
2022-04-08T12:17:36.3193697Z === RUN   Test_PushMetrics/NoWAL/staleNaNSummary_case
2022-04-08T12:17:36.3194455Z === PAUSE Test_PushMetrics/NoWAL/staleNaNSummary_case
2022-04-08T12:17:36.3194917Z === CONT  Test_PushMetrics/NoWAL/invalid_type_case
2022-04-08T12:17:36.3195341Z === CONT  Test_PushMetrics/NoWAL/staleNaNSummary_case
2022-04-08T12:17:36.3211911Z === CONT  Test_PushMetrics/NoWAL/doubleSum_case
2022-04-08T12:17:36.3242469Z === CONT  Test_PushMetrics/NoWAL/emptyGauge_case
2022-04-08T12:17:36.3293833Z === CONT  Test_PushMetrics/NoWAL/staleNaNEmptyHistogram_case
2022-04-08T12:17:36.3305283Z === CONT  Test_PushMetrics/NoWAL/staleNaNHistogram_case
2022-04-08T12:17:36.3329633Z === CONT  Test_PushMetrics/NoWAL/staleNaNSum_case
2022-04-08T12:17:36.3351280Z === CONT  Test_PushMetrics/NoWAL/staleNaNIntSum_case
2022-04-08T12:17:36.3372656Z === CONT  Test_PushMetrics/NoWAL/staleNaNDoubleGauge_case
2022-04-08T12:17:36.3407080Z === CONT  Test_PushMetrics/NoWAL/staleNaNIntGauge_case
2022-04-08T12:17:36.3426058Z === CONT  Test_PushMetrics/NoWAL/emptyCumulativeHistogram_case
2022-04-08T12:17:36.3476128Z === CONT  Test_PushMetrics/NoWAL/emptyCumulativeSum_case
2022-04-08T12:17:36.3483507Z === CONT  Test_PushMetrics/NoWAL/intGauge_case
2022-04-08T12:17:36.3505688Z === CONT  Test_PushMetrics/NoWAL/emptySummary_case
2022-04-08T12:17:36.3529680Z === CONT  Test_PushMetrics/NoWAL/5xx_case
2022-04-08T12:17:36.3559418Z === CONT  Test_PushMetrics/NoWAL/unmatchedBoundBucketHist_case
2022-04-08T12:17:36.3587772Z === CONT  Test_PushMetrics/NoWAL/histogram_case
2022-04-08T12:17:36.3612171Z === CONT  Test_PushMetrics/NoWAL/doubleGauge_case
2022-04-08T12:17:36.3617018Z === CONT  Test_PushMetrics/NoWAL/summary_case
2022-04-08T12:17:36.3690255Z === CONT  Test_PushMetrics/NoWAL/intSum_case
2022-04-08T12:17:36.3749433Z --- FAIL: Test_PushMetrics (78.03s)
2022-04-08T12:17:36.3750735Z     --- FAIL: Test_PushMetrics/WAL (0.01s)
2022-04-08T12:17:36.3752979Z         --- PASS: Test_PushMetrics/WAL/invalid_type_case (4.11s)
2022-04-08T12:17:36.3754960Z         --- PASS: Test_PushMetrics/WAL/emptyCumulativeSum_case (8.22s)
2022-04-08T12:17:36.3756820Z         --- PASS: Test_PushMetrics/WAL/5xx_case (4.10s)
2022-04-08T12:17:36.3759027Z         --- FAIL: Test_PushMetrics/WAL/unmatchedBoundBucketHist_case (4.11s)
2022-04-08T12:17:36.3760749Z         --- PASS: Test_PushMetrics/WAL/summary_case (8.20s)
2022-04-08T12:17:36.3762337Z         --- PASS: Test_PushMetrics/WAL/histogram_case (8.20s)
2022-04-08T12:17:36.3763917Z         --- PASS: Test_PushMetrics/WAL/intGauge_case (8.20s)
2022-04-08T12:17:36.3765799Z         --- PASS: Test_PushMetrics/WAL/emptyGauge_case (8.20s)
2022-04-08T12:17:36.3767422Z         --- PASS: Test_PushMetrics/WAL/doubleGauge_case (8.21s)
2022-04-08T12:17:36.3768999Z         --- PASS: Test_PushMetrics/WAL/intSum_case (8.21s)
2022-04-08T12:17:36.3770655Z         --- PASS: Test_PushMetrics/WAL/doubleSum_case (8.20s)
2022-04-08T12:17:36.3772330Z         --- PASS: Test_PushMetrics/WAL/staleNaNSummary_case (8.20s)
2022-04-08T12:17:36.3774169Z         --- PASS: Test_PushMetrics/WAL/staleNaNEmptyHistogram_case (8.20s)
2022-04-08T12:17:36.3775932Z         --- PASS: Test_PushMetrics/WAL/staleNaNHistogram_case (8.21s)
2022-04-08T12:17:36.3777653Z         --- PASS: Test_PushMetrics/WAL/staleNaNIntSum_case (8.21s)
2022-04-08T12:17:36.3779299Z         --- PASS: Test_PushMetrics/WAL/staleNaNSum_case (8.20s)
2022-04-08T12:17:36.3781021Z         --- PASS: Test_PushMetrics/WAL/staleNaNIntGauge_case (8.20s)
2022-04-08T12:17:36.3782837Z         --- PASS: Test_PushMetrics/WAL/staleNaNDoubleGauge_case (8.20s)
2022-04-08T12:17:36.3784537Z         --- PASS: Test_PushMetrics/WAL/emptySummary_case (8.20s)
2022-04-08T12:17:36.3786353Z         --- PASS: Test_PushMetrics/WAL/emptyCumulativeHistogram_case (8.20s)
2022-04-08T12:17:36.3787742Z     --- PASS: Test_PushMetrics/NoWAL (0.00s)
2022-04-08T12:17:36.3789374Z         --- PASS: Test_PushMetrics/NoWAL/staleNaNSummary_case (0.01s)
2022-04-08T12:17:36.3791084Z         --- PASS: Test_PushMetrics/NoWAL/invalid_type_case (0.01s)
2022-04-08T12:17:36.3792741Z         --- PASS: Test_PushMetrics/NoWAL/emptyGauge_case (0.00s)
2022-04-08T12:17:36.3794363Z         --- PASS: Test_PushMetrics/NoWAL/doubleSum_case (0.01s)
2022-04-08T12:17:36.3796087Z         --- PASS: Test_PushMetrics/NoWAL/staleNaNHistogram_case (0.00s)
2022-04-08T12:17:36.3798362Z         --- PASS: Test_PushMetrics/NoWAL/staleNaNEmptyHistogram_case (0.01s)
2022-04-08T12:17:36.3800133Z         --- PASS: Test_PushMetrics/NoWAL/staleNaNSum_case (0.00s)
2022-04-08T12:17:36.3801951Z         --- PASS: Test_PushMetrics/NoWAL/staleNaNDoubleGauge_case (0.00s)
2022-04-08T12:17:36.3803741Z         --- PASS: Test_PushMetrics/NoWAL/staleNaNIntSum_case (0.01s)
2022-04-08T12:17:36.3805651Z         --- PASS: Test_PushMetrics/NoWAL/emptyCumulativeHistogram_case (0.00s)
2022-04-08T12:17:36.3807451Z         --- PASS: Test_PushMetrics/NoWAL/staleNaNIntGauge_case (0.01s)
2022-04-08T12:17:36.3809273Z         --- PASS: Test_PushMetrics/NoWAL/emptyCumulativeSum_case (0.00s)
2022-04-08T12:17:36.3811012Z         --- PASS: Test_PushMetrics/NoWAL/emptySummary_case (0.00s)
2022-04-08T12:17:36.3812638Z         --- PASS: Test_PushMetrics/NoWAL/intGauge_case (0.01s)
2022-04-08T12:17:36.3814493Z         --- PASS: Test_PushMetrics/NoWAL/unmatchedBoundBucketHist_case (0.00s)
2022-04-08T12:17:36.3816157Z         --- PASS: Test_PushMetrics/NoWAL/5xx_case (0.01s)
2022-04-08T12:17:36.3817735Z         --- PASS: Test_PushMetrics/NoWAL/histogram_case (0.00s)
2022-04-08T12:17:36.3819355Z         --- PASS: Test_PushMetrics/NoWAL/summary_case (0.01s)
2022-04-08T12:17:36.3820994Z         --- PASS: Test_PushMetrics/NoWAL/doubleGauge_case (0.01s)
2022-04-08T12:17:36.3822586Z         --- PASS: Test_PushMetrics/NoWAL/intSum_case (0.00s)
2022-04-08T12:17:36.3823652Z === RUN   Test_validateAndSanitizeExternalLabels
2022-04-08T12:17:36.3824857Z === RUN   Test_validateAndSanitizeExternalLabels/success_case_no_labels
2022-04-08T12:17:36.3826514Z === RUN   Test_validateAndSanitizeExternalLabels/success_case_with_labels
2022-04-08T12:17:36.3827872Z === RUN   Test_validateAndSanitizeExternalLabels/success_case_2_with_labels
2022-04-08T12:17:36.3829255Z === RUN   Test_validateAndSanitizeExternalLabels/success_case_with_sanitized_labels
2022-04-08T12:17:36.3830656Z === RUN   Test_validateAndSanitizeExternalLabels/labels_that_start_with_underscore
2022-04-08T12:17:36.3832024Z === RUN   Test_validateAndSanitizeExternalLabels/labels_that_start_with_digit
2022-04-08T12:17:36.3833346Z === RUN   Test_validateAndSanitizeExternalLabels/fail_case_empty_label
2022-04-08T12:17:36.3834848Z === RUN   Test_validateAndSanitizeExternalLabels/success_case_no_labels#01
2022-04-08T12:17:36.3836156Z === RUN   Test_validateAndSanitizeExternalLabels/success_case_with_labels#01
2022-04-08T12:17:36.3839150Z === RUN   Test_validateAndSanitizeExternalLabels/success_case_2_with_labels#01
2022-04-08T12:17:36.3840633Z === RUN   Test_validateAndSanitizeExternalLabels/success_case_with_sanitized_labels#01
2022-04-08T12:17:36.3842066Z === RUN   Test_validateAndSanitizeExternalLabels/labels_that_start_with_underscore#01
2022-04-08T12:17:36.3843467Z === RUN   Test_validateAndSanitizeExternalLabels/labels_that_start_with_digit#01
2022-04-08T12:17:36.3844797Z === RUN   Test_validateAndSanitizeExternalLabels/fail_case_empty_label#01
2022-04-08T12:17:36.3846392Z --- PASS: Test_validateAndSanitizeExternalLabels (0.00s)
2022-04-08T12:17:36.3848280Z     --- PASS: Test_validateAndSanitizeExternalLabels/success_case_no_labels (0.00s)
2022-04-08T12:17:36.3850530Z     --- PASS: Test_validateAndSanitizeExternalLabels/success_case_with_labels (0.00s)
2022-04-08T12:17:36.3852524Z     --- PASS: Test_validateAndSanitizeExternalLabels/success_case_2_with_labels (0.00s)
2022-04-08T12:17:36.3854578Z     --- PASS: Test_validateAndSanitizeExternalLabels/success_case_with_sanitized_labels (0.00s)
2022-04-08T12:17:36.3856681Z     --- PASS: Test_validateAndSanitizeExternalLabels/labels_that_start_with_underscore (0.00s)
2022-04-08T12:17:36.3858654Z     --- PASS: Test_validateAndSanitizeExternalLabels/labels_that_start_with_digit (0.00s)
2022-04-08T12:17:36.3860593Z     --- PASS: Test_validateAndSanitizeExternalLabels/fail_case_empty_label (0.00s)
2022-04-08T12:17:36.3862534Z     --- PASS: Test_validateAndSanitizeExternalLabels/success_case_no_labels#01 (0.00s)
2022-04-08T12:17:36.3864491Z     --- PASS: Test_validateAndSanitizeExternalLabels/success_case_with_labels#01 (0.00s)
2022-04-08T12:17:36.3866466Z     --- PASS: Test_validateAndSanitizeExternalLabels/success_case_2_with_labels#01 (0.00s)
2022-04-08T12:17:36.3868557Z     --- PASS: Test_validateAndSanitizeExternalLabels/success_case_with_sanitized_labels#01 (0.00s)
2022-04-08T12:17:36.3870729Z     --- PASS: Test_validateAndSanitizeExternalLabels/labels_that_start_with_underscore#01 (0.00s)
2022-04-08T12:17:36.3872763Z     --- PASS: Test_validateAndSanitizeExternalLabels/labels_that_start_with_digit#01 (0.00s)
2022-04-08T12:17:36.3874720Z     --- PASS: Test_validateAndSanitizeExternalLabels/fail_case_empty_label#01 (0.00s)
2022-04-08T12:17:36.3875869Z === RUN   TestWALOnExporterRoundTrip
2022-04-08T12:17:40.4885034Z --- PASS: TestWALOnExporterRoundTrip (4.11s)
2022-04-08T12:17:40.4886374Z === RUN   Test_createDefaultConfig
2022-04-08T12:17:40.4888340Z --- PASS: Test_createDefaultConfig (0.00s)
2022-04-08T12:17:40.4889214Z === RUN   Test_createMetricsExporter
2022-04-08T12:17:40.4889921Z === RUN   Test_createMetricsExporter/success_case
2022-04-08T12:17:40.4935982Z === RUN   Test_createMetricsExporter/fail_case
2022-04-08T12:17:40.4937308Z === RUN   Test_createMetricsExporter/invalid_config_case
2022-04-08T12:17:40.4938131Z === RUN   Test_createMetricsExporter/invalid_tls_config_case
2022-04-08T12:17:40.4975658Z --- PASS: Test_createMetricsExporter (0.01s)
2022-04-08T12:17:40.4977020Z     --- PASS: Test_createMetricsExporter/success_case (0.00s)
2022-04-08T12:17:40.5069298Z     --- PASS: Test_createMetricsExporter/fail_case (0.00s)
2022-04-08T12:17:40.5070206Z     --- PASS: Test_createMetricsExporter/invalid_config_case (0.00s)
2022-04-08T12:17:40.5070798Z     --- PASS: Test_createMetricsExporter/invalid_tls_config_case (0.00s)
2022-04-08T12:17:40.5071169Z === RUN   Test_batchTimeSeries
2022-04-08T12:17:40.5071495Z === RUN   Test_batchTimeSeries/no_timeseries
2022-04-08T12:17:40.5077567Z === RUN   Test_batchTimeSeries/normal_case
2022-04-08T12:17:40.5078076Z === RUN   Test_batchTimeSeries/two_requests
2022-04-08T12:17:40.5078711Z --- PASS: Test_batchTimeSeries (0.00s)
2022-04-08T12:17:40.5079131Z     --- PASS: Test_batchTimeSeries/no_timeseries (0.00s)
2022-04-08T12:17:40.5079932Z     --- PASS: Test_batchTimeSeries/normal_case (0.00s)
2022-04-08T12:17:40.5080500Z     --- PASS: Test_batchTimeSeries/two_requests (0.00s)
2022-04-08T12:17:40.5080972Z === RUN   TestEnsureTimeseriesPointsAreSortedByTimestamp
2022-04-08T12:17:40.5081581Z --- PASS: TestEnsureTimeseriesPointsAreSortedByTimestamp (0.00s)
2022-04-08T12:17:40.5081930Z === RUN   TestWALCreation_nilConfig
2022-04-08T12:17:40.5082387Z --- PASS: TestWALCreation_nilConfig (0.00s)
2022-04-08T12:17:40.5082688Z === RUN   TestWALCreation_nonNilConfig
2022-04-08T12:17:40.5083161Z --- PASS: TestWALCreation_nonNilConfig (0.00s)
2022-04-08T12:17:40.5083464Z === RUN   TestWALStopManyTimes
2022-04-08T12:17:40.5083904Z --- PASS: TestWALStopManyTimes (0.00s)
2022-04-08T12:17:40.5084154Z === RUN   TestWAL_persist
2022-04-08T12:17:40.5084468Z --- PASS: TestWAL_persist (0.00s)
2022-04-08T12:17:40.5084800Z FAIL
2022-04-08T12:17:40.5085951Z FAIL	github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusremotewriteexporter	82.230s
2022-04-08T12:17:40.5111278Z FAIL
2022-04-08T12:17:40.5231951Z make[2]: *** [../../Makefile.Common:46: test] Error 1
2022-04-08T12:17:40.5233363Z make[2]: Leaving directory '/home/runner/work/opentelemetry-collector-contrib/opentelemetry-collector-contrib/exporter/prometheusremotewriteexporter'
2022-04-08T12:17:40.5234400Z make[1]: *** [Makefile:162: for-all-target-./exporter/prometheusremotewriteexporter] Error 2
2022-04-08T12:17:40.5238879Z make[1]: Leaving directory '/home/runner/work/opentelemetry-collector-contrib/opentelemetry-collector-contrib'
2022-04-08T12:17:40.5241353Z make: *** [Makefile:72: gotest] Error 2
2022-04-08T12:17:40.5259942Z ##[error]Process completed with exit code 2.
@jpkrohling jpkrohling added bug Something isn't working comp:prometheus Prometheus related issues flaky test a test is flaky labels Apr 8, 2022
@jpkrohling
Copy link
Member Author

cc code owners: @anuraaga @Aneurysm9 @alolita

@jpkrohling
Copy link
Member Author

And here: https://github.com/open-telemetry/opentelemetry-collector-contrib/runs/5889572864?check_suite_focus=true

@pmm-sumo
Copy link
Contributor

This seems to be another occurrence of that here: https://github.com/open-telemetry/opentelemetry-collector-contrib/runs/5973225036?check_suite_focus=true

@jpkrohling
Copy link
Member Author

Another one: https://github.com/open-telemetry/opentelemetry-collector-contrib/runs/5976410911?check_suite_focus=true

@gouthamve
Copy link
Member

So I'm looking into this and running this on a M1 Mac:

go test -v . -run=Test_PushMetrics/WAL/intGauge_case -count=10

The above command fails almost always (1-2 failures in the 10 count) with:

    testing.go:1090: TempDir RemoveAll cleanup: unlinkat /var/folders/6n/rn9lc80129z68tqqzd4tyrxm0000gn/T/Test_PushMetricsWALintGauge_case518192279/001/prom_remotewrite: directory not empty

From this thread golang/go#43547, it could be because a background goroutine is writing to the directory even after the end of the test. I tried using goleak to check:

diff --git a/exporter/prometheusremotewriteexporter/exporter_test.go b/exporter/prometheusremotewriteexporter/exporter_test.go
index 0d14b3ed6..d21752045 100644
--- a/exporter/prometheusremotewriteexporter/exporter_test.go
+++ b/exporter/prometheusremotewriteexporter/exporter_test.go
@@ -29,6 +29,8 @@ import (
 	"github.com/prometheus/prometheus/prompb"
 	"github.com/stretchr/testify/assert"
 	"github.com/stretchr/testify/require"
+	"go.uber.org/goleak"
+
 	"go.opentelemetry.io/collector/component"
 	"go.opentelemetry.io/collector/component/componenttest"
 	"go.opentelemetry.io/collector/config"
@@ -618,7 +620,9 @@ func Test_PushMetrics(t *testing.T) {
 		t.Run(name, func(t *testing.T) {
 			for _, tt := range tests {
 				t.Run(tt.name, func(t *testing.T) {
+					opt := goleak.IgnoreCurrent()
+					defer goleak.VerifyNone(t, opt)
 					server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
 						if tt.reqTestFunc != nil {
 							tt.reqTestFunc(t, r, tt.expectedTimeSeries, tt.isStaleMarker)

And the failures ALL stopped, even with count=100 🙃

I'm a little lost as to what to do next.

@gouthamve
Copy link
Member

gouthamve commented Apr 12, 2022 •
edited
Loading

(ignore the email sent through this comment, there is a t.Parallel() which caused me to detect extra routines. Removing t.Parallel() doesn't detect any leaks 🤦 )

@gouthamve gouthamve mentioned this issue Apr 13, 2022
Closed
@djaglowski
Copy link
Member

Here again: https://github.com/open-telemetry/opentelemetry-collector-contrib/runs/6028883054?check_suite_focus=true#step:8:3832

@djaglowski djaglowski mentioned this issue Apr 14, 2022
Merged
@jpkrohling
Copy link
Member Author

And here: https://github.com/open-telemetry/opentelemetry-collector-contrib/runs/6032198410?check_suite_focus=true

@gouthamve gouthamve mentioned this issue Jun 21, 2022
Closed
6 tasks
@github-actions
Copy link
Contributor

github-actions bot commented Nov 8, 2022

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

@github-actions github-actions bot added the Stale label Nov 8, 2022
@jpkrohling
Copy link
Member Author

@kovrus, would you like to take a look at this one?

@github-actions
Copy link
Contributor

Pinging code owners: @Aneurysm9. See Adding Labels via Comments if you do not have permissions to add labels yourself.

@github-actions
Copy link
Contributor

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

See Adding Labels via Comments if you do not have permissions to add labels yourself.

@github-actions github-actions bot added the Stale label Jan 25, 2023
@github-actions
Copy link
Contributor

This issue has been closed as inactive because it has been stale for 120 days with no activity.

@github-actions github-actions bot closed this as not planned Won't fix, can't repro, duplicate, stale May 26, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@kovrus kovrus

Labels
bug Something isn't working closed as inactive comp:prometheus Prometheus related issues exporter/prometheusremotewrite flaky test a test is flaky Stale
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@jpkrohling @djaglowski @gouthamve @kovrus @pmm-sumo