Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Flaky test in opentelemetry-sdk #3911

Closed
xrmx opened this issue May 7, 2024 · 0 comments · Fixed by #3938
Closed

Flaky test in opentelemetry-sdk #3911

xrmx opened this issue May 7, 2024 · 0 comments · Fixed by #3938
Labels
bug Something isn't working

Comments

@xrmx
Copy link
Contributor

xrmx commented May 7, 2024

Describe your environment

Flaky test in CI, we are asserting that the time will be greater than 500 but in this run it was less than that. We could probably use assertAlmostEqual instead.

______ TestBatchSpanProcessor.test_batch_span_processor_scheduled_delay _______

self = <tests.trace.export.test_export.TestBatchSpanProcessor testMethod=test_batch_span_processor_scheduled_delay>

    def test_batch_span_processor_scheduled_delay(self):
        """Test that spans are exported each schedule_delay_millis"""
        spans_names_list = []
    
        export_event = threading.Event()
        my_exporter = MySpanExporter(
            destination=spans_names_list, export_event=export_event
        )
        start_time = time.time()
        span_processor = export.BatchSpanProcessor(
            my_exporter,
            schedule_delay_millis=500,
        )
    
        # create single span
        resource = Resource.create({})
        _create_start_and_end_span("foo", span_processor, resource)
    
        self.assertTrue(export_event.wait(2))
        export_time = time.time()
        self.assertEqual(len(spans_names_list), 1)
>       self.assertGreaterEqual((export_time - start_time) * 1e3, 500)
E       AssertionError: 499.9737739562988 not greater than or equal to 500

Steps to reproduce

tox -e pypy3-opentelemetry-sdk

What is the expected behavior?

Test will pass everytime

What is the actual behavior?

Test is flaky

Additional context

@xrmx xrmx added the bug Something isn't working label May 7, 2024
xrmx added a commit to xrmx/opentelemetry-python that referenced this issue May 30, 2024
It happened that tests failed because the delay was fired some
microseconds earlier:

>       self.assertGreaterEqual((export_time - start_time) * 1e3, 500)
E       AssertionError: 499.9737739562988 not greater than or equal to 500

To remove some flakyness round to milliseconds the time difference.

Fix open-telemetry#3911
xrmx added a commit to xrmx/opentelemetry-python that referenced this issue Jun 3, 2024
It happened that tests failed because the delay was fired some
microseconds earlier:

>       self.assertGreaterEqual((export_time - start_time) * 1e3, 500)
E       AssertionError: 499.9737739562988 not greater than or equal to 500

We should probably revise all these skip on Windows Pypy once we have a
Python 3.9 baseline and Pypy >= 7.3.12.

Fix open-telemetry#3911
xrmx added a commit to xrmx/opentelemetry-python that referenced this issue Jun 12, 2024
It happened that tests failed because the delay was fired some
microseconds earlier:

>       self.assertGreaterEqual((export_time - start_time) * 1e3, 500)
E       AssertionError: 499.9737739562988 not greater than or equal to 500

Use assertAlmostEqual to accept a similar enough value (places=1) and
avoid too big values.

Fix open-telemetry#3911
xrmx added a commit to xrmx/opentelemetry-python that referenced this issue Jun 12, 2024
It happened that tests failed because the delay was fired some
microseconds earlier:

>       self.assertGreaterEqual((export_time - start_time) * 1e3, 500)
E       AssertionError: 499.9737739562988 not greater than or equal to 500

Use assertAlmostEqual to accept a similar enough value (places=0) and
avoid too big values.

Fix open-telemetry#3911
xrmx added a commit to xrmx/opentelemetry-python that referenced this issue Jun 13, 2024
It happened that tests failed because the delay was fired some
microseconds earlier:

>       self.assertGreaterEqual((export_time - start_time) * 1e3, 500)
E       AssertionError: 499.9737739562988 not greater than or equal to 500

Use assertAlmostEqual to accept a similar enough value (delta=1) and
avoid too big values.

Fix open-telemetry#3911
xrmx added a commit to xrmx/opentelemetry-python that referenced this issue Jun 13, 2024
It happened that tests failed because the delay was fired some
microseconds earlier:

>       self.assertGreaterEqual((export_time - start_time) * 1e3, 500)
E       AssertionError: 499.9737739562988 not greater than or equal to 500

Use assertAlmostEqual to accept a similar enough value (delta=16) and
avoid too big values.

Fix open-telemetry#3911
xrmx added a commit to xrmx/opentelemetry-python that referenced this issue Jun 13, 2024
It happened that tests failed because the delay was fired some
microseconds earlier:

>       self.assertGreaterEqual((export_time - start_time) * 1e3, 500)
E       AssertionError: 499.9737739562988 not greater than or equal to 500

Use assertAlmostEqual to accept a similar enough value (delta=16) and
avoid too big values.

Skip tests on windows pypy because of random huge spikes:

E AssertionError: 2253.103017807007 != 500 within 16 delta (1753.1030178070068 difference)

Fix open-telemetry#3911
xrmx added a commit to xrmx/opentelemetry-python that referenced this issue Jun 13, 2024
It happened that tests failed because the delay was fired some
microseconds earlier:

>       self.assertGreaterEqual((export_time - start_time) * 1e3, 500)
E       AssertionError: 499.9737739562988 not greater than or equal to 500

Use assertAlmostEqual to accept a similar enough value (delta=20) and
avoid too big values.

Skip tests on windows pypy because of random huge spikes:

E AssertionError: 2253.103017807007 != 500 within 20 delta (1749.1030178070068 difference)

Fix open-telemetry#3911
xrmx added a commit to xrmx/opentelemetry-python that referenced this issue Jun 13, 2024
It happened that tests failed because the delay was fired some
microseconds earlier:

>       self.assertGreaterEqual((export_time - start_time) * 1e3, 500)
E       AssertionError: 499.9737739562988 not greater than or equal to 500

Use assertAlmostEqual to accept a similar enough value (delta=25) and
avoid too big values.

Skip tests on windows pypy because of random huge spikes:

E AssertionError: 2253.103017807007 != 500 within 25 delta (1744.1030178070068 difference)

Fix open-telemetry#3911
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant