Not displaying full status when running large amount of tests

[paulacamargo25]

Testing #21306

Running this tests

First run the test with 100, and everything seems to work, then update the range to 100000. The tests start to run, the wheel appears in the previous 100, when the test ends I see that the previous 100 appear as complete, but the new ones appear as if they were never run.

[eleanorjboyd]

Hi! When you switched to 10,000 did you reload the test UI?

[paulacamargo25]

No, i only run the tests again

[eleanorjboyd]

I am still not able to repro this, these are the steps I followed. Could you provide steps or a video?

1. Open inc_dec_example_repo
2. discover tests with pytest
3. change from @pytest.mark.parametrize("num", range(1, 10)) to @pytest.mark.parametrize("num", range(1, 100))
4. do not click save instead just click the "run" button in the test explorer UI (double arrow at the top to run all tests)
5. I also tried using the gutter with the parameterize test to run it which also did not work in repro the bug
6. I see all the 100 tests with the green checkmark

[eleanorjboyd]

@paulacamargo25 to follow up, do you still experience this issue?

[paulacamargo25]

What i did extra is that i run the first tests. yes sure, im going to try again

[eleanorjboyd]

so like:

1. Open inc_dec_example_repo
2. discover tests with pytest
3. run pytest
4. change from @pytest.mark.parametrize("num", range(1, 10)) to @pytest.mark.parametrize("num", range(1, 100))
5. do not click save instead just click the "run" button in the test explorer UI (double arrow at the top to run all tests)
6. I also tried using the gutter with the parameterize test to run it which also did not work in repro the bug
7. I see all the 100 tests with the green checkmark

I tried these steps and still got it working as expected.

[paulacamargo25]

I was able to reproduce the issue, the problem occurs when you save the file and immediately run the tests afterwards.

Screen.Recording.2023-06-05.at.4.47.02.PM.mov

[eleanorjboyd]

Ok sweet I was able to repro the issue. this seems to be the fix so we can test when this is merged to make sure. Thanks!

[eleanorjboyd]

verification steps:

1. Open a repo with pytest parameterized tests
2. discover tests with pytest
3. run pytest
4. change from @pytest.mark.parametrize("num", range(1, 10)) to @pytest.mark.parametrize("num", range(1, 100))
5. save file
6. click the "run" button in the test explorer UI (double arrow at the top to run all tests)

[alexr00]

I am a python noob and I cannot get my test to be discovered. My steps:

1. pip install pytest
2. In my python project add a test folder and add mytest.py to it.
3. put this in mytest.py:

import pytest


@pytest.mark.parametrize("num", range(1, 100))
def test_odd_even(num):
    assert True

4. In the test view, Configure Python Tests and choose pytest and then my tests folder

No tests are discovered.

[alexr00]

Sorry to add the verification-steps-needed label, I know you have steps, but I am stuck despite the steps 😳.

[eleanorjboyd]

No worries!! I was rushing a bit when I wrote the steps so missed some important parts:

1. In your settings file set "python.experiments.optInto": ["pythonTestAdapter"],
2. Open a repo with pytest parameterized tests or create a new blank repo
3. create a file that starts with test_ or end with _test and add the code @alexr00 included above
4. discover tests with pytest
5. run pytest
6. change from @pytest.mark.parametrize("num", range(1, 10)) to @pytest.mark.parametrize("num", range(1, 100))
7. save file
8. click the "run" button in the test explorer UI (double arrow at the top to run all tests)

[rzhao271]

In Paula's demo, there is no step 7. Clicking the run button seems to save the file for me, and I still see a bug that way:

The green checkmarks seem to be updated more consistently after I change it from range(1, 10) to range(1, 1000), but the counter at the top still says 29/37 tests passed.

[eleanorjboyd]

delaying to next iteration as I have to make some design decisions around how to best render with dirty changes vs cache for pytest.