pytest parameterized tests with randomly generate names

[cquick01]

Type: Bug

Behaviour

Expected vs. Actual

When running pytest via VSCode, I get the following errors

Running tests (pytest): /home/user/test/tests/test_bytes.py::test_randbytes
Running test with arguments: --rootdir /home/user/test --override-ini junit_family=xunit1 --junit-xml=/tmp/tmp-522739qc0UGxP1z6FD.xml --capture no ./tests/test_bytes.py::test_randbytes
Current working directory: /home/user/test
Workspace directory: /home/user/test
Run completed, parsing output
Test result not found for: ./tests/test_bytes.py::test_randbytes[b8\xef\xef]

Test result not found for: ./tests/test_bytes.py::test_randbytes[\xb4d\xed\xaek\xac\xdbR\xde\xe2\x87\xcc5\x9c]\x95\x02.\xac\xd1)\x8a\xb3\xaf\x15\x87\xad\x83c\x0b\x94Z\x8d\x03\xcd<\xd6\xa0\xe9\xba\x11\x95]

Test result not found for: ./tests/test_bytes.py::test_randbytes[5?5\x9cIGQG\xba\x8a\xdb\xf7\xc8\xec k\x95\x08\xbb \x8c\x1eBb\xd33\x93\x1c\x93\xe6\xc2\x08\x7f\xd1\xa7\x95\xf5m\xca\xaa\xe2\x90\x8c\x97\x80\xfe\xd0\xd2#\xbdI\x1dV3>\x8e7\x01\xffg\r]

Test result not found for: ./tests/test_bytes.py::test_randbytes[k[\x02M\xe2\r*\xcd4\xc2Q+\xadbqP\xca\xac\xc4]

Test result not found for: ./tests/test_bytes.py::test_randbytes[\xdb\x1d\xe8\x04\n\x9b\x0b/.\xd55\xd2\x92\xbf\x05/\xc1\xa0\x06\x95\xd2s\x18\xd1\xed'\xd9Cu\x02S\xe1\x0b\xc4\xa6\x00)\xda\xf1]

Test result not found for: ./tests/test_bytes.py::test_randbytes[\xe4\x94`\xd0\xb9V\x97g%\xafa\xda\xc8\xd1\xbd\xd7\xc5Z\xf7\xb7\xdf\xbdd]

Test result not found for: ./tests/test_bytes.py::test_randbytes[\xb2\x8f]

Test result not found for: ./tests/test_bytes.py::test_randbytes[z:\x1f\xccq`\xd6\x88\xef\x8a\xbc*\x8f\x95\x8e\x8c\xf9O\xf1\xdbo(]

Test result not found for: ./tests/test_bytes.py::test_randbytes[\xd5h[\x7f\xe2\xf6m\xf1]

Test result not found for: ./tests/test_bytes.py::test_randbytes[_\x82p\xac\x07\xc1\xdbO\n\xe1<\x19\xb2\xbbW\x18\x95\x02H\xc8]

Total number of tests expected to run: 10
Total number of tests run: 0
Total number of tests passed: 0
Total number of tests failed: 0
Total number of tests failed with errors: 0
Total number of tests skipped: 0
Total number of tests with no result data: 10
Finished running tests!

But the debug window shows that it passes

============================= test session starts ==============================
platform linux -- Python 3.11.3, pytest-7.4.3, pluggy-1.3.0
rootdir: /home/user/test
collected 10 items

tests/test_bytes.py ..........

------------- generated xml file: /tmp/tmp-522739qc0UGxP1z6FD.xml --------------
============================== 10 passed in 0.10s ==============================

And it works fine when running on command line.

Steps to reproduce:

1. Setup Environment

.
├── poetry.lock
├── pyproject.toml
└── tests
    └── test_bytes.py

test_bytes.py

#!/usr/bin/env python3
import random

import pytest

RANDOM_BYTES = [random.randbytes(random.randint(0, 64)) for _ in range(10)]


@pytest.mark.parametrize("data", RANDOM_BYTES)
def test_randbytes(data):
    foo = data
    assert foo == data

pyproject.toml

[tool.poetry]
name = "test"
version = "0.1.0"
description = ""
authors = ["User <user@domain.tld>"]
readme = "README.md"

[tool.poetry.dependencies]
python = "^3.11"
pytest = "^7.4.3"


[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

2. Install with poetry

pip install poetry  # if needed
poetry lock
poetry install

3. Configure Python interpreter in VSCode

# to get path of environment
poetry env info -p

4. Run tests, errors occur. Running pytest in terminal passes.

Edit: Also, when adding this test with ints, most fail the same way, but two of them seem to pass.

But the debug window still shows all tests pass, 20 this time

============================= test session starts ==============================
platform linux -- Python 3.11.3, pytest-7.4.3, pluggy-1.3.0
rootdir: /home/cquick/test
collected 20 items

tests/test_bytes.py ....................

------------- generated xml file: /tmp/tmp-522739BQWWvKOgqoHN.xml --------------
============================== 20 passed in 0.10s ==============================

Diagnostic data

• Python version (& distribution if applicable, e.g. Anaconda): 3.11.3
• Type of virtual environment used (e.g. conda, venv, virtualenv, etc.): Poetry
• Value of the python.languageServer setting: Pylance

Output for Python in the Output panel (View→Output, change the drop-down the upper-right of the Output panel to Python)

2023-12-13 17:19:53.199 [debug] Clearing context for python dependencies not installed: /home/user/test/pyproject.toml
2023-12-13 17:19:53.202 [debug] Found cached env for /home/user/.cache/pypoetry/virtualenvs/test-zxyBamcN-py3.11/bin/python3
2023-12-13 17:19:53.202 [debug] Found cached env for /home/user/.cache/pypoetry/virtualenvs/test-zxyBamcN-py3.11/bin/python3
2023-12-13 17:19:53.219 [debug] Found cached env for /home/user/.cache/pypoetry/virtualenvs/test-zxyBamcN-py3.11/bin/python3
2023-12-13 17:19:53.234 [info] > ~/.cache/pypoetry/virtualenvs/test-zxyBamcN-py3.11/bin/python3 -m pytest --rootdir . --override-ini junit_family=xunit1 --junit-xml=/tmp/tmp-5227399oU3WLyyfScP.xml
2023-12-13 17:19:53.234 [info] cwd: .
2023-12-13 17:19:57.440 [debug] Clearing context for python dependencies not installed: ms-python.python.Python
2023-12-13 17:19:57.440 [debug] Found cached env for /home/user/.cache/pypoetry/virtualenvs/test-zxyBamcN-py3.11/bin/python3
2023-12-13 17:20:03.697 [debug] Clearing context for python dependencies not installed: /home/user/test/pyproject.toml
2023-12-13 17:20:03.700 [debug] Found cached env for /home/user/.cache/pypoetry/virtualenvs/test-zxyBamcN-py3.11/bin/python3
2023-12-13 17:20:11.268 [debug] Clearing context for python dependencies not installed: ms-python.python.Python
2023-12-13 17:20:11.268 [debug] Found cached env for /home/user/.cache/pypoetry/virtualenvs/test-zxyBamcN-py3.11/bin/python3

User Settings

languageServer: "Pylance"

testing
• pytestEnabled: true

Extension version: 2023.23.13471011
VS Code version: Code - Insiders 1.86.0-insider (f87fb29, 2023-12-13T08:07:18.340Z)
OS version: Windows_NT x64 10.0.19045
Modes:
Remote OS version: Linux x64 5.15.133.1-microsoft-standard-WSL2

System Info

Item	Value
CPUs	Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz (12 x 2592)
GPU Status	2d_canvas: enabled canvas_oop_rasterization: enabled_on direct_rendering_display_compositor: disabled_off_ok gpu_compositing: enabled multiple_raster_threads: enabled_on opengl: enabled_on rasterization: enabled raw_draw: disabled_off_ok skia_graphite: disabled_off video_decode: enabled video_encode: enabled vulkan: disabled_off webgl: enabled webgl2: enabled webgpu: enabled
Load (avg)	undefined
Memory (System)	31.70GB (5.56GB free)
Process Argv
Screen Reader	no
VM	0%

Item	Value
Remote	WSL: Ubuntu-22.04
OS	Linux x64 5.15.133.1-microsoft-standard-WSL2
CPUs	Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz (12 x 2592)
Memory (System)	11.68GB (6.36GB free)
VM	0%

[eleanorjboyd]

Hello, it looks like you are not on the new testing rewrite. Could you enable that and try this again?

It should be turned on for everyone but you might have disabled all experiments so could you check there? If not you could try adding this to your users settings.json "python.experiments.optInto": ["pythonTestAdapter"]. You can check if its enabled by looking at Experiment 'pythonTestAdapter' is active in your python logs. If you are unable to see the log for the experiment change your log level to info or trace and it should show up (can be done via the command palette). Thanks!

[cquick01]

Thanks for the response @eleanorjboyd . I enabled the pythonTestAdapter experiment but am still running into issues running the tests.

I see that the experiment is now activated

2023-12-15 09:48:03.524 [info] Telemetry level is off
2023-12-15 09:48:03.524 [info] Experiments are disabled, only manually opted experiments are active.
2023-12-15 09:48:03.524 [info] Experiment 'pythonTestAdapter' is active
2023-12-15 09:48:03.722 [info] Default formatter is set to null for workspace /home/user/test
2023-12-15 09:48:04.805 [info] Test server listening.
2023-12-15 09:48:05.027 [info] > ~/.cache/pypoetry/virtualenvs/test-zxyBamcN-py3.11/bin/python3 -I ~/.vscode-server-insiders/extensions/ms-python.python-2023.23.13481009/pythonFiles/get_output_via_markers.py ~/.vscode-server-insiders/extensions/ms-python.python-2023.23.13481009/pythonFiles/interpreterInfo.py
2023-12-15 09:48:05.667 [info] Python interpreter path: ~/.cache/pypoetry/virtualenvs/test-zxyBamcN-py3.11/bin/python3
2023-12-15 09:48:05.968 [info] > conda info --json
2023-12-15 09:48:06.339 [info] > . ~/.cache/pypoetry/virtualenvs/test-zxyBamcN-py3.11/bin/activate && echo 'e8b39361-0157-4923-80e1-22d70d46dee6' && python ~/.vscode-server-insiders/extensions/ms-python.python-2023.23.13481009/pythonFiles/printEnvVariables.py
2023-12-15 09:48:06.339 [info] shell: bash
2023-12-15 09:48:07.403 [info] Starting Pylance language server.

The tests still show errors, although they seem to be different now

CLIENT: Server listening on port 41559...
Received JSON data in run script
Running pytest with args: ['-p', 'vscode_pytest', '--rootdir', '/home/user/test', '/home/user/test/tests/test_bytes.py::test_randints[17]', '/home/user/test/tests/test_bytes.py::test_randints[46]', '/home/user/test/tests/test_bytes.py::test_randints[6]', '/home/user/test/tests/test_bytes.py::test_randints[18]', '/home/user/test/tests/test_bytes.py::test_randints[0]', '/home/user/test/tests/test_bytes.py::test_randints[48]', '/home/user/test/tests/test_bytes.py::test_randints[14]', '/home/user/test/tests/test_bytes.py::test_randints[60]', '/home/user/test/tests/test_bytes.py::test_randints[39]', '/home/user/test/tests/test_bytes.py::test_randints[64]', '/home/user/test/tests/test_bytes.py::test_randbytes[\\x80\\xde}=\\x8c2/7\\x00Y\\x10`\\x0b\\xce%\\xfd\\xbd\\xd0_\\xee\\x11\\xe8rW\\x8c\\xb9\\xf0N\\xa11<f\\xc9u\\xb3\\x92\\x18\\x0c\\xeb\\x99\\x136?\\x13\\xc7\\xda\\xbe\\xee\\xc1t\\x87\\xeaE\\xa7\\xebk\\x14\\x86\\xcdv]', '/home/user/test/tests/test_bytes.py::test_randbytes[c\\xd9\\xd9\\x98|e`V\\xc3\\xce\\x04N\\xfc*\\xfc)o\\xb5\\xa6\\xe6\\xedJ\\x0eo\\x94\\xea\\x17\\xef]', '/home/user/test/tests/test_bytes.py::test_randbytes[\\x9b\\xe4\\xf63yd\\xe1)\\x02\\x87\\x17\\xd3\\xc4\\xc7M\\xacc\\xab\\x1cdJ\\xca\\x04\\\\x81\\n\\xca.\\xe7\\x9f\\x13\\xfd\\xe3\\x86\\xa2c\\xe2\\xfe\\xaf]', '/home/user/test/tests/test_bytes.py::test_randbytes[\\x05\\xd5\\xed\\x12\\xa4!e\\xaf\\x16$\\xfb"<\\x91\\xce\\xd9\\xb9\\xdb`\\xbe\\xfd:\\xf0\\xd8/,\\x84\\xbdC\\xf7\\xdd\\x0c\\x94?\\xe4\\xe0\\xbe]\\xa2\\xb53\\xf6\\xb80\\x93\\xfd\\xbabC\\xfc\\xba\\xe5s<\\x00d]', '/home/user/test/tests/test_bytes.py::test_randbytes[\\xb8\\x0e\\xfcs\\xa2]\\xfe\\xb2\\xbe\\x06\\xd4\\x83\\x8e\\x9a\\xc1\\xee\\xd7K\\nL=^\\xb6\\x99\\xce1N\\xf5\\x06\\x9f\\xebM\\xf0\\xfb\\x9c\\xc0\\x81\\x04i\\x10\\x7f\\xa9]', '/home/user/test/tests/test_bytes.py::test_randbytes[\\x92k\\x11\\x1c\\xb0m\\x89\\x16j\\x02"\\xf3\\xf2\\xc9\\xcb\\x9aG]', '/home/user/test/tests/test_bytes.py::test_randbytes[\\xec"\\nZ\\xbc\\xd5T\\xb1\\xeczX\\xd1\\xee]', '/home/user/test/tests/test_bytes.py::test_randbytes[\\xe2\\xa4\\x84&\\xb2[Hk\\xc67\\xaf\\x13\\x96\\x18\\xa6\\xbd\\x0bC\\x1cS\\xd1\\xde\\x9e\\xfeJ\\x87,\\x00L\\xefr\\xc3\\xfb-\\x81\\xdb\\xf9\\xc5]', '/home/user/test/tests/test_bytes.py::test_randbytes[\\xdf\\xab)u\\x14\\xe2\\x8d\\xe0\\xd8\\x13\\xb5\\xddn\\xe9\\xe3K\\xf0E\\xb1]', '/home/user/test/tests/test_bytes.py::test_randbytes[\\x00i\\xb2\\x08\\x86\\x99\\x98"\\x1bw<\\xf4\\x84[\\xc4\\xb9\\x81\\x9f\\x00q\\xaes\\xce\\xfd\\xb4^\\xbe\\x12\\x8e\\xdcv\\xff|\\x06\\x9d\\x97]']
============================= test session starts ==============================
platform linux -- Python 3.11.3, pytest-7.4.3, pluggy-1.3.0
rootdir: /home/user/test
collected 2 items

============================ no tests ran in 0.01s =============================
ERROR: not found: /home/user/test/tests/test_bytes.py::test_randints[17]
(no name '/home/user/test/tests/test_bytes.py::test_randints[17]' in any of [<Module tests/test_bytes.py>])

ERROR: not found: /home/user/test/tests/test_bytes.py::test_randints[46]
(no name '/home/user/test/tests/test_bytes.py::test_randints[46]' in any of [<Module tests/test_bytes.py>])

ERROR: not found: /home/user/test/tests/test_bytes.py::test_randints[6]
(no name '/home/user/test/tests/test_bytes.py::test_randints[6]' in any of [<Module tests/test_bytes.py>])

ERROR: not found: /home/user/test/tests/test_bytes.py::test_randints[0]
(no name '/home/user/test/tests/test_bytes.py::test_randints[0]' in any of [<Module tests/test_bytes.py>])

ERROR: not found: /home/user/test/tests/test_bytes.py::test_randints[48]
(no name '/home/user/test/tests/test_bytes.py::test_randints[48]' in any of [<Module tests/test_bytes.py>])

ERROR: not found: /home/user/test/tests/test_bytes.py::test_randints[14]
(no name '/home/user/test/tests/test_bytes.py::test_randints[14]' in any of [<Module tests/test_bytes.py>])

ERROR: not found: /home/user/test/tests/test_bytes.py::test_randints[60]
(no name '/home/user/test/tests/test_bytes.py::test_randints[60]' in any of [<Module tests/test_bytes.py>])

ERROR: not found: /home/user/test/tests/test_bytes.py::test_randints[64]
(no name '/home/user/test/tests/test_bytes.py::test_randints[64]' in any of [<Module tests/test_bytes.py>])

ERROR: not found: /home/user/test/tests/test_bytes.py::test_randbytes[\x80\xde}=\x8c2/7\x00Y\x10`\x0b\xce%\xfd\xbd\xd0_\xee\x11\xe8rW\x8c\xb9\xf0N\xa11<f\xc9u\xb3\x92\x18\x0c\xeb\x99\x136?\x13\xc7\xda\xbe\xee\xc1t\x87\xeaE\xa7\xebk\x14\x86\xcdv]
(no name '/home/user/test/tests/test_bytes.py::test_randbytes[\\x80\\xde}=\\x8c2/7\\x00Y\\x10`\\x0b\\xce%\\xfd\\xbd\\xd0_\\xee\\x11\\xe8rW\\x8c\\xb9\\xf0N\\xa11<f\\xc9u\\xb3\\x92\\x18\\x0c\\xeb\\x99\\x136?\\x13\\xc7\\xda\\xbe\\xee\\xc1t\\x87\\xeaE\\xa7\\xebk\\x14\\x86\\xcdv]' in any of [<Module tests/test_bytes.py>])

ERROR: not found: /home/user/test/tests/test_bytes.py::test_randbytes[c\xd9\xd9\x98|e`V\xc3\xce\x04N\xfc*\xfc)o\xb5\xa6\xe6\xedJ\x0eo\x94\xea\x17\xef]
(no name '/home/user/test/tests/test_bytes.py::test_randbytes[c\\xd9\\xd9\\x98|e`V\\xc3\\xce\\x04N\\xfc*\\xfc)o\\xb5\\xa6\\xe6\\xedJ\\x0eo\\x94\\xea\\x17\\xef]' in any of [<Module tests/test_bytes.py>])

ERROR: not found: /home/user/test/tests/test_bytes.py::test_randbytes[\x9b\xe4\xf63yd\xe1)\x02\x87\x17\xd3\xc4\xc7M\xacc\xab\x1cdJ\xca\x04\\x81\n\xca.\xe7\x9f\x13\xfd\xe3\x86\xa2c\xe2\xfe\xaf]
(no name '/home/user/test/tests/test_bytes.py::test_randbytes[\\x9b\\xe4\\xf63yd\\xe1)\\x02\\x87\\x17\\xd3\\xc4\\xc7M\\xacc\\xab\\x1cdJ\\xca\\x04\\\\x81\\n\\xca.\\xe7\\x9f\\x13\\xfd\\xe3\\x86\\xa2c\\xe2\\xfe\\xaf]' in any of [<Module tests/test_bytes.py>])

ERROR: not found: /home/user/test/tests/test_bytes.py::test_randbytes[\x05\xd5\xed\x12\xa4!e\xaf\x16$\xfb"<\x91\xce\xd9\xb9\xdb`\xbe\xfd:\xf0\xd8/,\x84\xbdC\xf7\xdd\x0c\x94?\xe4\xe0\xbe]\xa2\xb53\xf6\xb80\x93\xfd\xbabC\xfc\xba\xe5s<\x00d]
(no name '/home/user/test/tests/test_bytes.py::test_randbytes[\\x05\\xd5\\xed\\x12\\xa4!e\\xaf\\x16$\\xfb"<\\x91\\xce\\xd9\\xb9\\xdb`\\xbe\\xfd:\\xf0\\xd8/,\\x84\\xbdC\\xf7\\xdd\\x0c\\x94?\\xe4\\xe0\\xbe]\\xa2\\xb53\\xf6\\xb80\\x93\\xfd\\xbabC\\xfc\\xba\\xe5s<\\x00d]' in any of [<Module tests/test_bytes.py>])

ERROR: not found: /home/user/test/tests/test_bytes.py::test_randbytes[\xb8\x0e\xfcs\xa2]\xfe\xb2\xbe\x06\xd4\x83\x8e\x9a\xc1\xee\xd7K\nL=^\xb6\x99\xce1N\xf5\x06\x9f\xebM\xf0\xfb\x9c\xc0\x81\x04i\x10\x7f\xa9]
(no name '/home/user/test/tests/test_bytes.py::test_randbytes[\\xb8\\x0e\\xfcs\\xa2]\\xfe\\xb2\\xbe\\x06\\xd4\\x83\\x8e\\x9a\\xc1\\xee\\xd7K\\nL=^\\xb6\\x99\\xce1N\\xf5\\x06\\x9f\\xebM\\xf0\\xfb\\x9c\\xc0\\x81\\x04i\\x10\\x7f\\xa9]' in any of [<Module tests/test_bytes.py>])

ERROR: not found: /home/user/test/tests/test_bytes.py::test_randbytes[\x92k\x11\x1c\xb0m\x89\x16j\x02"\xf3\xf2\xc9\xcb\x9aG]
(no name '/home/user/test/tests/test_bytes.py::test_randbytes[\\x92k\\x11\\x1c\\xb0m\\x89\\x16j\\x02"\\xf3\\xf2\\xc9\\xcb\\x9aG]' in any of [<Module tests/test_bytes.py>])

ERROR: not found: /home/user/test/tests/test_bytes.py::test_randbytes[\xec"\nZ\xbc\xd5T\xb1\xeczX\xd1\xee]
(no name '/home/user/test/tests/test_bytes.py::test_randbytes[\\xec"\\nZ\\xbc\\xd5T\\xb1\\xeczX\\xd1\\xee]' in any of [<Module tests/test_bytes.py>])

ERROR: not found: /home/user/test/tests/test_bytes.py::test_randbytes[\xe2\xa4\x84&\xb2[Hk\xc67\xaf\x13\x96\x18\xa6\xbd\x0bC\x1cS\xd1\xde\x9e\xfeJ\x87,\x00L\xefr\xc3\xfb-\x81\xdb\xf9\xc5]
(no name '/home/user/test/tests/test_bytes.py::test_randbytes[\\xe2\\xa4\\x84&\\xb2[Hk\\xc67\\xaf\\x13\\x96\\x18\\xa6\\xbd\\x0bC\\x1cS\\xd1\\xde\\x9e\\xfeJ\\x87,\\x00L\\xefr\\xc3\\xfb-\\x81\\xdb\\xf9\\xc5]' in any of [<Module tests/test_bytes.py>])

ERROR: not found: /home/user/test/tests/test_bytes.py::test_randbytes[\xdf\xab)u\x14\xe2\x8d\xe0\xd8\x13\xb5\xddn\xe9\xe3K\xf0E\xb1]
(no name '/home/user/test/tests/test_bytes.py::test_randbytes[\\xdf\\xab)u\\x14\\xe2\\x8d\\xe0\\xd8\\x13\\xb5\\xddn\\xe9\\xe3K\\xf0E\\xb1]' in any of [<Module tests/test_bytes.py>])

ERROR: not found: /home/user/test/tests/test_bytes.py::test_randbytes[\x00i\xb2\x08\x86\x99\x98"\x1bw<\xf4\x84[\xc4\xb9\x81\x9f\x00q\xaes\xce\xfd\xb4^\xbe\x12\x8e\xdcv\xff|\x06\x9d\x97]
(no name '/home/user/test/tests/test_bytes.py::test_randbytes[\\x00i\\xb2\\x08\\x86\\x99\\x98"\\x1bw<\\xf4\\x84[\\xc4\\xb9\\x81\\x9f\\x00q\\xaes\\xce\\xfd\\xb4^\\xbe\\x12\\x8e\\xdcv\\xff|\\x06\\x9d\\x97]' in any of [<Module tests/test_bytes.py>])

Finished running tests!

[eleanorjboyd]

ah I see now, we have been having some issues related to running pytest parameterized tests that generate different names for each run. The extension is looking for a given name but as the random number generates a new one, the name changes and it struggles to find what test to run. Im in the process of investigating, Ill update here with the results or another issue to reference. Thanks

[eleanorjboyd]

Hi! Gave this some investigation. It seems right now that running parameterize tests with renamed tests is not feasible. How we handle tests is via referencing all the test items and their IDs in order to handle UI like run, the pass or fail checkmarks etc. Without a stable id we can use to reference these items, we cannot run them from our UI. If the IDs were going to be the same random number every time it would work. So for example if you generate a set of random numbers, save these as constants then use that set of numbers (which will be stable between runs) for every run it will work but right now test ids which change between runs cannot be supported. Thanks