Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

meson test --repeat doesn't run the expected number of tests #13194

Open
DaanDeMeyer opened this issue May 8, 2024 · 0 comments
Open

meson test --repeat doesn't run the expected number of tests #13194

DaanDeMeyer opened this issue May 8, 2024 · 0 comments
Labels
test targets

Comments

@DaanDeMeyer
Copy link
Contributor

Describe the bug

meson --repeat 1000 --num-processes 50 <test> only runs 163 times.

To Reproduce

~/systemd (main)> SYSTEMD_INTEGRATION_TESTS=1 TEST_MATCH_SUBTEST=".*cryptsetup.*" ../meson/meson.py test -C build --no-rebuild --repeat 1000 --num-processes 50 TEST-70-TPM2
   1/1000 systemd:integration-tests / TEST-70-TPM2           OK               55.48s
   2/1000 systemd:integration-tests / TEST-70-TPM2           OK               56.45s
   3/1000 systemd:integration-tests / TEST-70-TPM2           OK               57.80s
   4/1000 systemd:integration-tests / TEST-70-TPM2           OK               57.81s
   5/1000 systemd:integration-tests / TEST-70-TPM2           OK               59.01s
   6/1000 systemd:integration-tests / TEST-70-TPM2           OK               59.28s
   7/1000 systemd:integration-tests / TEST-70-TPM2           OK               61.07s
   8/1000 systemd:integration-tests / TEST-70-TPM2           OK               61.16s
   9/1000 systemd:integration-tests / TEST-70-TPM2           OK               61.93s
  10/1000 systemd:integration-tests / TEST-70-TPM2           OK               62.05s
  11/1000 systemd:integration-tests / TEST-70-TPM2           OK               62.43s
  12/1000 systemd:integration-tests / TEST-70-TPM2           OK               65.06s
  13/1000 systemd:integration-tests / TEST-70-TPM2           OK               65.10s
  14/1000 systemd:integration-tests / TEST-70-TPM2           OK               65.48s
  15/1000 systemd:integration-tests / TEST-70-TPM2           OK               65.94s
  16/1000 systemd:integration-tests / TEST-70-TPM2           OK               66.86s
  17/1000 systemd:integration-tests / TEST-70-TPM2           OK               67.04s
  18/1000 systemd:integration-tests / TEST-70-TPM2           OK               67.72s
  19/1000 systemd:integration-tests / TEST-70-TPM2           OK               68.01s
  20/1000 systemd:integration-tests / TEST-70-TPM2           OK               68.71s
  21/1000 systemd:integration-tests / TEST-70-TPM2           OK               68.86s
  22/1000 systemd:integration-tests / TEST-70-TPM2           OK               69.02s
  23/1000 systemd:integration-tests / TEST-70-TPM2           OK               69.75s
  24/1000 systemd:integration-tests / TEST-70-TPM2           OK               70.92s
  25/1000 systemd:integration-tests / TEST-70-TPM2           OK               70.69s
  26/1000 systemd:integration-tests / TEST-70-TPM2           OK               72.66s
  27/1000 systemd:integration-tests / TEST-70-TPM2           OK               72.86s
  28/1000 systemd:integration-tests / TEST-70-TPM2           OK               73.22s
  29/1000 systemd:integration-tests / TEST-70-TPM2           OK               74.83s
  30/1000 systemd:integration-tests / TEST-70-TPM2           OK               75.16s
  31/1000 systemd:integration-tests / TEST-70-TPM2           OK               75.62s
  32/1000 systemd:integration-tests / TEST-70-TPM2           OK               77.10s
  33/1000 systemd:integration-tests / TEST-70-TPM2           OK               78.51s
  34/1000 systemd:integration-tests / TEST-70-TPM2           OK               78.95s
  35/1000 systemd:integration-tests / TEST-70-TPM2           OK               80.27s
  36/1000 systemd:integration-tests / TEST-70-TPM2           OK               81.76s
  37/1000 systemd:integration-tests / TEST-70-TPM2           OK               83.39s
  38/1000 systemd:integration-tests / TEST-70-TPM2           OK               86.86s
  39/1000 systemd:integration-tests / TEST-70-TPM2           OK               88.66s
  40/1000 systemd:integration-tests / TEST-70-TPM2           OK               89.20s
  41/1000 systemd:integration-tests / TEST-70-TPM2           OK               91.51s
  42/1000 systemd:integration-tests / TEST-70-TPM2           OK               92.66s
  43/1000 systemd:integration-tests / TEST-70-TPM2           OK               95.16s
  44/1000 systemd:integration-tests / TEST-70-TPM2           OK               95.34s
  45/1000 systemd:integration-tests / TEST-70-TPM2           OK               98.02s
  46/1000 systemd:integration-tests / TEST-70-TPM2           OK              102.47s
  47/1000 systemd:integration-tests / TEST-70-TPM2           OK              105.15s
  48/1000 systemd:integration-tests / TEST-70-TPM2           OK              106.65s
  49/1000 systemd:integration-tests / TEST-70-TPM2           OK              108.77s
  50/1000 systemd:integration-tests / TEST-70-TPM2           OK              111.83s
  51/1000 systemd:integration-tests / TEST-70-TPM2           OK               62.02s
  52/1000 systemd:integration-tests / TEST-70-TPM2           OK               63.04s
  53/1000 systemd:integration-tests / TEST-70-TPM2           OK               64.32s
  54/1000 systemd:integration-tests / TEST-70-TPM2           OK               67.03s
  55/1000 systemd:integration-tests / TEST-70-TPM2           OK               67.07s
  56/1000 systemd:integration-tests / TEST-70-TPM2           OK               67.44s
  57/1000 systemd:integration-tests / TEST-70-TPM2           OK               67.07s
  58/1000 systemd:integration-tests / TEST-70-TPM2           OK               68.34s
  59/1000 systemd:integration-tests / TEST-70-TPM2           OK               68.75s
  60/1000 systemd:integration-tests / TEST-70-TPM2           OK               68.50s
  61/1000 systemd:integration-tests / TEST-70-TPM2           OK               69.29s
  62/1000 systemd:integration-tests / TEST-70-TPM2           OK               66.87s
  63/1000 systemd:integration-tests / TEST-70-TPM2           OK               67.86s
  64/1000 systemd:integration-tests / TEST-70-TPM2           OK               69.18s
  65/1000 systemd:integration-tests / TEST-70-TPM2           OK               68.99s
  66/1000 systemd:integration-tests / TEST-70-TPM2           OK               69.30s
  67/1000 systemd:integration-tests / TEST-70-TPM2           OK               70.28s
  68/1000 systemd:integration-tests / TEST-70-TPM2           OK               71.51s
  69/1000 systemd:integration-tests / TEST-70-TPM2           OK               71.02s
  70/1000 systemd:integration-tests / TEST-70-TPM2           OK               72.00s
  71/1000 systemd:integration-tests / TEST-70-TPM2           OK               73.35s
  72/1000 systemd:integration-tests / TEST-70-TPM2           OK               73.45s
  73/1000 systemd:integration-tests / TEST-70-TPM2           OK               73.85s
  74/1000 systemd:integration-tests / TEST-70-TPM2           OK               74.34s
  75/1000 systemd:integration-tests / TEST-70-TPM2           OK               74.54s
  76/1000 systemd:integration-tests / TEST-70-TPM2           OK               73.58s
  77/1000 systemd:integration-tests / TEST-70-TPM2           OK               74.52s
  78/1000 systemd:integration-tests / TEST-70-TPM2           OK               75.55s
  79/1000 systemd:integration-tests / TEST-70-TPM2           OK               74.24s
  80/1000 systemd:integration-tests / TEST-70-TPM2           OK               75.18s
  81/1000 systemd:integration-tests / TEST-70-TPM2           OK               75.33s
  82/1000 systemd:integration-tests / TEST-70-TPM2           OK               74.60s
  83/1000 systemd:integration-tests / TEST-70-TPM2           OK               76.53s
  84/1000 systemd:integration-tests / TEST-70-TPM2           OK               76.31s
  85/1000 systemd:integration-tests / TEST-70-TPM2           OK               78.04s
  86/1000 systemd:integration-tests / TEST-70-TPM2           OK               76.03s
  87/1000 systemd:integration-tests / TEST-70-TPM2           OK               79.78s
  88/1000 systemd:integration-tests / TEST-70-TPM2           OK               77.85s
  89/1000 systemd:integration-tests / TEST-70-TPM2           OK               80.62s
  90/1000 systemd:integration-tests / TEST-70-TPM2           OK               83.86s
  91/1000 systemd:integration-tests / TEST-70-TPM2           OK               83.87s
  92/1000 systemd:integration-tests / TEST-70-TPM2           OK               84.56s
  93/1000 systemd:integration-tests / TEST-70-TPM2           OK               82.47s
  94/1000 systemd:integration-tests / TEST-70-TPM2           OK               83.63s
  95/1000 systemd:integration-tests / TEST-70-TPM2           OK               81.07s
  96/1000 systemd:integration-tests / TEST-70-TPM2           OK               78.64s
  97/1000 systemd:integration-tests / TEST-70-TPM2           OK               78.29s
  98/1000 systemd:integration-tests / TEST-70-TPM2           OK               76.73s
  99/1000 systemd:integration-tests / TEST-70-TPM2           OK               76.72s
 100/1000 systemd:integration-tests / TEST-70-TPM2           OK               73.95s
 101/1000 systemd:integration-tests / TEST-70-TPM2           OK               71.09s
 102/1000 systemd:integration-tests / TEST-70-TPM2           OK               70.87s
 103/1000 systemd:integration-tests / TEST-70-TPM2           OK               70.48s
 104/1000 systemd:integration-tests / TEST-70-TPM2           OK               71.26s
 105/1000 systemd:integration-tests / TEST-70-TPM2           OK               71.18s
 106/1000 systemd:integration-tests / TEST-70-TPM2           OK               72.43s
 107/1000 systemd:integration-tests / TEST-70-TPM2           OK               71.75s
 108/1000 systemd:integration-tests / TEST-70-TPM2           OK               77.41s
 109/1000 systemd:integration-tests / TEST-70-TPM2           OK               76.83s
 110/1000 systemd:integration-tests / TEST-70-TPM2           OK               77.47s
 111/1000 systemd:integration-tests / TEST-70-TPM2           OK               79.79s
 112/1000 systemd:integration-tests / TEST-70-TPM2           OK               84.95s
 113/1000 systemd:integration-tests / TEST-70-TPM2           OK               84.18s
 114/1000 systemd:integration-tests / TEST-70-TPM2           OK               93.75s
 115/1000 systemd:integration-tests / TEST-70-TPM2           OK               95.21s
 116/1000 systemd:integration-tests / TEST-70-TPM2           FAIL             60.88s   exit status 1
>>> UBSAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1:print_stacktrace=1 MESON_TEST_ITERATION=139 ASAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1 MSAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1:print_stacktrace=1 MALLOC_PERTURB_=191 /home/daandemeyer/systemd/test/integration-test-wrapper.py --meson-source-dir /home/daandemeyer/systemd --meson-build-dir /home/daandemeyer/systemd/build --test-name TEST-70-TPM2 --test-number 70 --storage volatile --firmware linux --

 117/1000 systemd:integration-tests / TEST-70-TPM2           FAIL             59.62s   exit status 1
>>> UBSAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1:print_stacktrace=1 MALLOC_PERTURB_=13 ASAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1 MSAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1:print_stacktrace=1 MESON_TEST_ITERATION=141 /home/daandemeyer/systemd/test/integration-test-wrapper.py --meson-source-dir /home/daandemeyer/systemd --meson-build-dir /home/daandemeyer/systemd/build --test-name TEST-70-TPM2 --test-number 70 --storage volatile --firmware linux --

 118/1000 systemd:integration-tests / TEST-70-TPM2           OK               98.82s
 119/1000 systemd:integration-tests / TEST-70-TPM2           OK               96.76s
 120/1000 systemd:integration-tests / TEST-70-TPM2           OK              100.11s
 121/1000 systemd:integration-tests / TEST-70-TPM2           OK               97.17s
 122/1000 systemd:integration-tests / TEST-70-TPM2           OK               99.31s
 123/1000 systemd:integration-tests / TEST-70-TPM2           OK               98.12s
 124/1000 systemd:integration-tests / TEST-70-TPM2           OK               99.14s
 125/1000 systemd:integration-tests / TEST-70-TPM2           OK              103.14s
 126/1000 systemd:integration-tests / TEST-70-TPM2           OK              110.80s
 127/1000 systemd:integration-tests / TEST-70-TPM2           OK              118.31s
 128/1000 systemd:integration-tests / TEST-70-TPM2           OK              121.84s
 129/1000 systemd:integration-tests / TEST-70-TPM2           OK              125.26s
 130/1000 systemd:integration-tests / TEST-70-TPM2           OK              126.58s
 131/1000 systemd:integration-tests / TEST-70-TPM2           OK              127.54s
 132/1000 systemd:integration-tests / TEST-70-TPM2           OK              125.91s
 133/1000 systemd:integration-tests / TEST-70-TPM2           OK              126.60s
 134/1000 systemd:integration-tests / TEST-70-TPM2           OK              124.41s
 135/1000 systemd:integration-tests / TEST-70-TPM2           OK              126.01s
 136/1000 systemd:integration-tests / TEST-70-TPM2           OK              125.63s
 137/1000 systemd:integration-tests / TEST-70-TPM2           OK              124.43s
 138/1000 systemd:integration-tests / TEST-70-TPM2           OK              124.16s
 139/1000 systemd:integration-tests / TEST-70-TPM2           OK              127.08s
 140/1000 systemd:integration-tests / TEST-70-TPM2           OK              123.76s
 141/1000 systemd:integration-tests / TEST-70-TPM2           OK              115.70s
 142/1000 systemd:integration-tests / TEST-70-TPM2           OK              117.06s
 143/1000 systemd:integration-tests / TEST-70-TPM2           OK              118.16s
 144/1000 systemd:integration-tests / TEST-70-TPM2           OK              116.15s
 145/1000 systemd:integration-tests / TEST-70-TPM2           OK              118.43s
 146/1000 systemd:integration-tests / TEST-70-TPM2           OK              118.52s
 147/1000 systemd:integration-tests / TEST-70-TPM2           OK              122.52s
 148/1000 systemd:integration-tests / TEST-70-TPM2           OK              123.13s
 149/1000 systemd:integration-tests / TEST-70-TPM2           OK              121.71s
 150/1000 systemd:integration-tests / TEST-70-TPM2           OK              124.56s
 151/1000 systemd:integration-tests / TEST-70-TPM2           OK              122.38s
 152/1000 systemd:integration-tests / TEST-70-TPM2           OK              125.55s
 153/1000 systemd:integration-tests / TEST-70-TPM2           OK              123.43s
 154/1000 systemd:integration-tests / TEST-70-TPM2           OK              122.78s
 155/1000 systemd:integration-tests / TEST-70-TPM2           OK              122.82s
 156/1000 systemd:integration-tests / TEST-70-TPM2           OK              121.56s
 157/1000 systemd:integration-tests / TEST-70-TPM2           OK              121.95s
q 158/1000 systemd:integration-tests / TEST-70-TPM2           OK              120.10s
 159/1000 systemd:integration-tests / TEST-70-TPM2           OK              121.37s
 160/1000 systemd:integration-tests / TEST-70-TPM2           OK              127.26s
 161/1000 systemd:integration-tests / TEST-70-TPM2           OK              120.02s
 162/1000 systemd:integration-tests / TEST-70-TPM2           OK              126.15s
 163/1000 systemd:integration-tests / TEST-70-TPM2           OK              114.98s
 164/1000 systemd:integration-tests / TEST-70-TPM2           OK              114.13s
 165/1000 systemd:integration-tests / TEST-70-TPM2           OK              130.51s

Ok:                 163 
Expected Fail:      0   
Fail:               2   
Unexpected Pass:    0   
Skipped:            0   
Timeout:            0   

Full log written to /home/daandemeyer/systemd/build/meson-logs/testlog.txt

Expected behavior

The test runs 1000 times.

system parameters

  • Is this a cross build or just a plain native build (for the same computer)? no
  • what operating system (e.g. MacOS Catalina, Windows 10, CentOS 8.0, Ubuntu 18.04, etc.) CentOS Stream 9
  • what Python version are you using e.g. 3.8.0 3.9.0
  • what meson --version latest git main
  • what ninja --version if it's a Ninja build 1.10.2
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
test targets
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@DaanDeMeyer @bruchar1