test-ref tests fail if supplied with --name argument

[mttr]

...but those same tests pass if I just let it run through all of the ref tests.

Failing:

$ ./mach test-ref --name acid1
Running cpu reftests...
Scanning /home/xxx/servo/tests/ref for manifests

        /home/xxx/servo/tests/ref/basic.list [164 tests]
        /home/xxx/servo/tests/ref/csswg/css21/css21.list [2 tests]

running 1 test
test acid1_a.html == acid1_b.html ... FAILED

failures:

---- acid1_a.html == acid1_b.html stdout ----
        task 'acid1_a.html == acid1_b.html' panicked at 'assertion failed: `(left == right) && (right == left)` (left: `exit code: 1`, right: `exit code: 0`)', tests/reftest.rs:257



failures:
    acid1_a.html == acid1_b.html

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured

Running gpu reftests...
Scanning /home/xxx/servo/tests/ref for manifests

        /home/xxx/servo/tests/ref/basic.list [164 tests]
        /home/xxx/servo/tests/ref/csswg/css21/css21.list [2 tests]

running 1 test
test acid1_a.html == acid1_b.html ... FAILED

failures:

---- acid1_a.html == acid1_b.html stdout ----
        task 'acid1_a.html == acid1_b.html' panicked at 'assertion failed: `(left == right) && (right == left)` (left: `exit code: 1`, right: `exit code: 0`)', tests/reftest.rs:257



failures:
    acid1_a.html == acid1_b.html

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured

Reference tests completed in 0.12s

Passing:

$ ./mach test-ref
Running cpu reftests...
Scanning /home/xxx/servo/tests/ref for manifests

        /home/xxx/servo/tests/ref/basic.list [164 tests]
        /home/xxx/servo/tests/ref/csswg/css21/css21.list [2 tests]

running 166 tests
test acid1_a.html == acid1_b.html ... ok
test abs_float_pref_width_a.html == abs_float_pref_width_ref.html ... ok
...

[mttr]

Blocks #4091

[mttr]

Interestingly, if you run the reftest directly (as in $ ./target/reftest-* cpu tests/ref acid1) the tests pass. So this is starting to look like a mach issue after all. I'll have a look!

[mttr]

Okay. Figured it out. At some point I seriously misread something and misunderstood the way test-ref handles name filtering. There is no --test argument (the fact that it coincidentally worked exactly as expected, test failures aside, was a major source of confusion).

Instead, it looks like mach passes unknown arguments to reftest after an -- (so for what I've been doing: ./target/reftest-* cpu tests/ref acid1 -- --test) which are subsequently passed on to servo. If servo doesn't know how to handle the argument, it explodes behind the scenes--failing the test instead of telling us something went wrong (and why).

The good news is that I can at least fix #4091 now.