Should the SVG 2 test suite use Shepherd?

[nikosandronikos]

In the past, the working group planned to host our test suite on Web Platform Tests, and use Shepherd.

Do we want to use Shepherd?

What does using Shepherd give us?

[nikosandronikos]

It looks like the following is gained by using Shepherd:

• widgets for specs indicating test results
• a way for users to access and execute tests via the spec
• a management interface (less useful now web-platform-tests is hosted on Github?)

Things I'm not clear about:

• Does registering with Shepherd provide any automatic execution of the test harness with capturing of results? Or are the results it captures purely based on users registering results?

[gsnedders]

To start with, a major point: the CSS WG has two (almost) entirely distinct systems, the Test Harness, which provides a UI to run tests and stores results, and Shepherd, the Test Suite Manager, which is essentially a glorified issue tracker for the testsuite (it has some other functions, but they're not really relevant here, as they mostly relate to storing data to compute cross-spec cross-references). Your above points all pertain to the Test Harness, rather than Shepherd.

Now:

The CSS WG has essentially decided to move away from Shepherd for issue tracking in favour of GitHub, so the SVG WG would become the only users, and I think everyone involved in web-platform-tests would prefer to have a single issue tracker for the repo! So I think the answer wrt Shepherd is a clear no, don't use it.

The Test Harness, on the other hand, is a thorn in my back. :) In short, it is currently reliant on the csswg-test build system, which the CSS WG want to get rid of, but can't because the CSS WG's tools rely on it. I suspect we're going to end up with a from-scratch rewrite, which I expect will end up being my major 4Q project. I think we're going to end up discussing what exactly it should be on Thursday/Friday at TPAC, so I'd suggest you come along and thrown your 2¢ into the bucket.

With regards to your question above: the Test Harness has, in theory, support for (whitelisted) users to bulk-upload results. In practice, it relies on users running all the tests in it. One thing we absolutely want a new system to do is be able to take results submitted from browser's CI systems, which should be running the vast majority of the tests (this obviously isn't workable currently for csswg-test given only MS are running any large part of it!).

[nikosandronikos]

It's pretty clear we do not want to use Shepherd. Most of the functions of Shepherd have been replaced by hosting on Github.