Run Solid test-suite on GitHub Actions

[michielbdejong]

Seems this was somehow forgotten when switching from Travis to GitHub Actions?

[ylebre]

Since the test suite is the newcomer here, I would expect there to be problems with it in a lot of different ways. However, I do believe that if interoperability is the goal, having a good test suite is a very important piece of the puzzle to have in the Solid ecosystem so we can all make sure that the implementations we build are compliant and compatible - something which is very hard to achieve by just reading the specs. We could really use help from the big implementations to help fleshing out the test suite. I fear that without it, the idea of interoperability between Solid implementations will be unreachable for most of us.

[timbl]

@RubenVerborgh When you say

"That's indeed the core issue: we are feature-complete for everything the test suite is testing, which we have verified through other means, yet this is not reflected in the score. While we could make changes to improve our score, it would be studying to the test: such changes wouldn't improve upon our spec-compatibility."

you aren't playing the game of trying constructively to get to a point where we have high quality interoperable spec compliant implementations which pass the test suite. If your score is not perfect, then the collaborative thing to do is triage the failing tests into those where you think the test suite got it wrong, then you are obliged to raise that as an issue. If you find the test suite was right, then you raise an issue on the server. If it is not clear which was right, then you escalate it and the design authority for the spec to (a) get a decision on which was it should go and (b) make the spec much clearer in that question, for example adding the example.

You complained about instability in the test suite early on, but it seems to have several implementations which have aligned with it and stayed aligned with it.

If internally CSS has "other means" of testing itself, then sharing that would of course be very valuable to share those more widely.

[ylebre]

I've opened an issue on the solid-crud-test repo to address the solid-0.1 spec for websockets: solid-contrib/solid-crud-tests#38

I think that the standpoint for the test suite is that it should test according to the spec. Which raises the question, is a draft spec like the websocket api considered spec that should be tested for? If it is the best we have, I would say yes.

[ylebre]

Agreed! I had already opened an issue for exactly that - to separate out the websocket pubsub tests from the regular crud tests. As a quick fix, we have an open PR to add a flag that will skip all the current tests around the pubsub. In due time those tests will be moved into an own set of tests for just the pubsub feature. This should also allow us to replace it to follow the development of the spec around this subject. As mentioned before, the test suite is the newcomer here, so it is expected that it will require a bit more work to accommodate the current ecosystem and future changes to the spec.