Adopt "test as you commit" policy

[anssiko]

Recently, there has been movement toward requiring a test change whenever a normative
spec change is landed.

For example, the SVG WG Charter has the following text:

The SVG WG will follow a test as you commit approach to specification development, for specifications in CR or above.

I would suggest the DAS WG will consider adopting similar policy for its revised Charter.

[dontcallmedom]

I don't know if I would necessarily codify it, but I would even suggest starting well before CR to apply that policy

[anssiko]

Perhaps we could make this policy "must" for CR and above and "should" at earlier maturity levels. (Not sure if we can use RFC 2119 terms in the Charter -- just to make a point.)

I feel this policy would be very helpful at earlier spec maturity levels too especially if there are implementation(s).

If there's just a spec draft that is undergoing rapid iteration without an implementation the policy could be relaxed to keep the cost of iteration down. At the other end of the spectrum, the policy will help ensure specs do not enter CR too lightly since past that point spec churn will have a higher cost.

I think we do not need to codify these nuances, just do the Right Thing.

[dontcallmedom]

maybe the existence of an implementation ought to be the trigger of making simultaneous test contributions expected?

[foolip]

FWIW, I would leave CR out of the equation unless you think the distinction is really useful. As I understand from @plehegar, it was used for CSS and SVG as a bit of a compromise, and if the group wants to treat all specs the same, they can.

On must vs. should, we've just gone with "highly appreciated" in most places, which I think is a should. I can't really see that must would ever really be appropriate, it's always possible that there will be good reasons to not do testing, or do it later.

Will this issue be discussed at TPAC, or what's the next step?

[anssiko]

@foolip, I personally like the "highly appreciated" wording. Is there some text we could borrow for charter purposes? Otherwise I'll propose something along those lines.

I'm happy to discuss this topic at TPAC too. Note DAS WG does not formally meet at TPAC. Some of us will still be around, so perhaps we could discuss and socialize the idea of "test as you commit" in https://www.w3.org/wiki/TPAC/2017/SessionIdeas#web-platform-tests breakout session, for example.

[foolip]

SVG is the only case I'm aware of where anything's been put into the charter, and it sais "The SVG WG will follow a test as you commit approach to specification development, for specifications in CR or above."

If you're about to recharter I guess that could make sense, otherwise maybe just update all of the repos, or create a group-wide CONTRIBUTING.md that all of the others point to?

If the group isn't meeting at TPAC and you don't expect any dissent, then I don't think there's any reason to wait for TPAC. But it's not up to me, I'm just a cheerleader here :)

[anssiko]

Proposed text to the new DAS Charter: #51

Text borrowed from the SVG WG Charter.

[foolip]

Yay, thanks @anssiko!

I see the CR condition is in #51, what does that mean in practice? Should CONTRIBUTING.md of all individual specs still be updated?

[anssiko]

@foolip, given this Charter is not yet in production, I suggest we test drive with the Generic Sensor API https://github.com/w3c/sensors/blob/master/CONTRIBUTING.md#tests and will get the policy adopted for all the group's deliverables automatically when the new Charter is in effect 1 February 2018 or so.

That said, nothing is preventing us from using the new policy already now for other specifications too. For example, https://www.w3.org/2009/dap/#sensors that inherit from Sensor will logically inherit this policy too.

I wouldn't worry too much about whether the CONTRIBUTING.md of each repo contains this text, we just have to do the right thing :-)