Add CR exit criteria. #180
Conversation
| each test in the <a href="https://w3c.github.io/rdf-canon/tests/">test suite</a> | ||
| MUST be passed by at least three independent implementations. | ||
| Two independent implementations MUST pass all tests.</p> |
There was a problem hiding this comment.
| each test in the <a href="https://w3c.github.io/rdf-canon/tests/">test suite</a> | |
| MUST be passed by at least three independent implementations. | |
| Two independent implementations MUST pass all tests.</p> | |
| there MUST be three independent implementations of every feature defined in the specification. | |
| Two independent implementations MUST pass all tests in the <a href="https://w3c.github.io/rdf-canon/tests/">test suite</a>.</p> |
I felt that the two sentences seemed to partially overlap regarding the test passage. How about revising it as above?
There was a problem hiding this comment.
This is the wording we agreed on at TPAC, but I wasn't sure that it was clear what "every feature defined in the specification" meant, but we do know what the tests are. In other WGs, the criteria were that each test be passed by at least two independent implementations, we've exceeded this by requiring two implementations to pass all tests. The implication to me was that each test be passed by at least three independent implementations.
I'm happy to accept your wording if people feel that the meaning of "every feature ..." is clear.
There was a problem hiding this comment.
I don't think the suggestion above provides any clarification.
A pointer to the TPAC discussion and resolution would be appreciated, as I do not recall these exit criteria.
[@gkellogg] we've exceeded this by requiring two implementations to pass all tests. The implication to me was that each test be passed by at least three independent implementations.
I don't see how requiring two implementations to pass all tests implies requiring any test, never mind each, to be passed by more than two implementations. It seems to me that requiring two implementations to pass all tests only implies requiring two implementations.
That said, I am not sure why we are requiring any implementation to pass all tests, to support all features. This is a massive step beyond previous CR->PR requirements, and I cannot think of any other CR with the same exit criteria!
There was a problem hiding this comment.
This is the link to the resolution from TPAC: https://www.w3.org/2023/09/11-rch-minutes.html#r03
The WG recommends the CR exit criteria be: 1. Three independent implementations of all features, 2. Two independent implementations that each pass all tests
There was a problem hiding this comment.
I am in favour of the original formulation of @gkellogg: "feature" is undefined, but what the individual tests are is clear.
As for
That said, I am not sure why we are requiring any implementation to pass all tests, to support all features. This is a massive step beyond previous CR->PR requirements, and I cannot think of any other CR with the same exit criteria!
Yes, it is a massive step beyond the previous CR-s, but I think if we can fulfill it (and we are fulfilling it already) it gives also a more-than-usual credibility for the value of the algorithm. Because we are talking about a security related algorithm and its implementations, this type of credibility is never too strong...
(Full disclosure: this was my idea at TPAC, ie, I proposed it during the discussion.)
There was a problem hiding this comment.
@gkellogg Thanks, I understand what you intended with your original phrasing. I withdraw my suggestion.
As for "massiveness", while the condition might indeed be a bit strong, given the implementations that have been presented so far, I believe it's within an acceptable range.
There was a problem hiding this comment.
we are fulfilling it already
OK. Somehow I had it in my head that it was fulfilled for the old version's features/tests but not necessarily for the new version's. I have no argument with it, as things actually stand.
There was a problem hiding this comment.
I think this is good with or without @yamdan 's suggestion, both are fine.
| each test in the <a href="https://w3c.github.io/rdf-canon/tests/">test suite</a> | ||
| MUST be passed by at least three independent implementations. | ||
| Two independent implementations MUST pass all tests.</p> |
There was a problem hiding this comment.
I am in favour of the original formulation of @gkellogg: "feature" is undefined, but what the individual tests are is clear.
As for
That said, I am not sure why we are requiring any implementation to pass all tests, to support all features. This is a massive step beyond previous CR->PR requirements, and I cannot think of any other CR with the same exit criteria!
Yes, it is a massive step beyond the previous CR-s, but I think if we can fulfill it (and we are fulfilling it already) it gives also a more-than-usual credibility for the value of the algorithm. Because we are talking about a security related algorithm and its implementations, this type of credibility is never too strong...
(Full disclosure: this was my idea at TPAC, ie, I proposed it during the discussion.)
Adds text to SOTD stating implementation requirements to pass to PR. (Note, this will need to be removed or modified after CR). Somewhat arbitrarily sets the CR Exit date to 2023-11-20.
Preview | Diff