CR1 exit criteria

[palemieux]

Documents the CR1 exit criteria of the ED1 of the IMSC HRM specification.

DO NOT MERGE

[palemieux]

I plan to expand the list of synthetic tests in the coming days. Your high-level feedback on the exit criteria is welcome.

[himorin]

Do we really want to mark these assets with ED1-CR1?
I assume tests are shared among CRs. Of course, additions or obsoletions could happen, but there might be no need to keep older sets since no one will back and check with older CR.

I would suggest to add a report page even with skelton-ish, which we need to add as a link for CRS publication.
No comment for criteria, compared with WG discussions, but I'd suggest to replace a link for charter with the actual URL but not using github repository.

[css-meeting-bot]

The Timed Text Working Group just discussed CR1 exit criteria w3c/imsc-hrm#56, and agreed to the following:

• SUMMARY: No immediate objections, Pierre to proceed. TTWG to continue reviewing.

The full IRC log of that discussion

<nigel> Subtopic: CR1 exit criteria #56
<nigel> github: https://github.com//pull/56
<nigel> Nigel: This is a draft PR, maybe that's why I did not notice it ahead of the call.
<nigel> Pierre: Walking through this.
<nigel> Nigel: [shares https://github.com//pull/56/files?short_path=37df7cf#diff-37df7cf5e1fb2c2cfdd1d6eed9ffb1ddaa7070ffc160a1bf1aa717ecf60db1cc on screen]
<nigel> Pierre: I thought adding a MD doc was a good way to kick around ideas.
<nigel> .. This lays out how we will demonstrate Implementation Experience.
<nigel> .. The idea is that each IMSC document is accompanied by an assertion that says if it passes or fails.
<nigel> .. The idea is that each document is tested against each implementation and we determine that it
<nigel> .. meets the assertion.
<nigel> .. Features at risk: it occurred to me that the current implementation of imsc-hrm does not support
<nigel> .. image profile, because nobody has asked for it. Unless we can find actual image profile documents,
<nigel> .. from people who use it at scale, we should just remove that feature from IMSC-HRM
<nigel> .. Then there's a bunch of synthetic tests to test the boundaries and features of the HRM.
<nigel> .. If we are happy with this structure, then I'll add more synthetic tests.
<nigel> .. That's my proposal for exit criteria.
<nigel> .. Atsushi asked if we want exit criteria to be different for each CR version.
<nigel> .. In the past I have found that CR exit tests are not necessarily useful for the community, long term.
<nigel> .. Right now the synthetic tests are just in a directory.
<nigel> Cyril: Two questions.
<nigel> .. First, the collection of sample IMSC documents.
<nigel> .. I'm wondering, e.g. for Netflix, how we can contribute them, for IP reasons.
<nigel> Pierre: Thanks for the offer! I would expect Netflix to run the implementation internally and just report
<nigel> .. on the results, I would be satisfied with that.
<nigel> Cyril: Okay, that makes sense.
<nigel> .. Second, do you have a list of features that you want to test?
<nigel> Pierre: Not yet. There are different ways the HRM can be exceeded,
<nigel> .. maximum glyph buffer size, maximum rendering time.
<nigel> .. You'd want a test for each of those.
<nigel> .. There's also IPD, a maximum render time, to test.
<nigel> .. We should also test failure on the first ISD of a sequence, because it's somewhat of a special case.
<nigel> .. We should test the ideograms as having a different rendering rate from non-ideograms.
<nigel> .. We should have test documentations that fail if the glyph buffer is not being used.
<nigel> .. I am planning to clean up the tests for the implementation and make them available.
<nigel> Cyril: I'm wondering if there's a list of those features?
<nigel> Pierre: Yes, we could add a feature test column.
<nigel> Cyril: In other standard bodies we're working on having explicit references from the spec to the tests.
<nigel> .. Tools like bikeshed or respec help, they can generate assert ids, which you add with special markup
<nigel> .. in the spec text. Then you can use those in the naming of the tests.
<nigel> .. Respec has a way to show you the tests, if you cross-reference them properly.
<nigel> .. It may be worth investigating.
<nigel> Pierre: Absolutely, I'll look at that, thanks.
<nigel> Nigel: Elephant in the room - this approach is going to be dependent on the result of the FO Council.
<nigel> .. I think we cannot finalise this approach until we have the FO Council outcome.
<nigel> Atsushi: Usually they have a single call to arrive at their conclusions, so we may hear something at the
<nigel> .. end of this month, at least. Of course it make take time to generate the formal document to be published.
<nigel> .. In any case I believe we can get a decision this month if the pattern follows previous FO Councils.
<nigel> Pierre: By definition the deliberation does not stop our work. Anyway end of month works for
<nigel> .. timing for requesting transition to CR.
<nigel> Nigel: For the collections of documents, you want an assertion for each document whether it passes or fails?
<nigel> Pierre: For the synthetic ones, its our responsibility to generate those.
<nigel> .. For third parties, e.g. Netflix, BBC, it's up to them to say whether they think a document should pass
<nigel> .. or fail, and report whether the implementation agreed with their expectation.
<nigel> .. If it agrees, there's nothing more to do.
<nigel> .. If it does not agree, then we have to determine what to do.
<nigel> .. It could be the content is invalid, an error in the spec, an error in the implementation.
<nigel> Atsushi: One concern with building a test suite for HRM. It needs to have at least
<nigel> .. two generated documents from different implementations.
<nigel> .. Should there be some example of presentation?
<nigel> .. And IMSC-HRM implementation will go over that and generate a pass/fail result.
<nigel> .. If I am correct I think we need to have some descriptive note for our test case?
<nigel> Pierre: My proposal is that we ask folks like BBC, like Netflix and others to test their commercial content.
<nigel> .. That would be their criteria. Content that they use commercially or plan to use commercially.
<nigel> .. We cannot specify any particular style, it's different for every provider.
<nigel> .. That's what's most useful for the industry - it will tell us if the HRM is useful, and right.
<nigel> Atsushi: Usually, just seeing it built against each API call or value definition is ideal. I'm not sure how
<nigel> .. to convince readers of the implementation report that it covers all defined functionality in the HRM.
<nigel> .. Like which test is related to each definition, etc. It's difficult to regenerate the cases.
<nigel> Pierre: I agree it's different than other specs, e.g. API based specs, for sure.
<nigel> .. We cannot control what people create commercially which ultimately is what people care about.
<nigel> .. It does not fit the template of API specs.
<nigel> Nigel: If we were to be able to show that the HRM is successful at selecting between content that plays
<nigel> .. successfully on some player and content that does not, that would be really useful.
<nigel> Pierre: We can fairly easily architect a synthetic test that presents alright but is likely to cause most players
<nigel> .. to have a problem, and is rejected by the HRM.
<nigel> .. Then we could invite folks with players to try it. We could do that too.
<nigel> Nigel: That's the big unknown to me, how well the HRM tracks real world playability - that's really hard to grasp.
<nigel> Pierre: It would be helpful actually to generate synthetic documents.
<nigel> .. Any objection to the overall approach, before I start working on it?
<nigel> Cyril: No, this is fine, but maybe we should wait for the FO Council?
<nigel> .. You might waste effort.
<nigel> Pierre: I think we can address their conclusions later.
<nigel> SUMMARY: No immediate objections, Pierre to proceed. TTWG to continue reviewing.

[css-meeting-bot]

The Timed Text Working Group just discussed IMSC-HRM CR1 Exit Criteria w3c/imsc-hrm#56 (Draft Pull Request), and agreed to the following:

• SUMMARY: Nigel to propose adjusted CR Exit Criteria wording and a location for an IR, Pierre to create an IR and a pull request to take the document to CR.

The full IRC log of that discussion

<nigel> Topic: IMSC-HRM CR1 Exit Criteria #56 (Draft Pull Request)
<nigel> github: https://github.com//pull/56
<nigel> Nigel: Last time we looked at this 28 days ago we said to proceed, and TTWG to continue to review.
<nigel> .. No commits since then.
<nigel> Atsushi: Now we have agreement on the charter, I propose we describe how the criteria meet the new
<nigel> .. charter requirements.
<nigel> .. We had a bunch of discussions and objections on this area so I believe it might be better
<nigel> .. to describe more logically step by step how the charter requirements lead to these exit criteria.
<nigel> .. Otherwise we may have a similar discussion during CR transition request.
<nigel> .. And when we do final AC review to go from PR to Rec.
<nigel> Nigel: I think with the charter wording I'd make a couple of changes.
<nigel> .. I'll do them as suggestions.
<nigel> .. First: implementation -> validating implementation
<nigel> .. Second, use the "Content Implementation" wording from the charter.
<nigel> Pierre: Yes, use the charter language
<nigel> Nigel: The rest looks fine to me, and I appreciate the start on tests.
<nigel> .. I think we need to create an Implementation Report and link to it from the CR.
<nigel> Atsushi: Yes, before the transition request it might be better to include the link in Respec metadata config.
<nigel> .. (I may be getting Respec and Bikeshed mixed up)
<atsushi> https://respec.org/docs/#implementationReportURI
<nigel> Nigel: Either way we need to make sure it is in the document.
<nigel> Atsushi: Pub Rule checks are getting stricter checking all required items.
<nigel> .. Some past publications are now failing for other WGs!
<nigel> Pierre: My main question is who will draft the IR?
<nigel> .. The Exit Criteria go into the spec.
<nigel> .. Do you want me to do it, and if so, where?
<nigel> Nigel: In the past we've always put the IRs into the TTWG wiki.
<nigel> Pierre: Correct. I'm happy to do it, there or somewhere else.
<nigel> Nigel: Yes please.
<nigel> Pierre: Where would you like the stub IR to be?
<nigel> Nigel: I'll take an action to look up what we did and propose a location.
<nigel> Pierre: Thanks, and I'll wait for your comments on the proposed exit criteria.
<nigel> Nigel: Okay. Any more for any more on this topic?
<atsushi> https://w3c.github.io/imsc-hrm/spec/imsc-hrm.html?specStatus=CR
<nigel> Atsushi: Testing current IMSC-HRM spec checklist using CR, it seems we need to define Implementation Experience
<nigel> .. somewhere in the document.
<nigel> Nigel: That's new.
<nigel> Atsushi: We need to fix that in the pull request.
<nigel> Nigel: Make sense to you Pierre?
<nigel> Pierre: No.
<nigel> Nigel: The words "implementation experience" are being picked up by Respec and it wants a definition of them.
<nigel> Pierre: That points to the IR I think.
<nigel> Nigel: Ah, it may well do, makes sense.
<nigel> Pierre: Once we create the IR and put a link, it should fix that I think.
<nigel> Atsushi: That's the second error, the third one is asking for "implementation experience" to be defined.
<nigel> Gary: It does not need to be in this document, it could be in the IR.
<nigel> .. Some place needs to define Implementation Experience.
<nigel> SUMMARY: Nigel to propose adjusted CR Exit Criteria wording and a location for an IR, Pierre to create an IR and a pull request to take the document to CR.
<nigel> Pierre: Thanks

[palemieux]

Replaced by #59 and https://www.w3.org/wiki/TimedText/IMSC-HRM-1ED-Implementation-Report