Rename tests to match w3c/epub-specs#1685

[dlazin]

As explained in w3c/epub-specs#1685, if this is acceptable, in a subsequent PR I will rename other existing tests (not written by me) to match, and add data-test attributes for them too.

[dlazin]

Discussed in face-to-face, and everyone is fine with this. I will merge this PR and rename others in a followup, and will wait for approval on w3c/epub-specs#1685.

[iherman]

The issue was discussed in a meeting on 2021-05-28

• no resolutions were taken

View the transcript

4. testing

See github pull request epub-tests#20, #1685.

Dan Lazin: See Spreadsheet on statements to test

Dan Lazin: state of testing: some are easy, many are tricky. I put together a spreadsheet that breaks down the rs spec at the time into testable normative statement.
… roughly 200 tests that need to be written; we are through about 40 - many of the MUSTS
… what we need is a 'testing guide' that organizes the tests so that people can jump in and contribute
… how to name, organize, track the tests
… we should use data tests attribute for test repo
… here are pull requests
… we could discuss whether to approve and merge the second PR here
… then I can write up a document explaining how to participate in testing
… having trouble in some cases finding reading systems viable to support tests
… I feel behind on testing; we need to chip away at these

Ivan Herman: need to think about how to report results in a consistent format - some sort of data file that includes metadata about the tests
… and we need advanced structure on how to report the testing result
… for audiobooks we had metadata that we could use to produce readable reports

Ivan Herman: See Audiobook test results

Dan Lazin: we are going to have a lot more failures than audiobooks testing

Ivan Herman: CR has to prove that spec is implementable through >2 cited implementations
… not worried about lots of failures, conceptually
… sometimes we get requests to retest after the publication as implementations catch up

dawhe: one example is manifest fallbacks, which is not widely supported across reading systems

Dan Lazin: maybe we need a regular testing meeting to help answer questions as they come up
… for example, SVG requirement is hard to test thoroughly

dawhe: I would attend such a meeting

Tzviya Siegman: we need help putting EPUB 3.3 into epubcheck

Matt Garrish: I've been doing this as we go along

Ivan Herman: what is the role of EPUBcheck in our testing strategy? In some sense it's an implementation
… epubcheck checks content, not RS
… what does the CR mean for the content document?
… with a declarative spec, the approach to testing is to see whether those terms and features are used in the community
… are fallbacks implemented or not? do publishers use those features effectively?
… we should have evidence for each feature in the package document to justify it to be included in the spec

Matt Garrish: we have a lot of features that are rarely used, or only used by a single feature. Does that mean we should deprecade stuff that isn't in wide use?

Ivan Herman: maybe we have to have a list of those features where we have doubts that they are widely used
… and ask the community in advance before we deprecate

Matt Garrish: we did a survey of publishers for 3.1

Ivan Herman: we are obliged to honor backwards compatibility, but don't have to keep features that nobody uses

George Kerscher: pronunciation task force is putting forth a new proposal; we should harmonize our requirements with theirs

Wendy Reid: we don't know if manifest fallbacks are used by anyone - should we do a survey across publishers and reading systems for this type of questionable feature?

Charles LaPierre: if we can't find implementations, we could deprecate those because we can't find 2 implementations
… for Dan's tests, epubtest.org might already have tests or you could use those as a blueprint

Tzviya Siegman: we did an investigation for epub cfi to see if they were in use in Google books
… so it is possible to this sort of survey

Matt Garrish: it gets complicated when we drop specs from authoring but not reading systems or vice-versa so it would have to have zero support anywhere to have no impact

Tzviya Siegman: +1 mgarrish

dawhe: it's a disservice to authors if they have stuff in their content that we know doesn't work
… would rather push a little to far and get blowback than not do anything

Charles LaPierre: new developers have to support legacy code; we shouldn't keep supporting if not necessary

Matt Garrish: if we drop fallbacks, are images in the spine valid? there is a ripple effect to getting rid of stuff

George Kerscher: mathml continues to be poorly implemented, but that's a core component of something we really need and support will slowly improve over time

Ivan Herman: there is no risk for mathml even though implementation needs to be improved

Dan Lazin: if we have a weekly meeting we can go through these topics. epubtest repo had a nested hierarchy - does this organization make sense or should it just be flat?

Dave Cramer: fine with me

Wendy Reid: sounds like we have agreed to a regular testing call. We can discuss organization of tests and how to formulate results.

Ivan Herman: Dan, you own the repository and can organize it the way you want.
… re: the SVG case, this applies to testing html and CSS as well. we need to clearly position ourselves that the reading systems must be compliant with html and css and we are not in a position to test that.
… we can only rely on the fact that reading systems render html

Brady Duga: there is a large provider that uses a bunch of different ways to render

Matt Garrish: do you want us to review the PR?

Dan Lazin: I would like people to review the individual tests to validate my interpretation of the spec
… for the PR, is this the right direction? are we polluting the spec?

Matt Garrish: so the specs will be coupled with the tests

Dan Lazin: yes
… will set of Doodle poll for testing meeting times
… need to be an owner of the spec to approve pr 1685
… would like to do a branch in the test repo