Minutes
- Feb 4, 2019
- Jan 28, 2019
- Jan 21, 2019
- Jan 14, 2019
- Jan 7, 2019
- Dec 17, 2018
- Dec 10, 2018
- Dec 3, 2018
- Nov 26, 2018
- Nov 19, 2018
- Nov 12, 2018
- Nov 5, 2018
- Oct 8, 2018
Participants: Avneesh, Matt, Romain, Tom
Rather slow progress last week. A (big) PR is on its way for implementations of new checks related to the EPUB Structural Semantics Vocabulary.
Question about the prism and msv reserved prefixes: they came out of the AHL stuff; low priority, but they will be registered in EPUBCheck since they're part of the spec.
Discussion about issue #870. This seems to be covered by more generic checks already (reporting RSC-001 or RSC-006). This led to discussing the new restriction on script elements (can't be a remote resource); the restriction is intended for JS script resources but not necessarily to other media types. This is a spec issue that will need to be brought to the CG.
We started to review and assign pending issues labelled as spec: EPUB 3.2.
- Issue #932 (validation of custom elements) needs somed investigation on what we can do and report. Maybe it needs to be postponed to when we better integrate with validator.nu.
- Issues about JS content: there's not much EPUBCheck can do. There's some existing code that will check the JS internals, but the parsing is regex based and quite brittle. We'll probably have to consider these out of scope and close as
wontfix. - Probably nothing to do for the Unicode spec reference update.
Issues review to be continued asynchronously…
Participants: Avneesh, Daniel K, Romain, Tom
The http://validator.idpf.org service has been updated to run EPUBCheck v4.1.1. There are known stability issues, which are not new, the server needs to be restarted more often than not. These will need to be further investigated if the service is to be maintained in the long term.
The master branch was merged to next/v4.2.0, to include the latest fixes from v4.1.1.
The Maven group ID is now org.w3c. Snapshots are automatically published to Sonatype’s OSS Maven repository, when building the next/v4.2.0 branch.
Some progress made on issue #870, tests now need to be written.
A couple pull requests need to be reviewed.
The next version will be v4.2.0-beta, to be released in February. The deadline is Feb 25, but an earlier release is possible. The beta will be nearly feature-complete, not necessarily 100%. A 100% feature-complate release candidate will follow in March.no more hindrance to the release process.
We discussed a few practical development questions:
- the whole
com.adobe.epubcheck.ctcpacakge will be refactored after v4.2.0 (deprecated and merged into the main checker code), so no need to spend much time on improving its API. - how to write new tests; more serious cleanup postponed until after v4.2.0
- the messages and reporting API could be simplified and revamped as well.
Participants: Avneesh, Daniel, Matt, Romain
The target date for the release of 4.1.1 is set to Jan 22 (tomorrow).
A few issues labelled as type: bug have been included to the milestone, to unpile the backlog.
There are a couple remaining issues and PRs to merge, but no hurdles.
4.1.1 was a good opportunity to unpile some old bugs; some were major flaws, some other quite old issues. Most of the remaining issues identified as type: bug are now mostly either planned for 4.2.0, or waiting for feedback, or blocked by bug fixes from 3d party dependencies. The are probably other bugs in the issue backlog, which may just not have been labelled as such.
We wil go through the backlog of issues before the final 4.2.0 release, to make sure everything is properly triaged, and any major issue is tackled in 4.2.0.
A few issues that had been fixed in Matt's Package doc schema update were close; there are 35 issues left, identified as impacting the support for EPUB 3.2.
Most of the remaining issues are related to the following topics:
- vocab-related issues
- resources-related issues (CMT changes, foreign/remote rule updates)
- some schema-related issues
- some CSS-related issues
We will publish a beta release on the 2nd half of February (nearly feature complete, but not necessarily 100%), and a first release candidate (100% feture complete) in the first half of March.
We didn't received a lot of feedback (issues) on alpha-1. We will ask again on Twitter.
We now have the credentials to connect to the (DAISY-owned) AWS space where the validator service is hosted. We will try to update the service to v4.1.1 as soon as it is released.
We will also try to gather usage stats, in order to estimate if it’s relevant to keep maintaining this service. We don't have web analytics, but we can try to extract some info from the log files. We will forward any relevant info or data to the steering committee (via Tzviya and Luc).
Participants: Daniel, Matt, Romain, Tom
EPUBCheck v4.2.0-alpha-1 was released today! 🎉
We discussed what to do with the existing checks related to the newly deprecated features. For isntance for epub:switch: it used to require declaring the switch property in the OPF; should we keep on checking that? or just remove this check altogether now that the element is deprecated?
RESOLUTION keep the tests. Deprecated only means it shouldn't be used, but if someone does EPUBCheck should still be reporting that the feature has been used properly.
Another issue is about the bindings element, as it used to be treated as an acceptable fallback for unknown object types. Now that bindings is deprecated, should it sill be accepted as a valid fallback?
RESOLUTION keep on considering bindings as a valid fallback, and raise the issue to the CG for confirmation.
All the static schemas have been updated to use the latest ones from validator.nu. That includes HTML (+RDFa), SVG, MathML (although this latter is pretty much a copy/paste from the MathML spec).
We will push a patch against the original sources, to document the tweaks we made (mostly changes about element/attribute datattypes).
Should we try to import these schemas as a git submodule, and add the tweaking as a build step?
RESOLUTION: keep updating schemas manually; time will be better spent later on doing a full migration to validator.nu.
The issues included in alpha-1 are marked as completed, but not closed. Do we want to keep them opened during the whole 3.2 work duration?
or instead close them as we merge them to the next/4.2.0 branch to unpile the tracker?
RESOLUTION close the issues as we include them in alpha/beta releases.
We will now prepare the v4.1.1 release.
We will can go through the tracker and prioritize/assign remaining EPUB 3.2 issues next week.
Participants: Avneesh, Matt, Tom, Romain
Nothing much new, due to flu + holidays. Some good progress was made on porting the schemas from validator.nu, but it's still not ready. Romain still targets an alpha release before Jan 10 if possible.
No news from w3t-sys about reusing org.w3 or org.w3c as the Maven group ID for EPUBCheck. The alpha can be released only as a ZIP on GitHub, no need for a Maven artifact.
Romain also looked at PR #933, looks good. It can be integrated that in the alpha.
No significant progress either; we are basically giving more time to translators. Romain will ping Tobias to make sure we're in sync before releasing.
There are a few other PRs to include; Romain started reviewing them and nothing problematic.
We’ll prioritize and assign all the issues after the alpha release. Until then, we looked at a couple easy picks, reasonably isolated: #870, #902, some of the vocabulary-related issues.
A couple things:
-
Romain didn't have the time to review PR #828 from Daniel Persson (test cleanup). Still high prio after the coming releases.
-
the http://validator.idpf.org service is currently outdated to v4.0.2, and a couple people asked on Twitter when it will be updated to v4.1.0. Romain will see with Marisa to try and update it to v4.1.x ASAP.
Regarding the domain name, it sounds a bit odd to keep the service under idpf.org. Given that maintaining the service has a cost, and the service is ostly a demo tool anyway (10MB limitation), it could even be shut down. In any case, the issue should be raised to the SC.
Participants: Avneesh, Daniel, Matt, Tom
Matt is giving the package schema changes a review. Wondering where should these be committed? Should they be copied there and then create a PR of the same in epubcheck, or is there any way to automate a pull of the files?
If the CG want our schemas to be referenced externally in specs or elsewhere, the we could possibly maintain two separate versions; but as it seems ongoing maintenance isn't needed, let’s put them all in the /30 directories and override the new schemas.
A couple months ago, the CG agreed that they will make EPUBCheck shema public and not invest on recreating EPUB 3.x schema. But the appendix still points at the epub revision repository. Maybe we should re-open an issue that the appendix should point to the epubcheck repository location? In any case, it’s better to have only one schema source to maintain.
Alpha release is scheduled in december. We will not distribute it as a Maven artifact until we decide whether to change the group ID to a W3C name. The alpha will be released as a ZIP on GitHub’s release page.
As for the license, Wendy Seltzer will suggest appropriate approach for licensing (MIT/BSD, Copyright and CLA) within this week.
Note: no meetings in the next two weeks, due to end of year holidays
Participants: Avneesh, Matt, Romain, Tom, Daniel P.
Romain started to look at updating the schemas. Git spelunking on the current schemas is impossible, as they were initially developed in a separate 3.x branch on svn and the old history was lost with the migration to github. Trying to figure out if a drop-in replacement with schemas from the Nu HTML Checker could be envisioned.
The schema readme says:
The modules are derived from the schemas used in the validator.nu service [3].
The schema for for EPUB3 XHTML Content Documents is not continuously updated to reflect the latest version of the W3C HTML5 specification; it is a static representation of the EPUB3 XHTML Content Document definition, and changes to it occur only in conjunction with new releases of the EPUB3 specification.
We can grab the nu stuff and just have to adapt it similarly to allow the epub elements. We need to extend the definition to add elements. Basically just use the same driver file but update to match the nu modules
We'll try to make EPUB-specific additions as external as possible, to possibly help non-conflicting updates to the validator.nu upstream
As for SVG schemas, it seems Makoto was in favor of updating to validator.nu schemas as well, see #779.
Tom started taking a look at adding the switch we discussed last week for being able to turn CSS validation on/off. Trying to see what's the best way to pass things through.
This info could belong to the ValidationContext object, if we add a Map<String,Object> named features (or config?) to hold the info (and potentially other things).
We looked at how all this is initialized in the primary EpubChecker.java entry points and code flows. The class contains duplicate code and entry points. Tom will try to refactor and simplify it.
Daniel Persson finished batch 1 of the test conversion. The old cli_Test, api_Test and the cli directory have a new home in the tools directory and API directory as would be appropriate and all tests are commented and the old are deprecated. Now waiting for feedback.
Participants: Avneesh, Matt, Romain, Tom.
This was discussed in the EPUB CG, and the issue is still under vote. It's likely that EPUBCheck will be asked to treat all 3.2 as the default version and just ignore 3.0.1. Votes are open until Thursday December 6th at 1700 UTC.
This may lead to a discussion about errors vs. warnings vs. info. This would be treated as a new issue, it’s independant to the 3.0.1 vs. 3.2 issue.
We can start integrating new 3.2 checks in the code base with no differentiation; and we'll follow whatever recommendation the CG makes on errors vs warnings.
We got a couple PR after v4.1.0. One is about a build step; the other is about completing missing translations. Once the translations are complete, we can release them as v4.1.1. This won't impact our release schedule for the 3.2 implementation; it's a patch release on v4.1.0.
Some more digging in the CSS handling: only certain attributes like font-family, size and a few specifics are being checked, the attributes like the ones mentioned #901 are ignored.
The CSS profile in EPUB 3.2 has a few slightly non-backward-compatible changes; we should follow the lead of the CG in what to validate.
- currently, we do validate some CSS, but only partially
- we do not cover all EPUB CSS rules
- should we handle anything specifically outlined in the spec? (at least at a warning level?)
For now we should maintain the status quo regarding what EPUBCheck does and doesn’t validate. We'll need to update the existing checks that are changed in 3.2; but general CSS validation is out of scope.
CSS validations could be triggered with an on/off switch, which people could either enable or disable. We’ll look at the implementation feasibility. One issue is deciding on the default behavior: validate or not? We can ask the CG to vote.
EPUB 3.2 support is due in Q1 2019. We will start releasing test versions earlier (first alpha in December?).
The most important/significant issues to tackle include:
- HTML and SVG upgrades (#892 and #893). Until we integrate with the Nu HTML Checker, we need to upgrade the schemas (see also #779).
- remote fonts
- core media types
- resources in the container without fallbacks
What about deprecation warnings? This could be added early on. Probably not the biggest prio, more a mid-tier change.
We'll setup 3 milestones: beta-1 (Dec?) / beta-2 (Jan?) / beta-3 (Feb?) and start sorting issues in these 3 buckets. Then we can review/reorganize these at will. RC testing can be done in March.
The version supporting EPUB 3.2 will be called v4.2.0, since the API and internal code won't change much.
Participants: Daniel, Matt, Romain, Tom.
EPUBCheck v4.1.0 was released today! See also the W3C announcement blog post. The changelog should be pretty thorough, it's based on the commit log.
One of the biggest refactoring is the one introducing localized reports:
previously EPUBCheck's locale was the JVM's default, now it can be configured
so some objects in EPUBCheck are now Locale-aware, including most of the reports, the CSSParser, etc.
The Locale is also available from the ValidationContext class but this all should be mostly transparent, it's not a breaking change
Sources compatibility has been upgraded to Java 7 at minimum; and various libraries have been updated.
Need to look more into #901 to make sure there is nothing to be done in EPUBCheck.
Housekeeping question: if we don't find anything that can be done, how should we tag/track that issue?
- explain the reasons in an issue comment
- assign a reviewer (Matt, Romain, or both)
- set the status label to
status: waiting for review - when the reviewer agrees, close the issue as
type: wontfix - OR if the issue is to be postponed to a later release, then set the status label back to
status:acceptedand update whatever milestone it's scheduled for
The Maven group ID is still org.idpf, maybe we should update that to org.w3c, let’s raise the issue to the steering committee
The latest translations were not pulled from the translation service (Transifex). Only one new string was impacted, and it only had one new translation (in German); but the release process should be made less error prone:
- document a release checklist in the wiki
- try to better integrate with Transifex and automatically pull new translations (tracked in #913). Maybe with web hooks (setup an address with Transifex and then Post a message to it to fire off their process).
A CHANGELOG.md file has been created.
Ideally the change log would be generated automatically from the issue tracker or commit log (à la "conventional changelog").
We will explore how this can be done from a Maven build, that would reduce the release work.
We'd like to configure an auto formatter, to reduce the whitespace-only diffs. If possible that would be integrated in the Maven build process.
Participants: Avneesh, Daniel, Matt, Romain, Tom.
Nothing really new to report, PRs will be closed soon.
Typically we should try to have all the PRs reviewed by a peer before merging. Merging PRs without peer review is acceptable if the change it quite straightforward; we do not always need systematic peer-review. The automated CI build is usually enough for QA on simple changes. "trusted" maintainers may auto-review when they're confident enough, given the lack of reviewers availability.
Tom started to look into #901 (-epub-text-orientation).
Should these new rules be setup in a new set of schematron files, and if so how much do we duplicate vs. referencing the existing files?
Sounds like a breaking change in EPUB 3.2.
It's currently not checked by EPUBCheck.
The bigger question is whether we want to keep a "3.0.1 forced mode" or if we just make 3.2 as the default and not allow checking 3.0.1 anymore. This question will be raised to the CG and BG.
We should also find out if proper CSS validation is even wanted. It could lead to a lot of errors/warnings if it gets implemented. It would take time to develop.
For the first 3.2-compatible release, we may not need to validate aspects that EPUBCheck didn't validate before. For example for #901, that means we may simply have nothing to act on.
Participants: Daniel, Matt, Romain, Tom.
Romain started to review issues and PRs. A deeper look at #859 indicates it's due to the schematron XSLT built in Jing. We have to take action since it's not only a development-time warning but will be rendered to users. Several solutions can be further explored; a mininmal repro project will be pushed to GitHub to file an issue with Jing.
Matt finalized the list of issues on the intermediary Google doc; all the required changes are now filed in the tracker with the spec: EPUB 3.2 label.
Old issues labelled as 3.1 have been moved to the spec: EPUB 3.2 label; Matt will clean up the issues that are no longer relevant.
Tom and Daniel will review these issues, and can start taking on some of them.
Romain started renaming the Github issue labels, following the suggestions in #862. Missing labels will now be added.
The branching guide hasn't been written yet, but we don't need a common 3.2 branch. New features and bug fixes can be developed in their own branch, that will later be merge to the master branch. We are suggesting to use a lightweight naming convention for branches: $type/### where ($type) indicates the type of work (e.g. feature, fix, chore, docs) and ### is either a short nickname or an issue number reference. The idead is to keep it simple, while making the branch name quickly understandable.
Participants: Avneesh, Daniel, Matt, Romain, Tom.
We discussed how we wanted to organize the communication within the core development team:
- weekly text chats using DAISY's
#epubcheckSlack channel - ad-hoc audio calls when needed
- Slack is used mostly for transient communication
- Resolutions and important information should be summarized in the meeting minutes on the wiki
We also discussed Slack best practices:
- we'll use a unique
#epucheckchannel as long as it works - we may add additional feature-specific Slack channels if discussions become difficult to manage in a single channel
- we won't use Slack's threading functionality, which is deemed more confusing than useful
- significant message editing is discouraged, or should be explicitly mentioned in the main discussion thread
- we'll use explicit '+1' messages rather than thumbs-up emoji reactions to upvoting important decisions (for accessibility reasons)
We discussed the issue labelling and potential use of a column-based issue management tool (Waffle, Zenhub, GitHub projects). Zenhub might be too complex for our use; GitHub projects isn't accessible. For accessibility, we want all the information to be available from the standard issue tracker and labels. Waffle could be a good option, as it is a pure visualization of GitHub's existing dataset.
Resolution:
- apply the labels proposed in #862 (pending comments from Daniel)
- stick to issues/labels/milestones as the primary source of information
- experiment with Waffle as an optional layer for visual issue management
Maintenance release, no big new features. The task is basically to wrap-up the work done by previous maintainers (and contributors). No plans to kill features like EDUPUB (yet). We should check that no "invalid" EPUB 3.1 validation is introduced.
Romain will handle 4.1.0, getting help from Daniel or Tom only if needed. The deadline is 3d week of November.
This task should not delay the work on EPUB 3.2.
A quick developers guide should be written on how to manage branches for working on multiple versions/features in parallel.
Status: exploring the source code, existing pull requests, EPUB 3.2 specs. Still quite low on the learning curve. Discussion will continue on Slack; Romain will answer questions and/or setup audio calls when requested.
Tom looked at issue #859 and the warnings appear to be raised at instantiation (and not for each test run). To be investigated.
The first step is to identify the list of stuff that changed (compared to EPUB 3.0.1) and need to be implemented. Matt will handle this task, with assistance from Daniel and Tom. Initial work will be done based on an Google Doc draft created by Matt during the 3.1 work.
The basic workflow will be:
- identify a change requiring to be implemented in epubcheck
- create a new issue
- someone else reviews and sets the label
accepted - someone voluntarily takes the issue, or we assign someone during our weekly chat
- this person thinks about the issue, possibly discusses it, and then sets the
waiting for revieworready for implemlabels
Participants: Avneesh, Daniel, Romain.
- set up infrastructure.
- Organize the issues + review the labelling strategy
- Move the repository to W3C.
- Do we need to rename the Java namespaces (legacy adobe and IDPF names). This can be handled in phase 2 during the code clean up.
- Infrastructure work can start Before TPAC
- Maintenance release for existing fixes (v4.10)
- Deadline is November.
- EPUBCheck updated to EPUB 3.2 - alpha release.
- In December or early January.
To help us triage GitHub issues, create "epics" meta-issues (group related issues, identify dependencies / development critical path, manage separate Pull Requests, etc.), and ultimately assign tasks / action items within a timeline, using:
- GitHub milestones
- GitHub labels
- possibly GitHub projects
...but there are also more powerful alternatives:
Romain knows Huboard; Daniel knows ZenHub; We all know Waffle, Trello, GitHub Projects.
Proposal (to be discussed further): exclude Trello (too disconnected from GitHub, more suitable for higher-level project management), exclude Waffle (there are slightly better alternatives than this simple GitHub "overlay"), exclude GitHub Projects (limited feature-set), consider Huboard and Zenhub (we need to verify accessibility).
ACTION ITEM: Avneesh to check screen reader accessibility of the current Github Projects (Avneesh already uses GitHub issue tracker label-based filtering, etc.)
ACTION ITEM: Romain to explore ZenHub and setup a test instance
We discussed the human resources needed to achieve the EPUB 3.2 milestone. ACTION ITEM: Romain should get in touch with the team and kick-off the development.
Communication channels for development team:
- the development team can use the #epubcheck Slack channel (in the daisy-dev orgnization)
-
- ad-hoc audio calls when necessary
- setup a weekly timeframe for developers chat
- the discussions will be summarized and published on a wiki page, under the heading of the date of call/interaction.
Communication call with steering committee:
- schedule a monthly audio call on Zoom.
- Create EPUBCheck twitter account? (requires approval of Luc/Tzviya). Its objective is to keep the public updated about the technical developments.
- We should formulate plan for providing some light weight updates that can be used by Luc/Tzviya for marketing. It may be posted on a blog or news, social media page etc.
ACTION ITEM: Avneesh to get in touch with Luc and Tzviya to invoke discussions on these topics.