Text-to-speech and content extraction

[mickael-menu]

Changelog

Added

Shared

• Extract the raw content (text, images, etc.) of a publication. Take a look at the user guide.

Navigator

• A brand new text-to-speech implementation. Take a look at the user guide.

Review notes

A good first step is to read the two user guides linked in the changelog above.

Then I suggest going in this order:

• Implementation of the Content Iterator proposal (with some modifications).
  • Content.kt contains the Content models as well as the Content.Iterator interface.
  • PublicationContentIterator.kt is a general implementation to iterate over a full publication, it delegates the actual extraction to ResourceContentIterator instances.
  • HtmlResourceContentIterator is a specialized Content.Iterator which can extract the content from an HTML Resource.
• shared/util/tokenizer is a new package providing utilities to tokenize pieces of data.
  • TextTokenizer.kt is used to split a String into multiple IntRange components. You can split a paragraph into sentences with it.
  • ContentTokenizer.kt splits a single Content.Element into a list of Content.Element.
    • The default TextContentTokenizer implementation delegates the actual text tokenizing to a provided TextTokenizer to split Content.TextElement into multiple Content.TextElement. Then, it will split the Locator.Text model to match the generated tokens.
• Language.kt is basically Locale but (ultimately) independent from the JVM.
• EpubNavigator was updated to not refresh the current resource when jumping to a locator in it. The TTS jumps to locator every seconds so it was causing a lot of visual refreshes.
• navigator/tts package.
  • Note that the TTS is completely independent from the Navigator, we could put it in shared or even a new readium-tts module. However I put it in navigator as it's usually used in combination with a Navigator. We could raise this discussion in a call.
  • TtsEngine is an interface to implement to support a third-party TTS engine. It is pretty simple with a speak() API to speak a single utterance. It's suspending so cancelling the job cancels the speech. Calling speak() multiple time should queue the utterances (the caller is suspended until its utterance is played).
  • AndroidTtsEngine is an implementation using the native Android TTS
  • PublicationSpeechSynthesizer is the high-level orchestrator that an app can use to read a whole publication aloud.
• In the Test App:
  • Removed the legacy TTS implementation which was entirely in the test app
  • Refactored the BaseReaderFragment options menu to have a single one. Before EPUB had a specific options menu layout which contained the TTS button.
  • Added a TtsViewModel which will be composed inside ReaderViewModel to make it easier to look at TTS stuff.
  • Added a Jetpack Compose overlay view on top of the reader in VisualReaderFragment. This is used to display the TTS controls above the publication, but also to prevent user interaction on the pages while the TTS is running.

Unfinished stuff

The following can be done in follow-up PRs as it's not critical right now:

• Supporting custom filters in the HtmlResourceContentIterator to skip specific HTML nodes.
• Better support for various HTML tags and ARIA attributes.
• Adding a ContentService implementation for PDF documents.

---

• Closes TTS support #68

[mickael-menu]

This was added to start the TTS from the currently visible page instead of the start of the publication. Using the progression in the currentLocator was not precise enough, and firstVisibleElementLocator() makes sure to target the first HTML element fully visible or starting on the page.

[mickael-menu]

Inspired by @DelicateCoroutinesApi. We can use it to tag a dangerous Readium API that should be used after carefully reading the doc.

[mickael-menu]

Not necessary for this PR, but this seems to be a useful helper. I used it in the example of the content.md user guide to build an index of images.

[mickael-menu]

This algorithm is largely stole inspired by jsoup's text(), to extract a normalized text of an element.

[mickael-menu]

I introduced this Language to have a higher language construct that doesn't depend on the JVM, and to add some useful helpers such as removeRegion().

[mickael-menu]

I removed the legacy TTS implementation that was entirely in the test app.

[mickael-menu]

This could be moved to BaseReaderFragment to simplify the subclasses.

[mickael-menu]

All the options menu items are now in a single layout inflated by BaseReaderFragment.

[mickael-menu]

This change seems pointless but I refactored it twice. At some point TtsViewModel was a proper Android ViewModel and so I had a composite view model factory.

[mickael-menu]

I added a Jetpack Compose overlay view on top of the reader in VisualReaderFragment. This is used to display the TTS controls above the publication, but also to prevent user interaction on the pages while the TTS is running.

[mickael-menu]

I split the setup to make it clearer to people wanting to integrate only one of these features.

[mickael-menu]

This is not really necessary, but I find the recommended approach for de-emphasizing elements really heavy and unclear.

[mickael-menu]

Using a Channel ensures that the utterances are played in the same order as the speak() calls.

[mickael-menu]

That's the solution I came up with to be able to skip utterances or interrupt a running utterance. As only one playback job is allowed at any time, when you start a new command the previous one is cancelled.

[mickael-menu]

I started with the usual:

val state by model.state.flowWithLocalLifecycle().collectAsState()

But as I used a single State object that is updated every time a word is spoken, that triggered a lot of recomposition. I came up with this asStateWhenStarted extension that select a property of the global state instead, and is basically (it's a bit more complicated, see Flow.kt) equivalent to:

val showControls by remember {
    model.state
        .map { it.showControls }
        .flowWithLocalLifecycle()
}.collectAsState()