TTS support

[stevenzeck]

Although the testapp currently has TTS, it is all bundled in the testapp module itself. This means that developers are required to write their own TTS implementation with little to no support from the Readium libraries themselves.

[paktalin]

Hey! Just a comment from my side.

I have been trying to implement TTS in our application and find it very challenging to start the TTS from the correct page (the one currently displayed to the user). so far I have been retrieving all the text from the current resource and using progression to approximate the currently visible text, but it's not a great solution. In most cases it would result in the text visible in the middle of the page, but sometimes it's text on the next page not even visible to the user... :(

I don't think the test app supports starting TTS from the currently visible page either.

My code:

private suspend fun Publication.getTextAfterLocator(locator: Locator): String? {
    val link = linkWithHref(locator.href) ?: return null
    val htmlText = get(link).getText() ?: return null
    val progression = locator.locations.progression ?: return null
    return htmlText.substringStartingFrom(progression)
}

private suspend fun Resource.getText(): String? {
    return DefaultResourceContentExtractorFactory().createExtractor(this)
        ?.extractText(this)
        ?.getOrNull()
}

private fun String.substringStartingFrom(progression: Double): String? {
    val index = (length * progression).roundToInt()
    val lastDotIndex =
        if (index == 0) 0 else lastIndexOf(". ", index).takeIf { it != -1 }?.plus(2)
    return lastDotIndex?.let(::substring)
}

[mickael-menu]

I have been trying to implement TTS in our application and find it very challenging to start the TTS from the correct page (the one currently displayed to the user)

That's definitely one of the more challenging parts. A strategy could be to iterate over the DOM until you find a paragraph that starts on the current page (maybe with getBoundingClientRect()). This wouldn't start the exact first word but I'm not sure it's desirable to start mid-sentence anyway.

By the way I will be working on TTS in Readium early next year. Feel free to open a PR (even a draft) if you want to contribute some code for this.

Another issue is to turn pages at the right time when reading a sentence. A possible solution could be to have two decorators: one for the current paragraph / sentence being read, and one for the current word. Following the current word, we could more easily turn the pages at the right time.

It looks similar to what Thorium is doing:

tts.mov

[paktalin]

Thanks a lot for the reply!

A strategy could be to iterate over the DOM until you find a paragraph that starts on the current page (maybe with getBoundingClientRect()).

I have tried the getBoundingClientRect() approach, but it seems to be even less precise compared to locating current resource progression. Specifically in cases when paragraphs are long enough to take several pages, then we would have to scroll several pages back or forward to start from a new paragraph... I don't think there is a web API to know what part of a specific DOM element is currently in the viewport.

Another issue is to turn pages at the right time when reading a sentence. A possible solution could be to have two decorators: one for the current paragraph / sentence being read, and one for the current word. Following the current word, we could more easily turn the pages at the right time.

Thanks! That sounds like a good idea.

By the way I will be working on TTS in Readium early next year. Feel free to open a PR (even a draft) if you want to contribute some code for this.

I'm hoping to get a bit more time by the end of January, so will definitely consider helping with that :)

[mickael-menu]

I have tried the getBoundingClientRect() approach, but it seems to be even less precise compared to locating current resource progression. Specifically in cases when paragraphs are long enough to take several pages, then we would have to scroll several pages back or forward to start from a new paragraph...

Yes it looks like Thorium is doing it this way. When a paragraph spans several pages, the TTS will automatically move the reader to the beginning of the current paragraph instead of starting in the middle of the paragraph. I think that's a fair compromise, but it would be useful to allow apps to customize this strategy.

[mickael-menu]

Implemented in #118

[mickael-menu]

@paktalin I solved the problem of starting from the visible page with this piece of JavaScript:

kotlin-toolkit/readium/navigator/src/main/assets/_scripts/src/dom.js

Line 10 in 6d68be2

export function findFirstVisibleLocator() {

It's exposed in the toolkit as VisualNavigator.firstVisibleElementLocator().

[paktalin]

@mickael-menu thanks for letting me know! We will try it out instead of our custom script :)