Feature request: Add indexing support

[mikolysz]

Some speech backends can indicate to the caller when a particular point in the requested utterance has been reached (or where that point exists in the buffer, when speaking to memory). This is particularly useful when reading larger text documents and when cursor tracking is desired.

This is a capability that Prism should expose to callers when it is supported by the backend.

The input side of this is basically implemented already (with SSML's mark element). All we now need is a way for backends to signal Prism that a particular SSML mark has been reached, and for that signal to be passed back to calling applications.

[ethindp]

@mikolysz Hi there,

Thanks for the request. I have been considering adding SSML support in general to Prism, although I have been weighed down by uncertainties about exactly how to go about it. Various backends support various SSML versions and grammars; for example, SAPI obeys SSML 1.0 but with Microsoft-specific extensions and regressions, MacOS supports a version of it that I'm not entirely sure about, and I have no idea what version Speech Dispatcher and Spiel supports (assuming either of them handle SSML at all and it isn't "let's just defer it to the underlying engine" which would complicate things even further). And in some cases, the SSML speech path and the ordinary one are identical.

I would like however to get to this at some point, as well as add a few other niceties as well; I admittedly have been distracted with other projects.

[ethindp]

@mikolysz So, I've converted this into a discussion, per the contributing guidelines (which, don't worry, I set up a couple days ago).

But I have actually looked into this, and this is way more difficult than I ever thought it would be. The biggest issue which arises is that SSML is pretty much a failed standard. The W3C working group (specifically, the Voice Browser Working Group) designed the standard but of course by then we already had SAPI as just one example. As a consequence, each and every backend that Prism supports which supports SSML does this in a way totally divorced from the standard to which it (supposedly) conforms with. I suspect the only substantial exception is OneCore, which claims SSML 1.1 support.

As an example, SAPI does not have speech marks, but instead calls them bookmarks; the element is <bookmark mark=...>, not <mark> as the standard requires. AVSpeech doesn't even specify the SSML standard it uses from what I could find (unless I missed it which is definitely possible given Apples labyrinthine docs). NVDA (for whatever reason) uses a process-wide SSML mark reached callback, which means we have to do some very careful and tricky threading of the needle to fit that into Prisms design, and it only supports <say-as>, <voice>, <break>, <prosody>, <speak>, and <mark>. (The process-wide thing isn't really a big obstacle, and can be worked around.)

To make things even worse for us, different engines have different event notifications for different things! Some do word boundaries. Some do word and sentence boundaries. Still others do audio levels, or paused/resumed, or marks, and the list goes on. Some do a combination of the above. This gets really thorny because we have to figure out how exactly to even tell the user (from the C API) what exactly is going on.

Mind you, this is before we factor in those engines that don't explicitly have some way of explicitly requesting that SSML be handled separately from ordinary text. If I'm not mistaken, Android TTS does this (I think it does support SSML, but I'm not entirely sure, and it has no API to distinguish text from an SSML document or fragment if it does support it).

Now, all of this is not to say that SSML support is permanently a no-go. However, I would appreciate your feedback (and community input as well) so that we can figure out the best way forward and the best way of abstracting this to generally apply to any backend instead of being specific to a small subset.

My (very tentative) idea is that Prism could define a canonical SSML dialect, parse documents into an AST, and each backend would then transpile that AST into whatever it actually understands. (Whether that document is strict SSML 1.0/1.1 or some well-defined subset of it is an open question in itself.) If we go this route we get a fair amount for free: we're parsing the document anyway, so we can validate well-formedness without any extra work; we can also serialize the AST back out as plain text for engines that can't distinguish SSML from ordinary text, which mostly takes care of the Android problem I mentioned above; and because we control serialization, we can rewrite caller-supplied mark names into prism-generated tokens and remap them on the way back out, which solves several problems in one go: backends with mark name restrictions stop being an issue, and if I'm not mistaken it takes care of the NVDA process-wide callback as well, since we can encode instance/utterance identity into the token itself, at which point the very careful and tricky threading of the needle I mentioned above pretty much becomes a table lookup.

I originally thought of just passing the document straight through, but we would then have this significant surface who's behavior is completely unknown compared to the rest of the library (i.e., it would be backend-dependent). And I'm very hesitant to make the caller have to deal with that. It would be the simplest option, of course, but I'm not entirely convinced that simple is better here.

However, as I stated above I would appreciate feedback and community input, since this is, at least I feel, a difficult problem to actually solve. However, I also have a tendency to over-design a solution, especially in domains such as this, so there could be a simpler solution I'm overlooking. I would appreciate your thoughts, because this is definitively a feature I want to add.

[the-byte-bender]

I cast my vote to having a standard AST representation a backend could parse as it likes. Then we could conform to the actual spec and translate to whatever custom tags the backend understands.
That solves only one problem, though. There's still the fact that some backends don't have a way to advertise their SSML support for certain because it depends on a voice or some other configuration. Perhaps we just be honest and expose SSML support as an enum? False, true, and "maybe?" Some web API's do that kind of thing. Document that if the return value is "Maybe", sending ssml might work, and if it doesn't, there is no guarantee what will be spoken. Some engines might strip the tags, some might literally speak them. The user can then make an informed decision. I don't like this way much either, but I think it's inevitable for something that needs to abstract over so many backends. TTS implementations on most platforms tend to be a mess that usually nobody cares to maintain or keep up to standards.

[mikolysz]

I don’t think ssml support should be a single capability; instead it should be broken up into sub-capabilities / behavior markers for every tag. SSML is the typical W3 spec; it says more about how the authors wish SSML worked than how backends actually interpret it. Many things that the spec considers mandatory are often left unimplemented, and backends love introducing mutually-incompatible extensions. I agree that there must be some kind of universal AST, perhaps with built-in transformation passes that backends can opt into to get it closer to the representation they actually support. This would include things like stripping unsupported tags, converting unsupported attribute values to ones that the particular backend can handle, or resolving SSML voice changes into concrete voices. Depending on how widely usable Prism should be, it might also be a good idea to introduce a “linearizer” that turns such an AST into a linear command stream, either stateful or stateless. Such a model is much closer to how some backends and synthesizers (think IBMECI, SAPI4, MacinTalk and most hardware synths) work.

[ethindp]

I agree about the linearizer (especially when we factor in the fact that that would, at least theoretically, give us SSML support on backends which don't actually support it). It would in effect turn SSML into a form of a mini-program.

If the universal AST/linearizer is what we collectively decide to actually go for (and I'm hoping more people chime in) then (I think) the question might become how to handle backends that natively claim SSML 1.1 support. For example, OneCore's SynthesizeSsmlToStreamAsync method claims SSML 1.1 support, but it is ambiguous whether that's the W3C's idea of SSML or if that's Microsoft's. So, assuming we choose to do that, do we have two different capabilities (possibly purely internal ones that are never exposed to the public API) that say "Hey, this requires an SSML command program" and "Hey, this natively supports SSML so pass it in" or do we just implement the spec ourselves and simulate it? Or both? (Both seems to me at first blush to be overly convoluted.)

If we go with simulation, this would theoretically make custom backend support trivial: a custom backend defines an execute_ssml_command function (or similar), which is how Prism would tell it to execute commands Prism can't do itself (emphasis, for instance), performs the command, and returns; where Prism can do it natively the custom backend just gets a set_voice call as normal and never even has to know that SSML is being handled.

[ethindp]

GH isn't letting me edit comments for some reason, but I just thought of a problem with my simulation idea: the in-progress utterance issue. A lot of SSML relies on utterance properties changing dynamically as the utterance progresses (for example, in a TTS-based skit, you might want a voices pitch to increase as they get more and more nervous). That would break down if we do it the "obvious" way of just set_pitch or what have you since that would apply to subsequent utterances and not the in-progress one. Which greatly complicates the entire idea since we potentially would need DSP/audio processing stuff. I think? 🤔

[mikolysz]

I agree about the linearizer (especially when we factor in the fact that that would, at least theoretically, give us SSML support on backends which don't actually support it). It would in effect turn SSML into a form of a mini-program.

I think there are ~5 kinds of backends we should be concerned with: 1. Natively AST-based. (I suspect AVSpeechSynthesizer is like this, although I’ve never reverse-engineered that framework to confirm). SAPI5 is definitely like this, although its native representation is its own variant of XML. 2. Linear stateful (E.G. hardware synths). Every value-changing command applies until the value is changed back. There may or may not be a logical concept of an utterance, but even if there’s one, changes still persist across utterances. 3. Linear stateless. I’m not sure if these exist; I know for a fact that SAPI4 and IBMTTS are command-based, as was Apple’s MacOS 9 era speech stack, parts of which still remain in MacOS. I’m not sure if any of them natively reset state across utterances though. 4. Utterance stateful (everything with setVoice / setRate / setPitch functions that can only speak complete strings, with no way to change attributes mid-utterance). 5. Utterance stateless, where you have to ship the attributes with the utterance and there’s no way to change them midway through, think AVSpeechSynthesizer and all the cloud APIs). Some backends are a mix of the two; AFAIK Google’s takes SSML (including plenty of proprietary extensions), but only allows voice changes through an external attribute for example. This makes me think that Prism potentially needs a way to break up a “logical utterance” (what the developer wants spoken) into multiple physical utterances (what the backend can handle), in cases where the developer requests attribute changes that the backend isn’t able to handle mid utterance. Whether this should be done in the linear or AST representations (and which one Prism should consider native) is IMO a matter for discussion. This would also enable cool things like cross-backend utterances, where different parts are synthesized with different voices.

If the universal AST/linearizer is what we collectively decide to actually go for (and I'm hoping more people chime in) then (I think) the question might become how to handle backends that natively claim SSML 1.1 support. For example, OneCore's SynthesizeSsmlToStreamAsync <https://learn.microsoft.com/en-us/uwp/api/windows.media.speechsynthesis.speechsynthesizer.synthesizessmltostreamasync?view=winrt-28000#windows-media-speechsynthesis-speechsynthesizer-synthesizessmltostreamasync(system-string)> method claims SSML 1.1 support, but it is ambiguous whether that's the W3C's idea of SSML or if that's Microsoft's. So, assuming we choose to do that, do we have two different capabilities (possibly purely internal ones that are never exposed to the public API) that say "Hey, this requires an SSML command program" and "Hey, this natively supports SSML so pass it in" or do we just implement the spec ourselves and simulate it? Or both? (Both seems to me at first blush to be overly convoluted.)

I wouldn’t be overly trusting here; I highly doubt anybody implements this spec in its entirety, including the complex language switching and pronunciation dictionary algorithms, although I’d love to be proven wrong. What I’d do is to add a raw SSML node to the AST, which is not to be parsed by Prism, and then write a simple compliance testing program that utilizes this node to check which SSML features a given backend actually supports, possibly by human ear. Even for backends with good spec coverage, some things that we may want to include in the AST but are not part of the spec may be implemented as different extensions by different backends. I think the wisest approach is to parse caller-supplied SSML into an AST / command program, do transforms on it based on declared capabilities, do physical utterance separation if needed, and then give the result to the backend to serialize. For built-in backends, I’d pass it directly as an `ASTNode` or `std::vector<Command>`, depending on whether it is an `ASTBackend` or a `LinearBackend`. It would be up to the backend to serialize it into whatever form it needs, either via some built-in `AstNode::serialize_to_ssml` function, or through custom serialization logic in case of SAPI5 for example. For third-party backends, we’d do the serializing, letting them deserialize it back and re-serialize into a custom format if needed. What the custom backend representation of a linear command stream should be is a matter for discussion, I’m not entirely convinced that SSML is the right idea here. Maybe a JSON array with `type` and `payload`?

If we go with simulation, this would theoretically make custom backend support trivial: a custom backend defines an execute_ssml_command function (or similar), which is how Prism would tell it to execute commands Prism can't do itself (emphasis, for instance), performs the command, and returns; where Prism can do it natively the custom backend just gets a set_voice call as normal and never even has to know that SSML is being handled.

I don’t like this. Even if Prism can do something natively; say change the rate; I think it should let the backend do it whenever supported. This is because native mid-utterance changes can potentially be less disruptive to the flow of speech than forcing the backend into physical utterance separation when that is not warranted.

[ethindp]

This makes me think that Prism potentially needs a way to break up a “logical utterance” (what the developer wants spoken) into multiple physical utterances (what the backend can handle), in cases where the developer requests attribute changes that the backend isn’t able to handle mid utterance. Whether this should be done in the linear or AST representations (and which one Prism should consider native) is IMO a matter for discussion. This would also enable cool things like cross-backend utterances, where different parts are synthesized with different voices.

I agree. I am not entirely sure how to do this (I'm not a speech scientist) but this would definitely be an incredibly powerful mechanism if we could figure it out.

I wouldn’t be overly trusting here; I highly doubt anybody implements this spec in its entirety, including the complex language switching and pronunciation dictionary algorithms, although I’d love to be proven wrong.

After seeing how utterly fragmented the SSML universe is, I'd have to agree. i wish MS actually specified what they do implement, but alas.

What I’d do is to add a raw SSML node to the AST, which is not to be parsed by Prism, and then write a simple compliance testing program that utilizes this node to check which SSML features a given backend actually supports, possibly by human ear.

I'm not entirely sure how a testing setup would work; how complete would the tests have to be? Like would we go through each element and test compliance against some baseline artifact? (I wonder if AI could help here: not generative, but something like VAD and maybe some other model to detect certain voice changes? I'm spitballing with this one and am not sure how this would actually be implemented, but I did use VAD to prove the silence trimmer works, but that's as far as I've gone with something like that.)

For built-in backends, I’d pass it directly as an ASTNode or std::vector<Command>, depending on whether it is an ASTBackend or a LinearBackend.

This is definitely going to require some thought and consideration: the current backend model is not designed for this degree of separation/distinction, other than by backend features and extension of the virtual base class. Depending on how we wish to delineate this distinction this may need to be delayed considerably.

When I originally wrote the interface I was not aware that SSML was so fragmented as to cause this level of difficulty, although I did want to add it at some point down the road. I am fine with restructuring implementation details, since they are private to Prism and an external caller needn't know or care about them, but unless we kick this down the road to version 2 I'd be hesitant to change something like this to such a fundamental degree that it would cause the C API to be changed in some manner. Yes, semantic versioning does permit libraries at version 0.x to change the public API randomly and without warning, but until now I have done what I can to ensure that the public API is forward-compatible and that upgrading is a transparent process.

Considering how difficult this project may be, it may be something that will, sadly, have to wait until version 2.x. Which is not at all something I enjoy saying. I believe that at the moment Prism is approaching a point at which version 1.x will be ready, although I was genuinely hoping to have SSML in before then. However, as they say, necessity compels, and this would not be the only thing that would need to wait, and I'm not entirely sure if end-users would be overly concerned that SSML was not in the initial release. Waiting until v2.x would also allow us to do this properly instead of fragmenting it across release boundaries, since once 1.x is out the public API will be frozen for that major version.

What the custom backend representation of a linear command stream should be is a matter for discussion, I’m not entirely convinced that SSML is the right idea here.

I agree that SSML is most likely not an appropriate representation (given that SSML is the input). That would effectively cause us to round-trip the data through a lossy process, and we would then be attempting to reconstruct an output from information we don't have at that point.

Maybe a JSON array with type and payload?

Are you specifically referring to how a backend would process the transformed string? As in: would we go input string -> output string?

I don’t like this. Even if Prism can do something natively; say change the rate; I think it should let the backend do it whenever supported. This is because native mid-utterance changes can potentially be less disruptive to the flow of speech than forcing the backend into physical utterance separation when that is not warranted.

It was an interesting idea but as I noted it does have significant issues, in particular utterance changes mid-utterance. Which is why I'm not sure it would survive implementation contact.

[mikolysz]

This makes me think that Prism potentially needs a way to break up a “logical utterance” (what the developer wants spoken) into multiple physical utterances (what the backend can handle), in cases where the developer requests attribute changes that the backend isn’t able to handle mid utterance. Whether this should be done in the linear or AST representations (and which one Prism should consider native) is IMO a matter for discussion. This would also enable cool things like cross-backend utterances, where different parts are synthesized with different voices. I agree. I am not entirely sure how to do this (I'm not a speech scientist) but this would definitely be an incredibly powerful mechanism if we could figure it out.

What do you find difficult about this speech wise? I think that correctly preserving state here is an engineering challenge for sure, but there isn’t too much speech processing work here. If we encounter an AST node (say `ChangeVoice`) that the backend doesn’t have mid-utterance support for according to its capability matrix, we break up the utterance into 3 physical utterances: the before, the current and the after. We surround all of them by the complete stack of ancestor nodes of the unsupported node, so that other attributes from up the tree are preserved. We then queue these 3 (or more) utterances. For cross-backend support later on, we only queue utterance n+1 once the backend of utterance n indicates that it is done processing. Things like speaking to memory and/or a specified output device would require all backends to support that manner of speaking; speaking to memory could be done in parallel and then re-assembled into a single buffer. It’s a lot of messy code and logic for sure, I wouldn’t envy anybody trying to write this, in C++ no less, but I don’t see any fundamental obstacles to getting it done. It’s actually easier if we consider the linearized representation to be native. Then, a command like `setRate` which a backend supports as a separate function call but not a mid-utterance command causes a single break, with the change delivered out of band, once the previous utterance finishes (or in parallel when the backend is capable of that).

What I’d do is to add a raw SSML node to the AST, which is not to be parsed by Prism, and then write a simple compliance testing program that utilizes this node to check which SSML features a given backend actually supports, possibly by human ear. I'm not entirely sure how a testing setup would work; how complete would the tests have to be? Like would we go through each element and test compliance against some baseline artifact? (I wonder if AI could help here: not generative, but something like VAD and maybe some other model to detect certain voice changes? I'm spitballing with this one and am not sure how this would actually be implemented, but I did use VAD to prove the silence trimmer works, but that's as far as I've gone with something like that.)

I'd do a/b tests. Compare `this is a test` to `this is a <high-pitched>test</high-pitched>` (attribute for illustrative purposes only). Is there a noticeable difference? If so, the attribute is supported. If there are multiple ways of specifying pitch (literals; HZ; semitones), have one test case for each of these 3. If there’s some weird edge case we discover on one backend (an attribute being supported in one context but not another, even when the spec mandates both), add that as a test case and (eventually) retest others for compliance. You could do AI for some of it; higher rates should take up fewer samples in a buffer or take up less time before the indexing callback fires; pitch detection algorithms exist and are not even considered AI, the rest would be 10 mins max of by-ear work for each new SSML capable backend. For AST-but-not-SSML and linear backends, this is moot. We never test compliance this way (no support for the raw SSML escape hatch). Just to be clear, I’m not advocating to add test-suite tests here, merely a development-only test program that you only run once to generate a backend’s capability list.

When I originally wrote the interface I was not aware that SSML was so fragmented as to cause this level of difficulty, although I did want to add it at some point down the road. I am fine with restructuring implementation details, since they are private to Prism and an external caller needn't know or care about them, but unless we kick this down the road to version 2 I'd be hesitant to change something like this to such a fundamental degree that it would cause the C API to be changed in some manner. Yes, semantic versioning does permit libraries at version 0.x to change the public API randomly and without warning, but until now I have done what I can to ensure that the public API is forward-compatible and that upgrading is a transparent process.

Can you explain why you believe this would require C API changes?

Waiting until v2.x would also allow us to do this properly instead of fragmenting it across release boundaries, since once 1.x is out the public API will be frozen for that major version.

I think a 1.x imperative-style API should always exist. Most basic “just make it speak” callers don’t need or want SSML, even those who want some more-advanced features (like indexing callbacks) might still prefer a more imperative calling style. The advanced SSML API could be a `speak_ssml` function, gated behind an `ALLOW_UNSTABLE_APIS` feature flag until we’re 100% sure what we want to do with it. Similarly on the custom backend side, backends that naturally lend themselves to an imperative style should have the ability to stay imperative, a function to take SSML or a command stream can be added later. If we want to do so, we can eventually deprecate the imperative APIs in 1.x and make 2.X fully based on SSML, although I don’t think we should.

Maybe a JSON array with type and payload? Are you specifically referring to how a backend would process the transformed string? As in: would we go input string -> output string?

For custom backends that think in terms of linear “from now, the rate is 50” commands, and not in terms of a tree of SSML nodes, those commands need to be communicated somehow. We can communicate a tree and let the backends linearize, but that’s a lot of repetitive work for very little gain. If we already have a linearizer for built-in backends, why not standardize on an output representation of that linearizer and give that to custom backends? I guess an array of C structs (C unions? Pointers to C unions) would actually be a more sensible approach, but I’m honestly not an expert in C ABI design by any means, so I have no idea what’s easiest to bind to.

[ethindp]

What do you find difficult about this speech wise? I think that correctly preserving state here is an engineering challenge for sure, but there isn’t too much speech processing work here. If we encounter an AST node (say ChangeVoice) that the backend doesn’t have mid-utterance support for according to its capability matrix, we break up the utterance into 3 physical utterances: the before, the current and the after. We surround all of them by the complete stack of ancestor nodes of the unsupported node, so that other attributes from up the tree are preserved.

We then queue these 3 (or more) utterances. For cross-backend support later on, we only queue utterance n+1 once the backend of utterance n indicates that it is done processing. Things like speaking to memory and/or a specified output device would require all backends to support that manner of speaking; speaking to memory could be done in parallel and then re-assembled into a single buffer. It’s a lot of messy code and logic for sure, I wouldn’t envy anybody trying to write this, in C++ no less, but I don’t see any fundamental obstacles to getting it done.

It’s actually easier if we consider the linearized representation to be native. Then, a command like setRate which a backend supports as a separate function call but not a mid-utterance command causes a single break, with the change delivered out of band, once the previous utterance finishes (or in parallel when the backend is capable of that).

Thanks, that helps. I was just unsure of how we would split textual utterances into logical ones, that's what was confusing me.

I'd do a/b tests. Compare this is a test to this is a <high-pitched>test</high-pitched> (attribute for illustrative purposes only). Is there a noticeable difference? If so, the attribute is supported. If there are multiple ways of specifying pitch (literals; HZ; semitones), have one test case for each of these 3. If there’s some weird edge case we discover on one backend (an attribute being supported in one context but not another, even when the spec mandates both), add that as a test case and (eventually) retest others for compliance.

You could do AI for some of it; higher rates should take up fewer samples in a buffer or take up less time before the indexing callback fires; pitch detection algorithms exist and are not even considered AI, the rest would be 10 mins max of by-ear work for each new SSML capable backend.

For AST-but-not-SSML and linear backends, this is moot. We never test compliance this way (no support for the raw SSML escape hatch).

Just to be clear, I’m not advocating to add test-suite tests here, merely a development-only test program that you only run once to generate a backend’s capability list.

Makes sense. Although I wonder if this would break down with things like Android? With the Android TTS backend the TextToSpeech class is a generic class across some arbitrary set of underlying engines. I assume that at least in theory this wouldn't change in that case, but I have no idea how TTS engines on Android work under the hood.

Can you explain why you believe this would require C API changes?

It's more alignment of the C API with it's underlying C++ counterparts. Depending on how the implementation is done this may not be needed. I suspect I might be hyper-focusing on minutia with my concerns here (e.g., a PrismBackend corresponds to a TextToSpeechBackend` internally for consistancy).

I think a 1.x imperative-style API should always exist. Most basic “just make it speak” callers don’t need or want SSML, even those who want some more-advanced features (like indexing callbacks) might still prefer a more imperative calling style.

The advanced SSML API could be a speak_ssml function, gated behind an ALLOW_UNSTABLE_APIS feature flag until we’re 100% sure what we want to do with it.

Similarly on the custom backend side, backends that naturally lend themselves to an imperative style should have the ability to stay imperative, a function to take SSML or a command stream can be added later.

If we want to do so, we can eventually deprecate the imperative APIs in 1.x and make 2.X fully based on SSML, although I don’t think we should.

I think we might've crossed wires. I'm not advocating for the deprecation of speak nor speak_to_memory. That's always going to stick around. But things like indexing typically requires SSML for marks.

My original idea was to take what we have for speak/speak-to-memory and keep the same signature with an SSML event callback. But then I started digging into it and I ended up developing this big struct with about 15 different fields to cover all the event types (mainly since I wasn't sure how Python cffi handles C unions). I discarded that since it seemed wasteful. The signature is something I'd like to keep though if only because it's similar to what already exists and it makes migration painless (just append a suffix, add a fourth argument, done).

Custom backends are a similar story: the vtable can always be extended without that extension counting as a new ABI generation (function pointers being appended at the end doesn't alter Prisms existing view of the struct as you move between versions). So we're free to extend that as we wish. A new generation only counts if you reorder or remove functions in the vtable (or add things in the middle).

For custom backends that think in terms of linear “from now, the rate is 50” commands, and not in terms of a tree of SSML nodes, those commands need to be communicated somehow. We can communicate a tree and let the backends linearize, but that’s a lot of repetitive work for very little gain. If we already have a linearizer for built-in backends, why not standardize on an output representation of that linearizer and give that to custom backends?

I agree with this -- less code. I just wasn't sure where JSON fit in so that's why I asked the question.

I guess an array of C structs (C unions? Pointers to C unions) would actually be a more sensible approach, but I’m honestly not an expert in C ABI design by any means, so I have no idea what’s easiest to bind to.

Binding is hard; it's very language-dependent. You can expose a C union (or even a struct with union(s) inside it) but it really comes down to how the language your binding to represents them. Assuming it can at all, anyway.

I feel like once we actually start building the implementation (and I would appreciate collaboration on it) a lot of my discomfort and confusion will disappear. That's what usually happens once I've got a concrete code-based model of how something works.

The ironic thing is that the AST is pretty much entirely mechanical. Claude could do it trivially (although reviewing it will be time-consuming just to maek sure it doesn't screw something up, and then testing it and iterating). The much more complex parts will be figuring out the commands, transformations, linearization and such. I don't have a lot of experience with linearizers and things like that so should we take this route it will definitely count as new ground for me.

[mikolysz]

A couple of things: 1. I think we should standardize on the tree representation as internal. I no longer consider my idea to perhaps use a linear one and derive a tree when needed to be worth it. This is because going tree->linear is easy and always possible, but going linear->tree is tricky. Consider a command stream like `rate(50), pitch(30), say(test), pitch(whatever_the_original_was), rate(20), say(test2). This would likely lead to a very deep tree, and figuring out when a command is intended to “get us out” of a current node is tricky. On the other hand, with a tree repr, this would be something like `(tree (rate 50 (pitch 30 (say test))) (rate 20 (say test2)))`. In the linearizer, we’d probably want to have a state structure that keeps track of what our desired state is and what we’ve emitted so far, so that we don’t emit state restore commands that are then immediately cancelled by other commands from the next sibling setting the same value, but that’s much easier to correctly linearize. 2. It’s very tempting to also do a DB of per-voice attributes. This would bloat the binary, but would be genuinely useful for things like accurate WPM calculations in book reader apps and such. Nobody gets these right, and people’s crude approximations are far cruder than they should be. This needs fallback for unknown voices ofc, as well as protection from speech synthesis nerds trying to bloat the db with voices that nobody uses, but that would solve the SSML on Android issue (if it exists at all). 3. I think the linearizer is easy (I can imagine the code in my head), you just recursively walk over the tree and emit an enter command + children + exit command for each node, with the deduplication in the emitter as explained above. What’s not easy is the utterance splitter and the logic to keep track of it all, particularly given different speaking strategies (to buffer, to audio device etc).