Discussion issue for #830: balloting of error handling

[aphillips]

Discussion thread for error handling

This issue provides a discussion space for questions or comments on the balloting of 'error handling' currently (2024-07-16 through 2024-07-21) taking place in issue #830.

Useful references:

• Errors is the current specification
• The spec should not force implementations to go against their standard error handling patterns #782
• Fix #782: give implementations more flexibility in error handling #795
• Add design doc for error handling #804
• Refine error handling text #816
• 2024-07-15 discussion
• 2024-07-08 discussion
• 2024-07-01 discussion
• 2024-07-24 discussion

Some terminology:

Formatting attempt means a call to a message format implementation for a given message with a set of arguments intended for formatting.

Signal an error is a deliberately vague, generic, neutral way of referring to how an implementation registers that an error has occurred during a formatting attempt with the caller. Common signaling mechanisms include throwing exceptions, returning a value that indicates an error, setting an error flag on the formatter object, and many more.

Provide a fallback representation means that there is some way for the caller to obtain a version of the message that is partially formatted according to the rules already provided in Formatting and notably, but not exclusively, here and here

MUST and SHOULD have their normal RFC2119 / BCP14 meaning.

[aphillips]

@macchiati commented:

---

I can't really answer unless the question is a bit more clear.

1. "for signaling errors" - If this were "Must provide a mechanism for
   detecting errors" I would pick a higher number. That is, it could be
   satisfied by throwing an exception, or by having an additional return
   parameter, or by providing a separate function to query whether there was
   an error.

2. I think the question might depend on the type of errors (This division
   doesn't align with the typology in the spec, because it is "behavior based"
   based.)

   1. no matter what the input parameters are — eg syntax errors like{$abc
      $def}
   2. call-site mismatch errors — eg format(myDateMessage,
      date="Einstein"), or missing input parameter
   3. others

Definitely for (1) I don't think there has to be a fallback message result

[aphillips]

@macchiati

for signaling errors" - If this were "Must provide a mechanism for detecting errors" I would pick a higher number. That is, it could be satisfied by throwing an exception, or by having an additional return parameter, or by providing a separate function to query whether there was an error.

This is precisely the meaning of "signal an error". See above. That is, we cannot (because of diversity in languages and frameworks) say exactly how errors are signaled to users.

Definitely for (1) I don't think there has to be a fallback message result

There can be a fallback message result for syntax and data model errors, but it will not be a very useful message, since the user's intention generally cannot be intuited from a broken message. The fallback string (unless overridden by an implementation-specific fallback, which is permitted by the spec) for syntax and data model errors is what we euphemistically call "the logo", i.e. the string "{�}".

One of the key questions in this balloting is whether we require that implementations provide access to a fallback representation in all cases or whether it is optional.

[macchiati]

Thanks for the background.

…

On Wed, Jul 17, 2024 at 10:36 AM Addison Phillips ***@***.***> wrote: @macchiati <https://github.com/macchiati> for signaling errors" - If this were "Must provide a mechanism for detecting errors" I would pick a higher number. That is, it could be satisfied by throwing an exception, or by having an additional return parameter, or by providing a separate function to query whether there was an error. This is precisely the meaning of "signal an error". See above. That is, we cannot (because of diversity in languages and frameworks) say exactly how errors are signaled to users. Definitely for (1) I don't think there has to be a fallback message result There can be a fallback message result for syntax and data model errors, but it will not be a very useful message, since the user's intention generally cannot be intuited from a broken message. The fallback string (unless overridden by an implementation-specific fallback, which is permitted by the spec) for syntax and data model errors is what euphemistically call "the logo", i.e. the string "{�}". One of the key questions in this balloting is whether we *require* that implementations provide access to a fallback representation in all cases or whether it is optional. — Reply to this email directly, view it on GitHub <#831 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACJLEMFQXJ7P5QEZUPSEZWDZM2TSDAVCNFSM6AAAAABLBCEKSSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMZTHA2DINBYHA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[macchiati]

Another clarification. "must signal errors" encompasses all and only those errors listed in the spec, right?

…

On Wed, Jul 17, 2024 at 10:51 AM Mark Davis Ⓤ ***@***.***> wrote: Thanks for the background. On Wed, Jul 17, 2024 at 10:36 AM Addison Phillips < ***@***.***> wrote: > @macchiati <https://github.com/macchiati> > > for signaling errors" - If this were "Must provide a mechanism for > detecting errors" I would pick a higher number. That is, it could be > satisfied by throwing an exception, or by having an additional return > parameter, or by providing a separate function to query whether there was > an error. > > This is precisely the meaning of "signal an error". See above. That is, > we cannot (because of diversity in languages and frameworks) say exactly > how errors are signaled to users. > > Definitely for (1) I don't think there has to be a fallback message result > > There can be a fallback message result for syntax and data model errors, > but it will not be a very useful message, since the user's intention > generally cannot be intuited from a broken message. The fallback string > (unless overridden by an implementation-specific fallback, which is > permitted by the spec) for syntax and data model errors is what > euphemistically call "the logo", i.e. the string "{�}". > > One of the key questions in this balloting is whether we *require* that > implementations provide access to a fallback representation in all cases or > whether it is optional. > > — > Reply to this email directly, view it on GitHub > <#831 (comment)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/ACJLEMFQXJ7P5QEZUPSEZWDZM2TSDAVCNFSM6AAAAABLBCEKSSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMZTHA2DINBYHA> > . > You are receiving this because you were mentioned.Message ID: > ***@***.***> >

[macchiati]

What I really think is that the policy should be:

1. Must provide a mechanism for detecting any errors specifically listed in the spec.
2. May also support detecting errors not listed in the spec.
3. In cases of an error:
   1. Should provide fallback string result, where it can convey useful fallback for the end-user
   2. May provide fallback string result, where it cannot.

Examples:
"{�}" conveys no useful information to users
"3 cm" conveys useful information to users, even though it should have been expressed as "3 Zentimeter"

[sffc]

I don't understand the difference between "(1) MUST signal errors and MUST provide fallback" and "(2) MUST signal errors or MUST provide fallback".

I think implementations must do both, but not necesarily at the same time. For example, an implementation with two functions formatWithFallback and formatOrThrowException should be conformant, but an implementation with one but not the other perhaps should not be conformant.

I do not think an implementation should be required to have a function that does both at the same time in order to be conformant. For example, a function that throws an exception that has a fallbackMessage field would be doing both at the same time. This is fine, but not required for confornace.

Does that mean I should vote for option 1 or option 2?

[eemeli]

@sffc Option 2 would mean that an implementation would need to provide at least one of the methods formatWithFallback and formatOrThrowException, but would not be required to provide both.

Option 1 allows for a compliant implementation to provide formatWithFallback and formatOrThrowException separately, or as a single method as in the current Intl.MessageFormat JS proposal.

[sffc]

Examples: "{�}" conveys no useful information to users

To jam off this a bit: in ICU4X 2.0 DateTimeFormatter, we have the following error handling:

Pattern: 'It is:' E MMM d y G 'at' h:mm:ssSSS a zzzz

Error Type	Output String
(success)	It is: Monday November 20 2023 CE at 11:35:03.000 a.m. Greenwich Mean Time
Missing Data	It is: mon M11 20 2023 ce at 11:35:03.000 AM +0000
Missing Input Fields	It is: {E} {M} {d} {y} {G} at {h}:{m}:{s}{S} {a} {GMT+?}

"Missing Input Fields" attempts to say what type of thing was expected in a particular placeholder position, which is more useful than completely omitting it.

[eemeli]

According to our current formatting spec, the only way to get to "{�}" is if the source message contains a syntax or data model error. In all other cases, we end up with at least some pattern to format. There is a possibility that if, as proposed in #603, we drop the * pattern requirement, which would open up another path to "{�}".

[macchiati]

I think we need to be careful in talking about who it is useful for.

Looking at Shane's,

{E} {M} {d} {y} {G} at {h}:{m}:{s}{S} {a} {GMT+?}

1. It might be useful for the developer (in debugging). Although for that I would argue that an even more useful message would be something like "Missing $datetime parameter". But for debugging, a good error message is even more valuable, with internal details.
2. It is not at all useful for the end user; you really wouldn't want that message to show up in production software — or show error messages with internal details.

[aphillips]

@eemeli suggested:

@sffc Option 2 would mean that an implementation would need to provide at least one of the methods formatWithFallback and formatOrThrowException, but would not be required to provide both.

This is correct as far as it goes. It would also be compliant to have a format method that threw for some errors and returned a fallback string for others (this is the example @mihnita gave in the call, the word "or" in the name is perhaps a logical OR). Whether that's a good idea or not is a separate question. It would also be valid with option 2 to do both at the same time:

int result = format(message, argMap, target);
if (result == NO_ERROR) {
   // happy path
   print(target);
} else {
   // target contains the fallback string
}

[sffc]

I think we need to be careful in talking about who it is useful for.

That's a good point that clarifies things. There are cases where an error is more helpful (usually a programmer error), and there might cases where a fallback string is more helpful, but I don't feel super confident in specifically enumerating those caes.

[aphillips]

I think this thread might be focused on the wrong thing.

The question here is really "what does MF2 normatively require for an implementation to call itself 'conformant'?" or maybe "what can we normatively require?"

Our spec carefully enumerates the error conditions and provides tests for them (in a way that implementers can "hook up" to their own implementation, indeed are required to "hook up" more-or-less by hand). But we don't, necessarily, require that you create a specific special error state/value/class for each one. It might be perfectly valid for a Java implementation to just throw RuntimeException with the message "Stuff happened" for every error type and still be conformant with "MUST" for signaling errors. Would that suck? Yes. But that's on the implementer.

Trying to require specific error behavior (including fallbacking) is tricky because we want to allow the signal to be shaped however the implementer feels is most natural for their users, including in environments where MF2 is wrapped by resource or string management APIs and including existing APIs, which are already called by existing code that cannot be changed.

[eemeli]

My take on the overall scope of what we're considering here is that the spec should define what happens when messages are formatted, and that this should include error cases. In fact, this is one of our deliverables:

A specification for resolving messages at runtime, including runtime errors.

If we leave error handling completely out of the spec, then I think we'd need to revisit this deliverable. And I at least would much rather not need to do so.

I'm also glad that we're taking error handling and fallback behaviour seriously, as my experience indicates that message formatting/localization fails in general more often than many other parts of UX code, as it includes additional steps due to the localization of said messages. Failures in message formatting are far more often not discovered until production, as automated tests very rarely test all localizations. Therefore, it's important for the MF2 spec to ensure that users can always get at least some representation of a message via fallbacking, so that a message formatting failure can be considered only a partial failure, rather than a complete failure, of the UI.

Well-defined fallbacking (which we currently have) also ensures that any two MF2 implementations will produce the same output for the same inputs, effectively a requirement for hydration and other techniques allowing a server and client to cooperate in building a UI.