Feature request: Support for human-readable blob values in the IDL

[brandondahler]

Context

Smithy currently supports defining blob values for traits as base64 encoded strings in both the IDL and JSON AST. Per both the 1.0 and 2.0 specs for trait node values, blob values are "A string value that is base64 encoded.".

Despite the spec being clear on the intended representation, existing trait validators and related code generators do not necessarily follow this specification due to the reality that most uses of blob values really are better expressed as human-readable strings.

For example:

• Some trait validators like the DefaultTraitValidator only warn on invalid values
• Some codegen implementations like smithy-typescript's HttpProtocolTestGenerator explicitly parse the blob value as an array of utf-16 bytes

Proposal

To support use-cases where human-readable blob values better serve the model author's expression of the value, we will support two new flavors of strings, referred to as byte string and byte text block, which are represented in the IDL as human-readable text but are automatically converted to the equivalent base64 value in the JSON AST. These two flavors will be differentiated by a prefix of b in front of the double quotes of the standard quoted text or text block token respectively. The parsing, whitespace, and escape character semantics will all work in the same manner as the standard token values.

Usage will look like the following:

$version: "2.0"

namespace smithy.example

@default("e30=")
blob previousString

@default(b"{}")
blob newString

@default("ewogICAgImZvbyI6ICJiYXIiCn0=")
blob previousTextBlock

@default(
    b"""
    {
        "foo": "bar"
    }
    """
)
blob newTextBlock

This would translate into the following JSON AST:

{
    "smithy": "2.0",
    "shapes": {
        "smithy.example#newString": {
            "type": "blob",
            "traits": {
                "smithy.api#default": "e30="
            }
        },
        "smithy.example#newTextBlock": {
            "type": "blob",
            "traits": {
                "smithy.api#default": "ewogICAgImZvbyI6ICJiYXIiCn0K"
            }
        },
        "smithy.example#previousString": {
            "type": "blob",
            "traits": {
                "smithy.api#default": "e30="
            }
        },
        "smithy.example#previousTextBlock": {
            "type": "blob",
            "traits": {
                "smithy.api#default": "ewogICAgImZvbyI6ICJiYXIiCn0="
            }
        }
    }
}

Open Questions

• How should a byte string be converted into a byte array in order to be base64 encoded?
  • Per the spec, IDL files are UTF-8 encoded and strings support Unicode characters with basic ASCII escape sequences and a Unicode code-point escape sequence.
  • There are significant parallels in this feature from Python's byte literals, which only support ASCII characters with hex escape sequences to allow arbitrary byte sequences. Unicode characters and the Unicode escape sequence are explicitly not allowed in those byte literals.
  • In the end, we must choose to either support arbitrary Unicode text with a specific encoding (presumptively UTF-8) or arbitrary byte sequences (by supporting hex-based escape sequences and not supporting Unicode characters or Unicode escape sequences). Support for these are mutually exclusive for a given string flavor.
  • My current inclination is that we should support Unicode characters by encoding the byte strings into UTF-8 bytes, given the prevalence of the UTF-8 encoding in modern computing. The expression of arbitrary byte sequences is already supported by base64 encoding standard strings and supporting them in byte strings is unlikely to make them any more understandable.

Changes Required

• Two new IdlTokens must be defined for the new string representations
• The Tokenizer implementations must be updated to recognize the new string representations
• The IdlNodeParser and IdlTraitParser must be updated to properly re-encode the parsed tokens as their equivalent base64 value
• Two new TreeTypes must be defined for the new string representations
• The FormatVisitor, BracketFormatter, and CapturedToken classes must be updated to properly handle the new IdlToken and TreeTypes.
• Documentation must be updated to explain the behavior of the new string flavors
• The language server and any other IDL syntax-supporting projects must be updated

Additional Commentary

This idea originated from a comment made by @JordonPhillips as part of the conversation in #2821 related to current inconsistency with blob value literal behaviors.

I plan to cut a draft PR shortly with a working proof of concept of this feature.

[JordonPhillips]

In the end, we must choose to either support arbitrary Unicode text with a specific encoding (presumptively UTF-8) or arbitrary byte sequences (by supporting hex-based escape sequences and not supporting Unicode characters or Unicode escape sequences). Support for these are mutually exclusive for a given string flavor.

is support for those mutually exclusive? Perhaps it was in Python, which had to deal with a very painful migration from 2.x where strings were somewhat ambiguous to 3.x where encodings were much more explicit. By contrast, Smithy has always declared strings to be utf8. So is there a problem in allowing arbitrary utf8 alongside hex escapes?

One motivator for me personally in doing this is the newly-introduced event stream protocol tests. One value you can specify in these is the binary format of an event in your protocol. As an example, let's take this test definition:

@eventStreamTests([
    {
        id: "BlobHeaderInput"
        protocol: restJson1
        events: [
            {
                type: "request"
                params: {
                    headers: { blobHeader: "foo" }
                }
                headers: {
                    ":message-type": { string: "event" }
                    ":event-type": { string: "headers" }
                    blobHeader: { blob: b"foo" }
                }
            }
        ]
    }
])

If you take the event from that test case, serialize it, and then base64-encode the result, you get: "AAAATQAAAD2dKQ+ADTptZXNzYWdlLXR5cGUHAAVldmVudAs6ZXZlbnQtdHlwZQcAB2hlYWRlcnMKYmxvYkhlYWRlcgYAA2Zvb5sbbGM="

That value means nothing to me, even though i'm very familiar with the encoding. However, as a byte string it's much more readable:

b"""
\x00\x00\x00\x4d\
\x00\x00\x00\x3d\
\x9d\x29\x0f\x80\
\x0d:message-type\x07\x00\x05event\
\x0b:event-type\x07\x00\x07headers\
\x0ablobHeader\x06\x00\x03foo\
\x9b\x1b\x6c\x63"""

At a glance I can see that all the header keys and values are in there, and were there a body I'd be able to see and read that too. It's also pretty easy for me to see that the types of the headers are properly declared. The rest of the framing may be a bit more obtuse, but I'd be able to notice if it were very wrong.

My question is: what would be the problem if some part of that message included, say, 🎺 ? Smithy can say that every non-escaped sequence is interpreted as utf8 bytes.

[brandondahler]

I guess my thought about their mutual exclusivity was based on the idea that they were fundamentally strings -- allowing hex escapes in a string that is in a particular encoding can result in strings that are invalid for that encoding; however, to your point, these ultimately are not strings as they are simply a fancy token that represents some bytes.

I think I fully agree with you. Having unicode characters mapped down to UTF-8 isn't a problem for alternatively encoded string => byte values (e.g. the UTF-16 representation of 🎺), it just means that you'll need to hex-encode said value.

---

In your example, I noticed two things:

1. The line continuation \<LF> doesn't currently exist and probably should for both this feature and for normal strings.
   • I think this would be acceptable for normal strings given the IDL spec states "Any other sequence following a backslash is an error."
   • Edit: This is already supported, I just missed its documentation in 18.8.3.3. Escapes in text blocks
2. A null byte escape value \0 would be useful since it is common in binary data

[brandondahler]

Having let it marinate a little, the only weirdness I can come up with is that the hex escapes (\xHH) would only be available in b-strings and explicitly not available in normal strings.

I think that's reasonable and would only pose a minor documentation challenge.