Overlay vs. new section ID

[bvisness]

@rossberg left a comment on #2 expressing disapproval of the 0x01 0xFF invalid-name tactic currently proposed in the overview:

Uh, I don't think this overlay is a good idea. 0x01 0xFF is a perfectly fine byte vector, it just isn't valid UTF-8. Some systems may not do UTF-8 validation (which would break by this), and even for the standard, we may want to keep it optional as a future option. So I don't think we should go down that particular road.

Instead, I would suggest a new section ID for compact import sections.

Our rationale for the current design can be found at the end of the overview. In short, adding a new section ID seems superfluous and raises questions of how import order would be mapped between the binary format and text format.

I am curious for thoughts on the matter. I dislike the idea of adding a new section ID, but it could be the better choice.

[rossberg]

In terms of mapping between binary and text format, this proposal creates a lot of room for questions regardless, in particular, because we consider one syntactic form sugar for the other. Depending on whether a tool implements it like that, it may already lose the information during the conversion. Would definitely be the case with the reference interpreter.

[bvisness]

I find it a bit surprising that 0x01 0xFF is seen as a problem; the spec is pretty clear that implementations should validate UTF-8. That said, I do understand that it is possible (and pretty easy) for an implementation to work without validating UTF-8. Is anyone aware of an actual wasm runtime that doesn't validate UTF-8 in its names?

If indeed the UTF-8 thing is a problem, then I would propose the following alternative sentinel encoding: 0x00 0x00 0xFF. That would decode as two zero-length names and an invalid externtype, which surely not even a naive implementation would parse (since there's no way to skip to the next import if you don't recognize the externtype). This would cost just one more byte and would, I admit, probably be more robust in practice.

Furthermore, this would provide a decent mechanism for doing fancier things with the import section in the future. If perhaps we wanted to add a third level, or arbitrary levels, of namespacing, you could easily signify this with 0x00 0x00 0xFE or some such. I'd much rather not create another section ID in the future when such an idea comes back around.

because we consider one syntactic form sugar for the other

Just because the spec implements an abbreviation as sugar doesn't mean implementations have to, but if we are ok with a mismatch there then I guess it doesn't really matter. I would be fine with (import "foo" "bar" ...) (import "foo" "baz" ...) being converted to the compact import encoding (with "foo" appearing only once), for example.

[rossberg]

Ah, yes. But why not go further and reuse the existing module name then? Let compact imports start with the actual module name, then an empty item name, and then a new import kind (say 0x7F, not 0xFF, to not shut the door for a future LEB reinterpretation).

Bimport ::=
  | Bname Bname Bexterntype
  | Bname 0x00 0x7F list(Bimportitem)

Bimportitem ::=
  | Bname Bexterntype

This takes one byte less than the 0x00 0x00 0xFF prefix and in fact is the natural counterpart to the intended text format.

Actually, I think we'd even want to relax that as follows:

Bimport ::=
  | nm1:Bname nm2:Bname Bexterntype
  | nm1:Bname nm2:Bname 0x7F list(Bimportitem)  -- if nm2 = ""

The difference being that this allows overlong LEB zeroes. That way, a decoder is even simpler, since it does not need to perform any look ahead at all. The only branch is at the 0x7F byte, where it branches anyway.

[bvisness]

That seems pretty good to me. Personally I bristle a bit at an empty name having semantic meaning, but at the end of the day I just want to get the irrelevant name out of the way in as few bytes as possible, without overly burdening parsers, so I'll take it.

I do also appreciate that you can just unconditionally parse the module name in this scheme. I can imagine a future where the module name could also be conditionally parsed, but in that case we could just pull the same trick again and use the two-empty-names thing I proposed. So what you propose here seems to check all the boxes for me.

@eqrion, any thoughts on the new encoding Rossberg has proposed? Personally I am happy with it.

[bvisness]

We like this encoding and are going to update the proposal to reflect it.