enum mpack_type_t only exposes only biggest int and uint type

[xor-gate]

Currently i'm implementing strict protocol checking on the messagepack transport. Only the mpack_type_t exposes int or uint types (8,16,32 -> 64 upcasted). But messagepack has much more types. For our current protocol we have to check if the value of a key is strict uint8_t but this is currently not possible with mpack. My suggestion would be to have a small API breakage and add all uint* and int* sizes. And still put the value in the uint64_t, int64_t in the union. What do you think?

[ludocode]

I spent a bit of time thinking this over. This isn't really the approach MPack takes to MessagePack types, so it may not be well suited for this. MPack doesn't consider the different integer widths as different types at all, which is why it transparently converts between them in both reading and writing. For example mpack_expect_u8() will allow any integer type (signed or unsigned) and do a range check against uint8_t rather than checking the width of the value.

MPack even transparently converts between signedness in all but the dynamic tag/node readers, and it's only because uint64_t has values not representable in the range of int64_t, so they need to be stored separately in the union. For example using mpack_write_tag() with the signed mpack_type_int and a positive value will actually write the shortest unsigned representation rather than signed.

Despite this though, I could see some value in exposing the native width as part of the tag. It could be useful for some more strictly typed uses of MessagePack. The spec doesn't require shortest representation for example, so an application could encode integers in fixed-size types regardless of value, and could check on reading that they are using exactly this type. This would allow encoding and decoding messages with an exact size and layout regardless of contained values (as long as your schema doesn't have variable-length arrays or maps.) It could also be useful for enforcing maximum message sizes, since MPack currently internally does not require the shortest representation.

But I don't think adding them as separate types in the enum is a good idea. This is actually what CMP does, and it makes it a real pain to switch on the type because every different type has so many possible representations.

Instead, a better approach might be to add a width value to tags and nodes. This could be an enum with values like mpack_width_fix, mpack_width_8, mpack_width_16, etc. Or it could just be an integer number of bytes; 0 for a fix type, 1 for i8/u8/map8/etc, 4 for i32/u32/map32/etc. Using an integer might make it easier to do ordered comparisons (especially in C++ where an enum isn't as easily/transparently converted to int). For example if you want your type to be encoded in at most the u16 type (so either fixuint, u8 or u16), you could check tag.width <= 2.

Note however that writing with MPack will still always encode in the shortest representation, and will convert non-negative signed values to unsigned. So you may need more work to actually generate data in this more strict schema that you're trying to verify. If an enum was used for width as above, we could potentially add a special width called mpack_width_auto. This would never be returned when reading a tag, but if specified when writing a tag (the default), it would mean to use the shortest representation; otherwise mpack_write_tag() would write exactly in the representation given regardless of value.

I do have some performance concerns with exposing the width. The size of nodes in particular are carefully tuned to be exactly 16 bits, and this would either increase their size or require a bitfield. This could dramatically slow down the Tree API (I made major performance improvements by removing the exttype from nodes for example which dropped the size to 16 bits.) But since this would likely be a rarely-used feature we could preproc it out under something like MPACK_EXPOSE_WIDTH. And node sizes may grow later anyway if new types are added to MessagePack (such as timestamps.)

If your concern is only in ensuring that "at most" the u8 type (for example) is used, so as to allow fixuint and u8 but not anything else, then I do have an entirely different proposal that would accomplish this goal. One of the things I wanted to do originally with MPack was to have a strict mode that requires a specific type representation for all values. I had actually started implementing this at one point, and I still have some code lying around here that has an MPACK_STRICT preproc. This would forbid overlong sequences (like how they are forbidden in UTF-8), and it would also require non-negative integers to be encoded as unsigned.

What this would mean is that if you were reading in strict mode and did a range check on [0, UINT8_MAX], you would be guaranteed that the value was encoded in either fixuint or u8. If it were encoded as a signed integer or as one of the larger unsigned integer widths, it would fail the strict checks. So this might be enough to accomplish your goals. In a sense, enabling strict mode does expose the underlying type, because there is only one possible underlying type for each value.

I didn't end up finishing strict mode and instead cleaned it out of the code because the spec doesn't require it and no other implementation (that I know of) does it. The author of the MessagePack spec in particular does not want to enforce representations as part of the spec at all unless it significantly simplifies decoding (for example here.)

But I think despite this, I do want to re-introduce strict mode and possibly even make it the default. No sane implementation would encode MessagePack inefficiently even if you were adhering to a schema, unless you wanted an exact message size and layout. Encoding non-negative signed values as unsigned is natural to properly optimize with fixuint, so all implementations already do this as well. So having it on by default would be perfectly fine, and would have the advantage of enforcing maximum message sizes which would add some security for free.

Ultimately we could always do both, have the width enum with an "auto" type for writing and the strict-mode preprocs, so MPack could do everything above. But I'd like to keep things as simple as possible and only implement what would actually be used. I imagine I would enable strict mode in my own apps so I'll probably implement this eventually. If it will solve your problem as well, I could do it now (and it's probably less work than exposing widths.) Or if you want widths to be exposed, we could do that as well. Let me know what you think.

[ludocode]

I still haven't implemented a strict mode to enforce minimum integer widths, and it looks like no other MessagePack implementations do either so I'll leave it out to keep things compatible. The protocol clarifications document explains MPack's integer conversions and I have no plans to preserve the original integer width in tags either so this can be closed.

[paulrpotts]

Sorry, I JUST NOW saw your reply to my comments on the Hacker News thread! https://news.ycombinator.com/item?id=22537250

I'm obviously not good about checking my comment follow-ups... and now can't reply because the topic has aged out, but thanks for your reply anyway. (I was furloughed for a few months due to the pandemic, and then re-hired, and had to try to pick projects up where I left off, only they were suddenly months further behind schedule).

Your explanation makes sense. The one case I had to support in my code, but which didn't make much sense to me, is that sometimes an unsigned 32-bit integer with a value that is small enough to be represented by a positive signed integer will be encoded as a signed value rather than an unsigned value. I've never seen it go the other way, encoding a positive signed value as unsigned.

No information is actually lost, but it requires my decoder to handle more special cases, which inflates the code size. That's not normally a big deal, but I'm running it on microcontrollers that are very small; my bootloader fits into 16K of compiled code, so I really try to optimize for size.

I am using my stripped-down decoding library successfully. I made myself some notes at one point towards writing an incompatible alternate version of MessagePack that indicates both source field types and encoding types, and would be "strict." This would likely be specific to C. However, I never wrote it, because then I'd have to write my own encoders, and that didn't seem worth it. I could use some kind of schema, but the thing I really liked about MessagePack was that it didn't need a schema, which could get out of sync with the destination data structure.