Language-agnostic types schema

[cameel]

The format/type schema currently seems strongly tied to the range of types available in Classic Solidity. I think it needs to be a bit more generic to accommodate types from languages with different or more flexible type systems. Even for Classic Solidity it's a bit limiting because it provides too little flexibility for defining new types even if they are simple variations of existing ones. If we keep it this way, it will have to be updated with each new language supporting it and also along with evolution of the languages that already do. It would be much better if the languages were free to change type representations without requiring changes to the spec each time.

Example

More concretely, the problem is that the current spec does not provide properties for exhaustively describing the types it has, instead implicitly assuming Solidity's choices. For example:

• {"kind": "int", "bits": 16} is specifically a big-endian, right-aligned, two's complement signed 16-bit integer. Different alignment cannot be expressed.
• {"kind": "address"} is a left-aligned, sequence of 20 bytes. There is no way to describe an address that always takes a full 32-byte slot (which we are planning to introduce in the near future).
• {"kind": "array", "count": 10, "contains": {"type": {"kind": "int80"}}} is an array that packs three elements per slot in storage (left-aligned), but 1 element per slot in memory (right-aligned). There is no way to have an array that is not packed in storage (and Core Solidity provides non-packed types; in fact it does not support packed ones yet).
• {"kind": "function", "internal": false} is a combination of a 20-byte address and a 4-byte selector. Packed into a single slot in calldata, but taking two separate slots on the stack. Again, no way to express different packing or address size.
• {"kind": "string"} is a sequence of (variable-length) UTF-8 characters and a length expressed in bytes.

In some cases these are EVM's design choices (e.g. the endianness and negative integer representation) and it's unlikely for a language to diverge from them (though not impossible), but others (e.g. alignment, packing and character encoding) are not universal. The latter should have corresponding properties in the spec.

Proposed solution

There are two aspects to the current types, which I think should be decoupled:

• Interpretation of the value: whether it should be presented as a number, byte, character, address, collection, function, etc. This is how a human interprets it.
• Encoding of the value: byte representation of the value and its placement in the region of memory assigned to it.

I think that types should map to interpretations and properties should define the encoding. For example I'd introduce a single "Number" type for all kinds of numbers (integers/fractions, signed/unsigned, fixed-/floating-point). It should have properties expressing the encoding: size, alignment, signedness and precision. int16 could then be described as:

{
    "kind": "number",
    "representation": "fixed-point",
    "alignment": "right",
    "signed": true,
    "bits": 16,
    "decimals": 0
}

In some cases where there are only minor differences between interpretations, we could have them covered by a single type. For example an addresses and a contract is almost the same thing. Both can be payable or not, both are internally an address, both will need to be extended to 32 bytes in the future. The difference is just that a contract type has a definition. We could cover both with an address type with an optional definition property.

[gnidan]

Thanks for suggesting this, @cameel! I'm keen to fix the type schema, since it's pretty janky (you've probably heard me say).

Before I play around with what this might look like for more examples, do you have any thoughts on what we should do about the complex types? Maybe those are already less restricted inherently and don't need as much revision.

[gnidan]

Oh, and to note: I really like your proposed approach, esp. since it matches nicely with the aesthetics of the format already! Way better than the existing type schema.

[cameel]

I do :) I'm actually just finishing up the second part of this, which is a concrete list of types I'd introduce. Will post that in a moment. It's more of a sketch than a finished design so I did not want to include that directly in the description.

[gnidan]

Oh man, the best kind of a bombshell like this is the kind that comes with more examples!

[cameel]

Done. Here's the sketch. Those are not examples specifically, just a list of types and a general description of what they are and what properties they need. I can try to provide some examples for specific cases though - let me know which aren't clear enough. Especially the composite types are admittedly a bit vague compared to the primitive ones, but we can try to flesh them out in subsequent comments.

I tried to include some types I think we'll need for Core Solidity. That's just scratching the surface though - we will need a separate discussion with solcore for that.

Proposed types

Word

Opaque value without high-level interpretation, equivalent to Core Solidity's word. Defines only the most general properties of the representation: width and alignment, which should be available for all types.

In Core Solidity it's always 32 bytes but I'm generalizing it here, because it can serve as a base for the encoding of basically everything else.

Boolean

True/false. Represented as a word.

In particular, it should be possible to express a representation packed into a single bit.

Number

The main representation is a binary fixed-point number. This covers all the flavors of uint, int, ufixed, fixed. Based on word, with two additional properties:

• signedness
• number of decimal places

We may also need rational numbers - arbitrary precision nominator/denominator representation. These are currently used for literals. In Core Solidity that will extend to general comptime calculations. I'm not sure how the debuggers will want to accomodate debugging comptime code, but we may need this representation if we want to be able to describe not just the final value but also the components of the calculation that lead to it.

Another possible representation are floating-point numbers. These have little relevance currently, so it would be fine to skip them and add later if necessary.

Decimal numbers are also a distinct possibility (be it fixed- or floating-point). This would be something like decimal in Python. We probably don't need to cover them right now either. Vyper is the closest to Python and it does not have that. Its decimal type seems to be a binary fixed-point representation instead (equivalent to fixed168x10 in Solidity)

Address

Representation based on word. Since that implies having a size, it would allow expressing both 20- and 32-byte address types.

This is one of the types where we'd have some interpretation-specific properties:

• definition: if it's a plain address, it has no definition. If it's a contract reference, we can point at one.
• payability: it does not change how it is stored. It's only relevant as a hint to the debugger that it might want to display the value differently (e.g. show balance next to it or use a different color to indicate it's not supposed to receive ether). For contracts, the compiler can indicate payability based on whether the contract has a payable fallback/receive. This property should be optional since not all languages have to make this distinction. It should be possible to express both non-payability and unknown payability.

Selector

Hash of a function signature. Represented the same way as as a fixed bytes array. In Solidity this is not a distinct type, but could be, the same way address is. Function and error selectors are 4 bytes, event selectors are 32 bytes.

Enumeration

This should have a word representation. Solidity used to use uint for it, now it uses uint8 so being able to specify width is necessary.

Interpretation properties:

• values: an ordered list of possible values
• definition (optional). Solidity enums always have definitions, but technically built-in enums without definitions could exist.

Function

To the debugger the interpretation here is a function definition in the source code it can point at. The value is useless outside of that.

There may be some other things known about the type, which do not impact interpretation:

• Parameter names. In Solidity these are allowed but not required when defining a function pointer type.
• Signature constraints: number and types of parameters and return values, mutability, etc. Note that these are not necessarily a property of the value (i.e. the function being pointed at) but the type itself. For example in Solidity one can assign a pure function to a view function pointer. Some languages could also allow assigning functions with optional arguments to pointers whose signature does not include them.

Possible representations:

• External: contract address + selector.
• Internal: opaque word.
• Lambda: opaque word.

Representations of internal and lambda functions are implementation-dependent in Solidity and this is likely to be the case in other languages too. ethdebug should have some mechanism for mapping them to source locations. However, that requires an assumption that they can be treated as IDs, which may not always be the case so it will need more consideration.

Byte

The representation is always a single byte, interpreted as a hexadecimal value in the range of 0x00..0xff.

Character

The representation here must allow specifying if it's fixed- or variable-width and the set of available characters. Both are uniquely defined by character encoding so we should have a list of those. At least ASCII, UTF-8, maybe more, e.g. UTF-16.

In Solidity the only choice is UTF-8. It's also not a standalone type, but only a component of the composite string type, which is a sequence of variable-length characters. The type does not even provide the [] operator for this reason.

Sequence

Sequence of items that fuse together to represent something uniform. Unlike an array, it's not a loose collection of independent items.

• Sequence of bytes: interpreted as a blob of binary data.
• String of characters: interpreted as a piece of text.

Representation can be static (like in bytesNN types in Solidity) or dynamic (like bytes or string). The latter has a length field. In full generality there could be representations with padding between elements, but no one is going to do that in practice so it's fine to assume no padding. On the other hand we need to allow specifying alignment for the sequence as a whole.

Array

Ordered collection of independent items.

For the purpose of presenting the value there's no difference between a static and a dynamic array, as evidenced by the fact that being able to use the [a, b, c] syntax for both is highly requested by Solidity users. This is a representation concern, expressed by the presence and placement of the length field. The representation must also define how elements are packed and aligned. The array as a whole has a separate alignment as well.

Array slices fall under that too. They have the same interpretation but potentially a different encoding.

Variant

A sum type. The equivalent of a union in C++ or enum with data in Fe or Core Solidity.

Mapping

Dictionary mapping one set of values to another.

In Solidity key and value can have names. These are a part of the interpretation, just like parameters in function pointers.

The idiomatic representation is a dictionary spanning the whole storage, but mappings can be implemented in many ways. For example one could have in-memory mappings that fit it a small hash table. Whether keys are stored or not is another property that should be possible to express.

Struct / tuple

Tuples and structs are almost the same thing. The minor difference in interpretation is whether fields have names.

Type alias

There are two kinds of these:

• Transparent alias: just a different label. Same interpretation and representation.
• Opaque alias: equivalent to UDVTs in Solidity. No interpretation but same representation as the underlying type. Technically, it does have an interpretation, but it is defined by the implementation so not easily expressible by the compiler during code generation.