Start of new types on the client

[pbiggar]

This started as an attempt to move all the client types to match the backend ProgramTypes and RuntimeTypes, but it got to be a lot so I stopped early. So far, just all this really does is updates ID/TLIDs to use int32s, and fixes the fallout (and adds a few types that aren't used yet).

[pbiggar]

The integration tests are failing because some packages have uint64 tlids. In dev, the one package created has a 64bit tlid. In production, about half the packages have 64bit tlids.

It seems also that 1000 production toplevels have tlids above 2147483647 (all for some reason being a number like 1587041099384 but 156, 157, or 158.

Also, int_of_string is crashing so we shouldn't use it anyway. But it looks like it will make sense to switch to 64 bit ints here if possible anyway :(

[pbiggar]

Wait, if we json parse 64-bit ints, what makes it to rescript? 64bit ints or strings, or floats with missing precision?

Does it matter whether they're a field in an array (as in the json representation of variants) or as ids in a dictionary).

[pbiggar]

Parsing 64 bit ints gets us floats with a loss of precision.

I wonder if we're safe to just convert the tlids in the DB to a lower precision (except for users editing). I don't think tlids are used except to tell the apiserver what db/handler is being deleted.

[pbiggar]

Parsing 64 bit ints gets us floats with a loss of precision.

I wonder if we're safe to just convert the tlids in the DB to a lower precision (except for users editing). I don't think tlids are used except to tell the apiserver what db/handler is being deleted.

They are binary serialized into ops. Ugh.

[pbiggar]

Related: the expression 923483483489348934 saves properly but evaluates to 923483483489349000. This is due to loss of precision when encoding Dvals as int64 in json.

OCamlTypes has a string in EInteger and FPInteger, which is why programs stay valid using OCamlTypes (when sent back and forth from the apiserver). But dvals don't! Either when sent from a trace, or when sent back from Wasm.

But, I think if we encoded tlid and int64 as strings when they're too big, we could handle this ok.

[pbiggar]

OK, the plan here is to serialize int64s and uint64s (including ids and tlids using strings when they're over the max that can fix in an int. That should fix a bunch of small bugs. I'll do that in a separate PR, and then in this PR I'll switch the client to using uint64s (like the backend).

[pbiggar]

This addresses all the issues by creating a UInt64 type (backed by a combination of BigInt for printing/serialization as strings, and Int64 for use in-memory), and then using UInt64 for the TLID/ID types.

Numbers over 2^53 are serialized as strings because JSON has a 53bit limit for integers being represented accurately as themselves.

[pbiggar]

Summary and description

This has gone on a while, so I'm going to restate it all.

Originally, this started as an attempt to replace client types with the types used by the server. And indeed, this adds those types. However, after I made a few subsequent changes, it got big enough that now it's time to stop and get this merged.

The major change was changing the type of TLIDs (and IDs). The backend used uint64s, and the client used strings. Using strings led to some pretty bad situations: the backend used uint64s and so while it was mostly fine to use uint64-encoded-as-strings, in practice we snuck in lots of things that weren't uint64s.

In some cases, those were test items like ID("fake-ac-data1"). But in one place we actually put both traceIDs and IDs into the same data by wrapping the traceID in the ID constructor and pretending everything was OK. That has been fixed.

The attempt to switch to a real uint64 may have been too much work for the value we got out of it, but it's done now. I initially tried to store it as a bs-zarith U.UInt64.t type, but this was incredibly slow in practice. I also tried to store it as a JS BigInt, but using a JS type broke the comparators used in TLID.Dict (and lots of other things probably, there were thousands of broken tests).

As a result, it made more sense to come back to using uint64s-in-theory but int64s-in-practice. That is, I'm using an int64 as a uint64. Since int64 is a known type in Rescript, comparators work. Since it's simply two ints together, the performance is good.

Both uint64 and int64 use 64 bits. There wasn't tooling to just cast them between each other, so I explicitly made the translation (using the negative int64 range to hold the values above the positive int64 range).

The complexity here was converting negative int64s to positive uint64s, especially during stringification (needed for encoders and decoders). For that, I used JS BigInts - this allowed me more range, and also to use their toString functions to avoid needing to create my own.

Along the way I tried a bunch of other things, but they all came out more complex in one way: encoding it as a tuple of 2 int32, manually scaling int64s, my own version of toString, using bs-zarith, etc. All things I came up with were more complex. In retrospect, leaving them as strings and just making the type opaque might have worked, but I'd probably have missed some edge cases so this isn't a terrible outcome.

This PR also:

• renames dval encode to ocamlDval
• add RuntimeTypes to the client
• recompiles on .resi file changes
• adds tuple encoding/serialization tests
• refactors client-side encoding/decoding tests (and TestRpcs)

[StachuDotNet]

Should this file include some Tuple types? it doesn't right now

[pbiggar]

Yes it should. Wrote this before they merged, nice catch.

[StachuDotNet]

it feels a bit offputting to use negative values to represent those out of bounds, but I can't think of anything better. Looks good!

[pbiggar]

it feels a bit offputting to use negative values to represent those out of bounds, but I can't think of anything better.

Yeah. The other option is to keep the type opaque and use strings. Any preferences?

[StachuDotNet]

it feels a bit offputting to use negative values to represent those out of bounds, but I can't think of anything better.

Yeah. The other option is to keep the type opaque and use strings. Any preferences?

Not really, they're both kinda bad :) So might as well go with the thing we've already coded.