Make VarZeroVec format configurable

[Manishearth]

This finishes the 1.0 polish work necessary for #1410, switching us over to Index16 by default with Index32 being used for BlobDataProvider.

Stuff that should be done soon but is not necessary for ICU4X 1.0:

• ZeroMap support (we didn't need this, and it can be added incrementally)
• Format strategy support in custom derive
• Actual nice errors for VZV mutation operations
• Perhaps clean up the code to not cast to usize as often (and instead cast to the relevant index type)

Stuff to do eventually:

• Introduce an "Auto" strategy (will need a rewrite of the code)

[Manishearth]

datagen-full fails since the ZM2D gets too big, looks like I'm going to have to add the ZeroMap stuff after all. I'll do it tomorrow.

[sffc]

datagen-full fails since the ZM2D gets too big, looks like I'm going to have to add the ZeroMap stuff after all. I'll do it tomorrow.

Which key is overflowing?

[sffc]

Is it just the blob provider that's overflowing? Too many locales?

If the proper fix is too tricky, this is the only place where we have totally private structures, so you could make a one-off VarULE type that's transparent over [u8] and has a bigger ZeroMapKV container.

[Manishearth]

Is it just the blob provider that's overflowing? Too many locales?

Yeah.

If the proper fix is too tricky, this is the only place where we have totally private structures, so you could make a one-off VarULE type that's transparent over [u8] and has a bigger ZeroMapKV container.

Nah I'll do the proper fix. It's a bit complicated but I don't anticipate problems.

[sffc]

My bigger worry is that there's some locale that uses supplementary code points and long strings and just has really big data that overflows the time zone names struct. If that happens, we're in a bit of a pickle with the path we've chosen.

[Manishearth]

We can use Index32 in those places if we're worried, right?

[sffc]

We can use Index32 in those places if we're worried, right?

The places where we may sometimes need to use Index32 are exactly the same places where using Index16 has the biggest impact. The design of FlexZeroVec is that the 95% case gets smaller data and the 5% case can still grow.

However, hopefully this isn't actually an issue. In such cases, I may advocate to instead just break the data struct into smaller pieces (perhaps a different map for each continent or something like that).

[Manishearth]

Yeah, that's a good call.

I think that's also an acceptable place to introduce a new data struct version with Flex. That works well as an "if we're unlucky" path.

[sffc]

Cool! Don't have any changes to suggest

[sffc]

(I had already approved)