feat: bit optimization

[is-it-ayush]

the problem.

Currently our token are 8 bits (u8) in length. The thing is, we only have 8 tokens and this perfectly fits into 3 bits. We can substantially reduce the size of the encoded data by migrating from 8 bit tokens to 3 bit tokens.

so what.

The PR switched from 8 bit tokens to 3 bit tokens.

gotchas.

• We gotta be extra careful around String and Bytes since the only way to decode them is to march forward on them. The problem is when marching forward you could encounter string bits that can resemble token bits causing you to exit early. The only workaround that works for this is to use fixed length tokens in the 8 bit space. Hence for strings and bytes. We return back to 8 bit space.

before.

after.