New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for a new class of "uniquely trackable" tokens #340

Open
zathras-crypto opened this Issue Jan 27, 2016 · 24 comments

Comments

Projects
None yet
3 participants
@zathras-crypto
Copy link

zathras-crypto commented Jan 27, 2016

A new class of tokens would be created that supports addressing each token in the collection individually.

For example, Alice creates a new property "AliceToken", which receives property ID 77. Alice grants 1000 tokens. These tokens automatically receive identifiers 1 to 1000. Each of these tokens is then uniquely addressable from a balance and movement perspective (eg 77_454).

Existing features would not support this new class of token, so the "how" of ensuring existing features only work with non-unique properties needs to be considered.

The nomenclature for this new type of property needs to be considered.

An initial proof of concept would provide:

  • The ability to create properties that allow the tokens to be addressed uniquely
  • The ability to track the owner (address) of each unique token
  • The ability to grant tokens of this new property type (^1)
  • The ability to move individual tokens of this new property type (^2)
  • The ability to move lists of tokens of this new property type
  • The ability to move ranges of tokens of this new property type

^1 Note, grants should return a range (eg start and end) noting the unique identifiers that were created (eg granting 10 to a property with 1000 existing tokens should return start:1001, end: 1011)

^2 Note, initially it is suggested that a simple send payload may contain two additional INT64 fields following the property id that denotes the range of token id (eg start: 5, end: 5 to send just token ID 5, or start: 5, end 10 to send range 5 to 10). The question of lists is open. The question of what to do by default with a simple send of just a token balance (no range) is also open..

A simple initial test would be:

  1. Alice creates a property A
  2. Alice grants some tokens 1-1000
  3. Alice sends property A_123 and A_321 to Bob
  4. Bob sends property A_123 to Charles
  • Alice should have a balance of 998 of property A, and listing her tokens should show that she owns A_1 to A_122, A_124 to A_320, and A_322 to A_1000.
  • Bob should have a balance of 1 of property A, and listing his tokens should show that he owns A_321.
  • Charles should have a balance of 1 of property A, and listing his tokens should show that he owns A_123.

@dexX7 dexX7 added the feature label Jan 28, 2016

@dexX7

This comment has been minimized.

Copy link
Member

dexX7 commented Jan 28, 2016

Just some food for your thoughts:

We need to be smart in regards to the data representation. Consider the worst case: for each token/property there are 2^63-1 identifiers in theory. If each unqiue token/identifier is represented by a single bit, then that's more than 1 exabyte of data. x_x

@marv-engine suggested to use sparse arrays, but the worst case here (user owns MAX tokens) is similar bad, so I think we need to work with ranges right from the start. The downside would be: the more mixed all tokens are, the more space we need.

@zathras-crypto

This comment has been minimized.

Copy link

zathras-crypto commented Jan 28, 2016

I see - so if someone granted 9 gazillion tokens, and then we stored those as 9 gazillion unique entities then we would be storing too much data. I think the key is in working with ranges, and being efficient in how we store data.

I completely agree "Alice owns 1-998 and 1000-5000 & Bob owns 999" is far more feasible than "Alice owns 1, Alice owns 2, Alice owns 3, Alice owns 4, {Alice owns ...}, Bob owns 999".

Do you have much experience working with ranges, and any thoughts on what kind of data types we could use?

@dexX7

This comment has been minimized.

Copy link
Member

dexX7 commented Jan 28, 2016

I completely agree "Alice owns 1-998 and 1000-5000 & Bob owns 999" is far more feasible than "Alice owns 1, Alice owns 2, Alice owns 3, Alice owns 4, {Alice owns ...}, Bob owns 999".

That's a great example. :)

Do you have much experience working with ranges, and any thoughts on what kind of data types we could use?

Not really. Just some thoughts: we could represent a range with two INT64 (start, end), and balances, on the lowest level, as set/collection of ranges. The actual balance/amount then would be the sum of ranges. When adding tokens, a new range would be inserted, and when removing tokens, one or more ranges could be removed or replaced. Ideally after each update there would a merge/combination of adjoined ranges (if there are any).

@marvgmail

This comment has been minimized.

Copy link

marvgmail commented Jan 29, 2016

The spec should state that "uniquely trackable" (UT) tokens are indivisible.

Existing features would not support this new class of token

@zathras-crypto are you referring to the current versions of existing features or future versions as well?

  • If we use the dollar bill metaphor, people pay with cash without any consideration for the serial numbers on each bill. At a minimum, the UI/UX should mimic this, and I don't see why the protocol shouldn't support it directly. If the user doesn't specify which individual tokens he's transacting, then I think the protocol just needs a deterministic mechanism to select the ones affected, e.g. lowest numbered or highest numbered in the affected balance.

We have to prevent double use of UT tokens, so any tx that reserves tokens would have to be enhanced to capture the serial numbers of those tokens so those serial numbers aren't part of the available balance of tokens.

  • As for initial implementation, I suggest we identify the max number (or order of magnitude) of UT tokens that could be supported in the code. If that number meets the initial application requirements, then we should go with that to start, keeping in mind that we don't want to make any decisions that preclude a larger number in the future.
  • Just like with blocks in files, over time the sequence of serial numbers for a UT token owned by an address is likely to become fragmented. We need to confirm the implementation can handle the worst case, maybe through a combination of protocol and application.

@zathras-crypto in your ^1 above, the "end" value should be 1010, not 1011.

@marvgmail

This comment has been minimized.

Copy link

marvgmail commented Feb 1, 2016

naming suggestion - call them uniquely identifiable tokens, or UID tokens.

@dexX7

This comment has been minimized.

Copy link
Member

dexX7 commented Feb 4, 2016

If the user doesn't specify which individual tokens he's transacting, then I think the protocol just needs a deterministic mechanism to select the ones affected, e.g. lowest numbered or highest numbered in the affected balance.

Based on the minimum requirements we don't need to support the normal transaction types, but we are fine with one transaction type to grant/create tokens, and one to move tokens.

Other than that I fully agree, and more to this point soon.

We have to prevent double use of UT tokens, so any tx that reserves tokens would have to be enhanced to capture the serial numbers of those tokens so those serial numbers aren't part of the available balance of tokens.

Internally we use different "buckets" for each balance type (i.e. "available", "reserved", ...), and if we enhance the balance handling for unique tokens, then we'd basically move a bunch of UT from one bucket to another, without running into the risk of some fuzzyness.

As for initial implementation, I suggest we identify the max number (or order of magnitude) of UT tokens that could be supported in the code.

How would you tackle this? It's hard for me to determine which amount is too much.

Just like with blocks in files, over time the sequence of serial numbers for a UT token owned by an address is likely to become fragmented. We need to confirm the implementation can handle the worst case, maybe through a combination of protocol and application.

We can't handle the worst case, see: #340 (comment)

@dexX7

This comment has been minimized.

Copy link
Member

dexX7 commented Feb 4, 2016

A few notes about what I had in mind: I believe we might support UID tokens with all transaction types with moderate adjustments:

Currently all balance/tally related updates go through update_tally_map, so this is the layer we need to tackle. As Marv already proposed: if we come up with a deterministic rule set for movements of unspecified batches of tokens, we could support normal transaction types as well.

For example, Alice owns [0-9, 100-125, 300-350] and wants to send 15 UID tokens via a normal simple send to Bob, then we could take 5 tokens either from the beginning or the end, so Alice ends up with [0-9, 100-125, 300-345] and Bob ends up with [345-350].

1) Refine balance operations

We'd need more specific and atomic balance operations to:

  • Create new tokens (e.g. when creating properties, granting tokens, ...)
  • Destroy tokens (e.g. when burning STO fees, revoking tokens, ...)
  • Transfer tokens (e.g. when sending tokens, ...)
  • Change the state of tokens (e.g. when reserving tokens, ...)

This could be adopted right now without tackling unique tokens at all, simply by wrapping update_tally_map with something like this.

Furthermore, to support sending a single unique token or a specific range of unique tokens, we'd also need a new balance operation to:

  • Transfer specific collection/range of tokens

2) Update global tally map to support unique tokens

The backend needs to be adjusted to handle unique tokens. I'm not yet sure how exactly, and it's probably not so simple. @zathras-crypto pushed already a new balance backend, which looks very, very handy. It could be thinkable to handle all balances as if they were uniquely trackable, and it would be a change on the lowest levels (i.e. CMPTally, mp_tally_map, ...).

3) Introduce a new token type

Currently we have the token types:

  • 1: indivisible
  • 2: divisible

And I propose to add:

  • 5: trackable, indivisible
  • 6: trackable, divisible

The normal transaction types would only care whether tokens are indivisible or divisible and use the deterministic rule set for moving tokens, while a new transaction type to move a specific range of tokens would require the trackable property.

4) Add new transaction type to move unique tokens

As previously discussed: there should be one or more new transaction types to move a specific collection of unique tokens. This new transaction type only works with trackable tokens.

5) Add unique token specific balance retrievals

Only for trackable tokens, it should be possible to retrieve balances on a unique token level, while retrieving balances of non-trackable tokens should not show exactly who owns which tokens.


These are just some ideas, and I'm curious what you guys think. :)

@zathras-crypto

This comment has been minimized.

Copy link

zathras-crypto commented Feb 4, 2016

I believe we might support UID tokens with all transaction types with moderate adjustments:

I'm concerned about "taking on too much" here. The principle of unique tokens is that "each token in a collection is different". I don't see how we can support all existing transaction types fundamentally when those existing transaction types were designed to operate on "each token in a collection is the same" principle. They seem quite different principles.

Let's take MetaDEx for example, say I want to trade 5 ZathrasCoins, but ZathrasCoin[1] is worth ten times as much as ZathrasCoin[2] - it just doesn't seem to 'fit' with existing code. With 'default' rules about automatically using the top or bottom tokens from the collection, doesn't that open the door to "I didn't mean to do that"? I've always liked the fact Omni requires you to be explicit about what you want to do :)

Update global tally map to support unique tokens

This is where I'm really having a hard time, because I don't see how we can use something like the tally map for unique tokens. A tally by definition is just a sum (eg "tally up the votes", or "tally up the income" etc - it's just an amount, not a collection of unique items). The requirement to use ranges to keep the addressable space down is another complicating factor.

Introduce a new token type:
5: trackable, indivisible
6: trackable, divisible

I support this, sure :)

Add new transaction type to move unique tokens

I was actually kind of thinking just a new version of simple send, except this time it supports a start and end token ID instead of an amount field (eg send {100,200} of prop X). So v0 simple sends would be used for existing token types, and v1 simple sends would be used for unique token types.

Just a thought, we could have an entirely new send type also - but I'm not sure the best approach.

Only for trackable tokens, it should be possible to retrieve balances on a unique token level, while retrieving balances of non-trackable tokens should not show exactly who owns which tokens.

Agreed here - the levelDB stuff I did had a helper "GetUniqueTokenOwner" which would find the address that owns a particular token. The reverse also needs to be available (listing which tokens an address owns).

These are just some ideas, and I'm curious what you guys think. :)

Sure sure, and please don't think I'm being negative - I'm just really struggling to find a model to move towards implementation. I know @CraigSellars is really keen on a PoC soon, and whilst I've played with a few ideas, the levelDB stuff was the only tangible thing that I came up with to work with unique tokens - not to say that it's the right way to go - just that it's all I've been able to come up with so far hehe :)

Thanks dude

@dexX7

This comment has been minimized.

Copy link
Member

dexX7 commented Feb 5, 2016

Fair enough, I see your points. I'm probably overthinking this, and since we're just tackling a PoC, the simpler the better, otherwise we probably open Pandora's box. :)

My only concern with this is that I assume we might start to reinvent the wheel at some point, because more and more transaction types are needed, and we then end up with two systems within one.

[MetaDEx] just doesn't seem to 'fit' with existing code. With 'default' rules about automatically using the top or bottom tokens from the collection, doesn't that open the door to "I didn't mean to do that"?

Yeah, the idea would be to take the first/last, and in this case it would be really unhandy. For the creation transactions (e.g. crowdsales, creating fixed supply tokens, creating managed tokens) it would be a nice way to distribute the initial "blank" tokens though.

Just a thought, we could have an entirely new send type also - but I'm not sure the best approach.

I tend to prefer a new transaction type, especially if the normal and unique token systems are completely seperate.

On the other hand, this raises also the question about issuance: initially I thought we might extend the creation transactions and issue trackable tokens, if they are created with type 5 or 6, but now I'm not longer sure.

Sure sure, and please don't think I'm being negative - I'm just really struggling to find a model to move towards implementation.

Don't worry, and thanks for your feedback. I agree with you here now, and we should prioritize a proof of concept first. We can always refine and update everything later, if we needed. :)

@dexX7

This comment has been minimized.

Copy link
Member

dexX7 commented Feb 7, 2016

It would be nice, if we could define what we need and are going to do.

In particular, what we need to decide: do we want to use:

  1. the old transaction types with a version pump (e.g. simple send v1, issue managed v1, ...), or rather new transaction types (e.g. send unique, issue unique managed, ...)?
  2. are we going with new property types (e.g. unqiue and divisible, ...)?

I think with @zathras-crypto great basis, which looks awesome, with a few potential optimiziations here and there (and reorg support!), getting a PoC out is pretty straight forward once those questions are answered.

As a blue print for adding new transaction types, the send all PR could be used: https://github.com/OmniLayer/omnicore/pull/155/files

@zathras-crypto

This comment has been minimized.

Copy link

zathras-crypto commented Feb 7, 2016

Sorry for delayed response bud, I've been away this weekend...

My only concern with this is that I assume we might start to reinvent the wheel at some point, because more and more transaction types are needed, and we then end up with two systems within one.

That's a really good point, and I completely agree. We do not want to end up with two systems in one.

Yeah, the idea would be to take the first/last, and in this case it would be really unhandy. For the creation transactions (e.g. crowdsales, creating fixed supply tokens, creating managed tokens) it would be a nice way to distribute the initial "blank" tokens though.

My understanding at this stage is that the requirements are based on token management, so for the time being only managed issuance is required (though fixed would be simple enough too). Crowdsales we could look at later I think.

I tend to prefer a new transaction type, especially if the normal and unique token systems are completely seperate.

I'm happy with a separate transaction type for sending unique tokens, though as per your earlier comment about systems within systems I'm not sure the normal and unique systems should be entirely separate. Though if we did separate them perhaps the simplest model is to have an additional ecosystem.

It would be nice, if we could define what we need and are going to do.

Sure sure, I've just kind of been playing with ideas - we'll need to go to something structured at some point I'd wholeheartedly agree! Basically I've just been trying to deliver the items for the PoC I listed in the OP in some form of functional manner, which we can then work on refining.

In particular, what we need to decide: do we want to use: t
the old transaction types with a version pump (e.g. simple send v1, issue managed v1, ...), or rather new transaction types (e.g. send unique, issue unique managed, ...)?
are we going with new property types (e.g. unqiue and divisible, ...)?

My votes in this area are:

  • New types (5 and 6) for unique tokens
  • Create managed transaction needs no changes to packet structure to support unique tokens so doesn't really need a version change & thus should be re-used
  • Grant transaction needs no changes to packet structure to support unique tokens so as above
  • Simple send I am ambivalent on whether we go a new version or a new transaction type

I think with @zathras-crypto great basis, which looks awesome, with a few potential optimiziations here and there (and reorg support!), getting a PoC out is pretty straight forward once those questions are answered.

Thanks dude... FYI I'm getting close to a functional PoC based on my levelDB stuff which I can push up to your repo when it's functional over the next day or two. I figured if I at least gave us a starting point we could rip and replace pieces as we saw fit (even replace it all if we need to) but at least we'd have a base to collaborate from :)

What do you think?

@zathras-crypto

This comment has been minimized.

Copy link

zathras-crypto commented Feb 7, 2016

Bad copy/paste, apologies.

@zathras-crypto

This comment has been minimized.

Copy link

zathras-crypto commented Feb 7, 2016

Trying to help out an old colleague, don't think he would be too happy about me pasting his scripts into GitHub hehehe :)

@zathras-crypto

This comment has been minimized.

Copy link

zathras-crypto commented Feb 8, 2016

Just a note here, while working this some more I wonder whether supporting divisible types is even feasible when working with unique tokens? If a unique token is divisible, can two parties own "half" a unique token each?

For now I'm just going to focus on indivisible.

@dexX7

This comment has been minimized.

Copy link
Member

dexX7 commented Feb 8, 2016

I'm happy with a separate transaction type for sending unique tokens, though as per your earlier comment about systems within systems I'm not sure the normal and unique systems should be entirely separate.

Our logic often checks only the transaction type, but not the transaction version (e.g. in the UI). This is mostly the reason I suggested a new type instead of a version bump. I agree that the systems should not be completely seperate (if possible).

Bad copy/paste, apologies.

No worries. :)

Just a note here, while working this some more I wonder whether supporting divisible types is even feasible when working with unique tokens? If a unique token is divisible, can two parties own "half" a unique token each?

That's a fair point. I always considered divisibility as hint for the UI, but it really makes no sense. :)

What do you think?

Sounds perfect!

@marvgmail

This comment has been minimized.

Copy link

marvgmail commented Feb 17, 2016

@zathras-crypto @dexX7 @achamely and anyone else, here's my proposal for representing each range of UID tokens in transaction messages. It's based on the chat conversation during yesterday's All Hands mtg. The goal is to use as few bytes as possible to represent the range. Each unsigned integer value is represented in the fewest number of bytes possible.

Notes:

  1. A range can include 1 or more valid UIDs for the referenced UID token, e.g. UID 3, UIDs 27-1234.
  2. A UID and the number of UIDs in the range can be represented as an unsigned integer in 1 to 8 bytes.

The first byte contains two 4-bit values:

  • the lower 4 bits are the number of bytes, SB, (1 to 8) required to represent the starting UID
  • the upper 4 bits are the number of bytes, IB, (0 to 8) required to represent the incremental number of UIDs in the range, beyond the starting UID.

The next SB number of bytes contains the starting UID.

The next IB number of bytes contains the incremental number of UIDs to reach the end of the range. If just a single UID is in this range, then IB is 0 and this incremental number field is not in the message.

Examples:

  1. 1 to 1
    • byte 1: 1 (IB = 0, SB = 1)
    • byte 2: 1 (starting UID)
    • no bytes for the increment because IB = 0
  2. 256 to 256
    • byte 1: 2 (IB = 0, SB = 2)
    • bytes 2,3: 256 (starting UID, a 2-byte integer)
    • no bytes for the increment because IB = 0
  3. 251 to 255
    • byte 1: 17 (IB = 1, SB = 1)
    • byte 2: 251 (starting UID, a 1-byte integer)
    • byte 3: 4 (incremental UIDs to reach the end of the range, a 1-byte integer)
  4. 50,000 to 2,000,000
    • byte 1: 50 (IB = 3, SB = 2)
    • bytes 2,3: 50000 (starting UID, a 2-byte integer)
    • bytes 4,5,6: 1950000 (incremental UIDs to reach the end of the range, a 3-byte integer)
  5. 50,001 to 50,200
  • byte 1: 18 (IB = 1, SB = 2)
  • bytes 2,3: 50001 (starting UID, a 2-byte integer)
  • byte 4: 199 (incremental UIDs to reach the end of the range, a 1-byte integer)
@dexX7

This comment has been minimized.

Copy link
Member

dexX7 commented Feb 17, 2016

I feel like these are two different topics:

  • how does the message format look like?
  • can we compress numbers?

The latter question is a broad one and I'd really like the idea of hovering the water, e.g. with the new "send list of unique ranges" transaction type. It would be great, if we use something that could be adopted for other transaction types as well at some point in the future.

@zathras-crypto

This comment has been minimized.

Copy link

zathras-crypto commented Feb 18, 2016

Payload compression I would also like to attack "holistically" rather than per message. We should have a helper function that any payload can be pushed into regardless of content which returns a compressed payload for attaching to a transaction.

Basically a "Class D" that supports payload compression (and potentially multiple payloads per transaction, but that's another topic) I think is the way to go to maximize efficiency of data storage within transactions.

I do agree that we also need to be efficient within the payload but if we stay with fixed size integers (like we have been doing before), any compression we add in holistically is going to have no problems compressing a bunch of leading zeros.

Note, since UT stuff is all testnet only, we can add in compression and use it as a test case for sure :)

@dexX7

This comment has been minimized.

Copy link
Member

dexX7 commented Feb 18, 2016

To summarize, we basically want either one or two new transaction types:

  • send a list of unique tokens: [1, 3, 5, 6, 7, ...]
  • send a list of unique token ranges: [1-1, 3-3, 5-7, ...]

For the message format, without anything facy, we probably need to push the number of elements to come:

[version] [tx type] [elements to come] [element 1] [element 2] [element 3] ...

Whereby an element in the first case would be a unique token identifier, and in the second case a range (consisting of two numbers).

For the first case, it could look like this:

UINT16 UINT16 UINTxx INT64 INT64 INT64 ...
^      ^      ^      ^     ^     ^ token id 3
|      |      |      |     | token id 2
|      |      |      | token id 1     
|      |      | elements to come
|      | tx type
| version

And for the second:

UINT16 UINT16 UINTxx INT64 INT64 ...
^      ^      ^      ^     ^
|      |      |      |     | range end/...
|      |      |      | range start
|      |      | elements to come
|      | tx type
| version

Marv proposed a new approach to encode numbers, by prepending the number of bytes of a number to come for each range.

My point was just: instead of using a new approach tailored to these transaction types, it would be great, if we could use something that might be used for other transaction types in the future as well. Just some examples:

Compress the whole payload:

COMPRESS(UINT16 UINT16 UINTxx INT64 INT64 ...)

Or use some variable length encoding approach:

VARINT(UINT16) VARINT(UINT16) VARINT(UINTxx) VARINT(INT64) VARINT(INT64) ...`

My other point was: compression/smarter number representation and the UIT stuff seem like two different topics, and while I'd welcome to tackle this, I'm not really a fan of a tailored approach, so to speak, which is only useful for the "send list of ..." transaction types. :)

But as Z mentioned: it's testnet, and this is a proof of concept, so we can test anything, including Marv's scheme. :)

@marvgmail

This comment has been minimized.

Copy link

marvgmail commented Feb 18, 2016

Here is a method to reduce the number of bytes required to represent the number of tokens and a token UID in a transaction. This method can be used wherever the number of tokens or a token UID is represented.

This approach takes advantage of the fact that both the number of tokens and a token UID is never < 0 (and maybe never < 1). So we can use the sign bit as an indicator that what would be the high order byte of an 8-byte signed integer instead contains a count of the number of following bytes that represent the value.

Integer Value Indicator Byte # Value Bytes # Bytes Used # Bytes Saved
0 1000 0000 0 1 7
1 to 255 1000 0001 1 2 6
256 to 65,535 1000 0010 2 3 5
65,536 to 16,777,215 1000 0011 3 4 4
16,777,216 to 4,294,967,295 1000 0100 4 5 3
4,294,967,296 to 1,099,511,627,775 1000 0101 5 6 2
1,099,511,627,776 to 281,474,976,710,655 1000 0110 6 7 1
>= 281,474,976,710,656 none 8 8 0

To Read a Numeric Value

Examine the high order bit in what would be the high order byte of an 8-byte signed integer. That's the sign bit. For the number of tokens or a token UID, negative numbers are not allowed.

If that bit = 1 then the low order 3 bits in that byte contain the number (0-6) of following bytes that represent the numeric value.
Else if that bit = 0 then that byte is the high order byte of an 8-byte signed integer representing the numeric value.

@marvgmail

This comment has been minimized.

Copy link

marvgmail commented Feb 18, 2016

There are 4 unused bits in the Indicator Byte, but I haven't thought of a way to make use of them.

@zathras-crypto

This comment has been minimized.

Copy link

zathras-crypto commented Feb 18, 2016

Thanks guys...

I think I may be in the minority here, but I am concerned that we're making this overly complex for the sake of saving a byte here and there (overly complex for the user and overly complex from a coding standpoint).

I agree there are minor efficiencies we can achieve if we engineer the payload for the exact specific use cases, but I disagree this is the right thing to do... To be max efficient we would need to structure the packet to the case, having to provide for example:

  1. omni_senduniquesingletoken (a single token ID)
  2. omni_senduniquesingletokenrange (a start and end token ID)
  3. omni_senduniquelistofsingletokens (a list of single token IDs)
  4. omni_senduniquelistoftokenranges (a list of start and end token IDs)

On the opposite side, I believe it is simpler to simply have:

  1. omni_sendunique (which would do everything)

This may cost us an extra byte or two here and there, but avoids lots of over-engineering IMO. The packet structure could simply be:

UINT16 UINT16 UINTxx INT64 INT64 INT64 ...
^      ^      ^      ^     ^     ^ second range start
|      |      |      |     | first range end
|      |      |      | first range start  
|      |      | number of ranges
|      | tx type
| version

This covers all use cases:

  1. Send unique token X:
    number of ranges = 1, first range start & end = X
  2. Send unique token range X-Y:
    number of ranges = 1, first range start = X, first range end = Y
  3. Send list of N tokens:
    number of ranges = N, first range start & end = X1, second range start & end = X2, third range start & end = X3 and so on,,,
  4. Send list of N token ranges:
    number of ranges = N, first range start = X1, first range end = Y1, second range start = X2, second range end = Y2, third range start = X3, third range end = Y3 and so on...

I do agree that at first look we may not necessarily be fully optimized here, but we keep things simple (which I believe is a good engineering principle hehe).

Of course there are arguments against, for example when sending single tokens we are setting start and end to the same value, thus duplicating data we are storing in the packet - but since we're going to start compressing payloads (all payloads I vote) anyway that duplicated data would be removed in the compression stage.

Compress the whole payload

This is where my vote is - payload compression should be done across all payloads, I am quite against doing payload encoding/decoding differently depending on the type of transaction - it's just going to get very messy - I view the next transaction class as having compressed payload full stop (not special payloads for special tx types).

Bit of a ramble hehe, and the ideas here are great ideas, but I just think we're heading down a path (a bunch of stuff that's unique to a single transaction type) that I don't think is in our long term interests...

Just my 2 cents :)

@marvgmail

This comment has been minimized.

Copy link

marvgmail commented Feb 18, 2016

If payload compression does the trick, then I'm all for it, as long as we understand what the limits are and what happens if we exceed those limits.

@dexX7

This comment has been minimized.

Copy link
Member

dexX7 commented Feb 19, 2016

To be max efficient we would need to structure the packet to the case, having to provide for example: ...

Actually I think having very specific transactions should be the end goal, and I'd welcome, if we untangle the protocol like this in the long run. It's similar to the overhauled MetaDEx transactions, which were split into more than one.

However, this (and also compression and alike) are low level details, something we can change under the hood without letting the end user notice (well, mostly), and for a proof-of-concept we can of course stick to a simple "one for all" transaction. :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment