Add encoding of consumer group

[Yangsx-1]

Add consumer group related data structure.

[torwig]

Added some ideas. Please, double check the following

the timestamp of the last consumer interaction
the timestamp of the last successful consumer interaction (actually read or claimed some entries).

maybe I missed something (I think, it's hard to store the last idle time because there was no activity by the definition :) )

[torwig]

=>

The consumer group metadata contains the basic information of the consumer group used in the XINFO GROUPS command.
The key starts with the stream name, version, consumer group name. The next segment is a delimiter - a string METADATA (hardcoded value).
After the delimiter, there are:

• number of consumers in the group
• the length of the group's PEL (pending entries list)
• the ID of the last entry delivered to the group's consumers
• the ID of the last entry delivered to the group's consumers
• the number of entries in the stream that are still waiting to be delivered to the group's consumers.

[torwig]

=>

A consumer group contains several consumer and each consumer also has its own metadata which is used in the XINFO CONSUMERS command.
The key starts with the stream key, version, consumer group name, consumer name. The next segment is a delimiter - a string METADATA (hardcoded value).
After the delimiter, there are:

• the number of entries in the PEL
• the timestamp of the last consumer interaction
• the timestamp of the last successful consumer interaction (actually read or claimed some entries).

[Yangsx-1]

Added some ideas. Please, double check the following

the timestamp of the last consumer interaction
the timestamp of the last successful consumer interaction (actually read or claimed some entries).

maybe I missed something (I think, it's hard to store the last idle time because there was no activity by the definition :) )

Oh, it's the name's problem. The last idle time actually stores the timestamp of the last consumer interaction and the last active time actually stores the timestamp of the last successful consumer interaction. :)

I'll change the name in kvrocks code.

[PragmaTwice]

Several questions regarding the encoding:

• how to distinguish between EID MS|EID SEQ and group_name|..? since EID MS and EIQ SEQ is encoded as binary integer 64 bit, any value is possible (from 0 to 2^64).
• how to list all group names? if we cannot distinguish between group encoding and normal subkey encoding.

[Yangsx-1]

Several questions regarding the encoding:

• how to distinguish between EID MS|EID SEQ and group_name|..? since EID MS and EIQ SEQ is encoded as binary integer 64 bit, any value is possible (from 0 to 2^64).
• how to list all group names? if we cannot distinguish between group encoding and normal subkey encoding.

For the first question, actually there is a sizeof(stream_name), sizeof(group_name), sizeof(consumer_name) before the corresponding field which help us distinguish the different field.

For the second question, the group subkey is longer than the normal subkey of the stream entry which makes it possible to distinguish group subkey and normal stream entry subkey. You can refer https://github.com/apache/kvrocks/blob/9b1ebcd65a111ab8f8aded158d7427b7409ec4cc/src/types/redis_stream.cc#L298

[PragmaTwice]

For the first question, actually there is a sizeof(stream_name), sizeof(group_name), sizeof(consumer_name) before the corresponding field which help us distinguish the different field.

Actually only the group_name length field cannot help us to distinguish them, we should compare the total length of the subkey to distingush them (always > 16 bytes), which is very weird to me.

For the second question, the group subkey is longer than the normal subkey of the stream entry which makes it possible to distinguish group subkey and normal stream entry subkey.

We cannot list all group_names without iterating all subkeys of the stream, which is inefficient.

AS IS:

key|version|EID MS (8 bytes)|EID SEQ (8 bytes)

key|version|group_name_length (8 bytes)|group_name|METADATA(8 bytes)

key|version|group_name_length (8 bytes)|group_name|consumer_name_length (8 bytes)|consumer_name|METADATA(8 bytes)

key|version|group_name_length (8 bytes)|group_name|EID MS (8 bytes)|EID SEQ (8 bytes)

For example, if we have such an encoding:

key|version|EID MS (8 bytes)|EID SEQ (8 bytes)

(since EID MS can hardly be the max value)

key|version|(uint64_t)(-1) (8 bytes)|GROUP_META (1 byte)|group_name_length (8 bytes)|group_name

key|version|(uint64_t)(-1) (8 bytes)|CONSUMER_META (1 byte)|group_name_length (8 bytes)|group_name|consumer_name_length (8 bytes)|consumer_name

key|version|(uint64_t)(-1) (8 bytes)|PEL_ENTRY (1 byte)|group_name_length (8 bytes)|group_name|EID MS (8 bytes)|EID SEQ (8 bytes)

GROUP_META = (uint8_t)1
CONSUMER_META = (uint8_t)2
PEL_ENTRY = (uint8_t)3

Now we can list all group names just by scanning the prefix key|version|(uint64_t)(-1)|GROUP_META.
Also, we can list all PEL entries if we want in a very efficient way.

Most importantly, the lexicographic order of key distinguishes several different subkey types, instead of a mixing order.

[Yangsx-1]

For the first question, actually there is a sizeof(stream_name), sizeof(group_name), sizeof(consumer_name) before the corresponding field which help us distinguish the different field.

Actually only the group_name length field cannot help us to distinguish them, we should compare the total length of the subkey to distingush them (always > 16 bytes), which is very weird to me.

For the second question, the group subkey is longer than the normal subkey of the stream entry which makes it possible to distinguish group subkey and normal stream entry subkey.

We cannot list all group_names without iterating all subkeys of the stream, which is inefficient.

AS IS:

key|version|EID MS (8 bytes)|EID SEQ (8 bytes)

key|version|group_name_length (8 bytes)|group_name|METADATA(8 bytes)

key|version|group_name_length (8 bytes)|group_name|consumer_name_length (8 bytes)|consumer_name|METADATA(8 bytes)

key|version|group_name_length (8 bytes)|group_name|EID MS (8 bytes)|EID SEQ (8 bytes)

For example, if we have such an encoding:

key|version|EID MS (8 bytes)|EID SEQ (8 bytes)

(since EID MS can hardly be the max value)

key|version|(uint64_t)(-1) (8 bytes)|GROUP_META (1 byte)|group_name_length (8 bytes)|group_name

key|version|(uint64_t)(-1) (8 bytes)|CONSUMER_META (1 byte)|group_name_length (8 bytes)|group_name|consumer_name_length (8 bytes)|consumer_name

key|version|(uint64_t)(-1) (8 bytes)|PEL_ENTRY (1 byte)|group_name_length (8 bytes)|group_name|EID MS (8 bytes)|EID SEQ (8 bytes)

GROUP_META = (uint8_t)1
CONSUMER_META = (uint8_t)2
PEL_ENTRY = (uint8_t)3

Now we can list all group names just by scanning the prefix key|version|(uint64_t)(-1)|GROUP_META. Also, we can list all PEL entries if we want in a very efficient way.

Most importantly, the lexicographic order of key distinguishes several different subkey types, instead of a mixing order.

Make sense. I'll change the encoding later.

[PragmaTwice]

Also the group/consumer name length can be in 4 bytes instead of 8 bytes

[PragmaTwice]

@Yangsx-1 Hi, have you had a chance to implement the encoding change recently?

[Yangsx-1]

@Yangsx-1 Hi, have you had a chance to implement the encoding change recently?

I'm a little busy these days, but i'll try to finish this work and xpending command this month.

[PragmaTwice]

Hi @Yangsx-1 , could you update the document according to the changes when you have some time?