Empty Redis Stream Message Elasticache Redis 5.0.4

[StasKolodyuk]

Hi,

I've stumbled upon a situation when I have empty messages inside redis stream.
As far as I understand it's not possible to publish an empty message to a stream with XADD command.

Example:

XREADGROUP GROUP group a NOACK STREAMS advices 0
*1
*2
$7
advices
*14
*2
$15
1580490601492-0
*-1
*2
$15
1580492237190-0
*-1
*2
$15
1580496259890-0
*-1

Are you aware of cases that can result in such situation?
Thanks!

[itamarhaber]

Hello @StasKolodyuk

This indeed should not be happening.

[StasKolodyuk]

@itamarhaber is there smth I can provide you so you can start investigating? I have elasticache redis backup with those empty messages. Will it help?

[itamarhaber]

I'm not quite sure how to investigate this yet so no need for backups just yet.

Is this reproducible somehow?

[madolson]

@StasKolodyuk Hey, any chance you still have the context about this?

[StasKolodyuk]

@madolson hey, what context do you need? This is still happening in our system in prod though rather rarely (1-2 times a week).

[asalabaev]

I actually like this feature. A nil result means that the event was added but then expired and I missed that. An extra piece of information that I could use to at least understand why it happens and how many events I miss.

[madolson]

@asalabaev What do you mean by it was added and then expired?

@StasKolodyuk Perhaps some way to reproduce it so we can get to the bottom of the behavior :)

[asalabaev]

@madolson sorry wrong word,
the case can be like the following:

• XADD
• XREADGROUP - but never acknowledge, e.g. processing failed and the consumer would like to reprocess it later. the item is in the pending list
• XADD with MAXLEN (or probably XTRIM)
• the stream was capped and the first event is gone
• the event ID is still in the pending list! The consumer will get the nil body
  Here I know I missed that and the stream / consumers need reconfiguration. A useful information is not lost.

the original command now contains NOACK but it reads from the pending list hence it could happen in the original scenario as well