[QUESTION] Redis crashes with the following error on multiple nodes. What could be the reason?

[mukundv18]

Redis crashes with the following logs on multiple nodes (at different times). What could be the reason for the same. Please let me know if you need any details to be collected. Thanks!

5730:S 22 Sep 2022 22:02:28.920 # Error accepting a client connection: Broken pipe (conn: fd=11)
5730:S 22 Sep 2022 22:02:28.922 # Error accepting a client connection: Broken pipe (conn: fd=11)
5730:S 22 Sep 2022 22:02:28.923 # Error accepting a client connection: Broken pipe (conn: fd=11)
5730:S 22 Sep 2022 22:02:28.924 # Error accepting a client connection: Broken pipe (conn: fd=11)
5730:S 22 Sep 2022 22:02:28.926 # Error accepting a client connection: Broken pipe (conn: fd=11)
5730:S 22 Sep 2022 22:02:28.928 # Error accepting a client connection: Broken pipe (conn: fd=11)

Please keep in mind that this issue tracker should be used for reporting bugs or proposing improvements to the Redis server.

Generally, questions about using Redis should be directed to the community:

• the mailing list
• the redis tag at StackOverflow
• /r/redis subreddit
• github discussions

It is also possible that your question was already asked here, so please do a quick issues search before submitting. Lastly, if your question is about one of Redis' clients, you may to contact your client's developers for help.

That said, please feel free to replace all this with your question :)

[mukundv18]

After these errors the number of keys in Redis becomes '0' and it then recovers from either the persistence files or its corresponding master nodes

[oranagra]

@mukundv18 a few questions:

1. which version of redis?
2. if it crashes, can you please post the crash report. (you'll see REDIS BUG REPORT START)
3. seeing number of keys become 0, might mean you are being hacked, please make sure you have a password protecting redis, or a bind config. please post more of your log files, maybe there are some hints.

[mukundv18]

@oranagra

1. Redis version 7.0.2
2. Where do I see the crach report? Is it in redis.log?
3. I doubt it is being hacked. I believe the redis server restarts which is causing the keys to become '0' and eventually it recovers

[oranagra]

yes, you see the crash report in the redis log file.
if you don't see it, it probably means redis was killed by a SIGKILL, so i would suggest checking the syslog, maybe it was an OOM kill or some other kill -9.

[mukundv18]

@oranagra
I don't see any line with 'REDIS BUG REPORT START' in the redis.log file. So may be it didn't crash. What is causing the following logs? What could be the reason?

5730:S 22 Sep 2022 22:02:28.920 # Error accepting a client connection: Broken pipe (conn: fd=11)
5730:S 22 Sep 2022 22:02:28.922 # Error accepting a client connection: Broken pipe (conn: fd=11)
5730:S 22 Sep 2022 22:02:28.923 # Error accepting a client connection: Broken pipe (conn: fd=11)
5730:S 22 Sep 2022 22:02:28.924 # Error accepting a client connection: Broken pipe (conn: fd=11)
5730:S 22 Sep 2022 22:02:28.926 # Error accepting a client connection: Broken pipe (conn: fd=11)
5730:S 22 Sep 2022 22:02:28.928 # Error accepting a client connection: Broken pipe (conn: fd=11)

[mukundv18]

I see the following logs in the syslog from the time the keys became '0' until it recovered. It appears the data is loading in the slave nodes during that time. Any clues from these logs?

redis_exporter[32317]: time="2022-09-22T22:02:22Z" level=error msg="Couldn't connect to redis instance (rediss://<ip-address>)"
redis_exporter[32317]: time="2022-09-22T22:02:29Z" level=error msg="Redis CLUSTER INFO err: LOADING Redis is loading the dataset in memory"

[oranagra]

i'm out of ideas.
it's either that something is doing kill -9 to the redis process, or we have an exit() somewhere without a log print.
@yossigo maybe you have something to contribute here? (or about the EPIPE prints)

[mukundv18]

These are the logs from redis.log file with some prior log statements and post log statements. Do these give any clue?
Does Redis not accept client connection while loading RDB files?

5730:S 22 Sep 2022 22:01:00.228 * Discarding previously cached master state.
5730:S 22 Sep 2022 22:01:00.228 * MASTER <-> REPLICA sync: Flushing old data
5730:S 22 Sep 2022 22:02:25.385 * MASTER <-> REPLICA sync: Loading DB in memory
5730:S 22 Sep 2022 22:02:25.430 * Loading RDB produced by version 7.0.4
5730:S 22 Sep 2022 22:02:25.430 * RDB age 8988 seconds
5730:S 22 Sep 2022 22:02:25.430 * RDB memory usage when created 153566.32 Mb
5730:S 22 Sep 2022 22:02:28.920 # Error accepting a client connection: Broken pipe (conn: fd=11)
5730:S 22 Sep 2022 22:02:28.922 # Error accepting a client connection: Broken pipe (conn: fd=11)
5730:S 22 Sep 2022 22:02:28.923 # Error accepting a client connection: Broken pipe (conn: fd=11)
5730:S 22 Sep 2022 22:02:28.924 # Error accepting a client connection: Broken pipe (conn: fd=11)
5730:S 22 Sep 2022 22:02:28.926 # Error accepting a client connection: Broken pipe (conn: fd=11)
5730:S 22 Sep 2022 22:02:28.928 # Error accepting a client connection: Broken pipe (conn: fd=11)
5730:S 22 Sep 2022 22:22:25.684 * Done loading RDB, keys loaded: 41224136, keys expired: 0.
5730:S 22 Sep 2022 22:22:25.684 * MASTER <-> REPLICA sync: Finished with success

[oranagra]

redis does normally accept client connections during this times.

[muvelus]

Do you need to see any specific config parameters to confirm the same?

[mukundv18]

I observed that during the time of loading the RDB file, many nodes (master and slave), the key count goes to '0' and increases until the RDB file loads completely. This happens during the time between the following log statements.

27301:M 22 Sep 2022 15:21:39.783 * Loading RDB produced by version 7.0.4
27301:M 22 Sep 2022 17:11:12.878 * Done loading RDB, keys loaded: 41210325, keys expired: 0.

[oranagra]

that's normal, the replica flushes it's data before loading the new data.
And no, there's no special config needed for redis to accept connections while loading.