DragonFly instance unresponsive: Error: Stalled heartbeat() fiber for 62453444040472 seconds because of big value serialization

[wernermorgenstern]

Describe the bug
A DragonFly instance wasn't responding to any queues, and all we could see was a lot of
Stalled heartbeat() fiber for 62453444040472 seconds because of big value serialization messages in the logs. The seconds changed, every second

To Reproduce
Steps to reproduce the behavior:

1. Run a DragonFLy instance for a period of time (not specific time)
2. For BullMQ Pro, unable to add new jobs to the queue, and no workers are able to access it. But no BullMQ Errors
3. Check the DragonFly Logs

Expected behavior
DragonFly instance should be able to run for a long time

Environment (please complete the following information):

• OS: AWS Linux
• Kernel: Linux dragonfly-bullmq-admin-response-0-primary-0 5.10.233-224.894.amzn2.aarch64 #1 SMP Mon Jan 27 16:53:26 UTC 2025 aarch64 aarch64 aarch64 GNU/Linux
• Containerized?: EKS Kubernetes, instance type: c7gn.xlarge
• Dragonfly Version: 1.27.2

Additional context
We run BullMQ Pro with this specific DragonFly Instance

[adiholden]

@wernermorgenstern Thanks for reporting this.
Do you run replication? or did you have snapshot running at this time?
Did you go back to earlier version? can you run "info all" and attach the output?

[wernermorgenstern]

@adiholden ,
yes, we do run replication.
And we are running snapshots as well (every 30 minutes), and snapshots go to S3.

Right now, after restarting the instance, the error is gone. I will keep an eye on it.

[adiholden]

@wernermorgenstern
The error that you saw means that dragonfly is in the middle of writing a value to sink for a long time.
In replication we have a logic to detect that if the write takes long time and therefor the master hangs we cancel the replication. I don't think we have such a thing for snapshoting there for I suspect its related to writing to S3 that hangs.
Having said that the server should be responsive to some none transactional commands f.e ping command. If this reproduces please share with up if you are able to get response for ping and also please share info all output at this stage which should be also responsive.

[wernermorgenstern]

Ok, will do. I will keep an eye on it.

[adiholden]

@wernermorgenstern I am closing this issue now as we can not take any action if we don't have additional info
If this happens again please reopen it