Disk Buffering Persistence Correctness Test
This test is designed to ensure data is persisted to disk properly when disk buffers are configured. This is an important test to reduce data loss and prevent disruption when downstream services fail.
- Configure all subjects to use disk-buffers output to a downstream service.
- Start the subject.
- Shutdown the downstream service.
- Send data to the subject.
- Restart the subject.
- Bring the downstream service back online.
- Assert that data was evetually received.
This test is not designed to verify at least once delivery, some data loss is acceptable. This test verifies that data is reasonably persisted to disk and made available between restarts.
- After many hours of trying, were unable to get Logstash to work at all for this test. It seems unlikely that their persistent queues feature does not work, and it's more likely that we do not understand something. The documentation failed to help us.
bin/test -t disk_buffer_persistence_correctness