Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Separate per event and bulk-event queues #649

Merged
merged 2 commits into from
Jan 7, 2016
Merged

Separate per event and bulk-event queues #649

merged 2 commits into from
Jan 7, 2016

Commits on Jan 7, 2016

  1. Separate per event and bulk-event queues 

    Have separate queues with different buffer sizes
for single events and bulk events in publisher.

Default queues size for single events is 1000 elements and for bulk events is set to 0.

Default values dramatically reduce total number of buffered events and memory usage in case elasticsearch/logstash become unresponsive.

Introduce (hidden) configuration in shipper section:
  - queue_size
  - bulk_queue_size

When using async publisher number of buffered events is about given by:

Q_e = queue size (default 1000)
Q_b = bulk queue size (default 0)
B = bulk_max_size
W = number of load balancing workers (only if load balancing is enabled)
N = number of events in memory

    N = 2*(Q_e) + (2 + Q_b + W)*B

using default values:
   Q_e = 1000
   Q_b = 0
   B = 2048

  - topbeat/packetbeat with default values + single output:

      N = 2*1000 + 2*2048 = 6096 events

  - topbeat/packetbeat with default values + load balancing with 4 workers:

      N = 2*1000 + 6*2048 = 14288 events
    urso committed Jan 7, 2016
    Configuration menu
    Copy the full SHA
    acad1cd View commit details
    Browse the repository at this point in the history

  2. shipper queue size configuration

    urso committed Jan 7, 2016
    Configuration menu
    Copy the full SHA
    6865d1a View commit details
    Browse the repository at this point in the history