Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[packetbeat] memory usage is growing continuously when capturing redis #12657

Closed
romber2001 opened this issue Jun 25, 2019 · 15 comments · Fixed by #12741
Closed

[packetbeat] memory usage is growing continuously when capturing redis #12657

romber2001 opened this issue Jun 25, 2019 · 15 comments · Fixed by #12741
Assignees
Labels

Comments

@romber2001
Copy link

@romber2001 romber2001 commented Jun 25, 2019

server: virtual machine
cpu: 4 logic cpu/8 logic cpu
ram: 32GB

os: centos 7.2 x86_64
packetbeat: 7.1.1(i used binary release not rpm package)
redis: 4.0.8
kafka: 2.9.2-0.8.2.2

we have a redis cluster which has 8 dedicated redis servers, and running 2 redis instances on each server, totally 16 redis instances.
and i deployed 8 packetbeat on those servers to capture redis traffic and output to kakfa cluster.
what is strange is that only one of the packetbeat process uses memory more and more,
but it output content to kafka without problems.
on the other side, memory usage of other 7 packetbeat processes did not grow continuously,
and they shared the same config file and binary release.

i uploaded config file and also the pprof png file,
could anyone help me on this problem?
thanks in advance.

config file:

path.home: /data/app/packetbeat
max_procs: 1
logging:
  level: info
  to_files: true
  files:
    name: packetbeat.log
    keepfiles: 5
    rotateeverybytes: 20971520
    permissions: 0644
    interval: 168h

packetbeat.ignore_outgoing: true
packetbeat.interfaces.device: any
packetbeat.interfaces.type: af_packet
packetbeat.interfaces.buffer_size_mb: 128

packetbeat.flows:
  enabled: false

processors:
- add_locale:
    format: offset
- drop_fields:
    fields: ["host", "ecs", "agent"]

# packetbeat.interfaces.bpf_filter: "port 3306 or port 7001 or port 7002"
packetbeat.protocols:
  # mysql
  - type: mysql
    enabled: false
    ports: [3306, 3307, 3308, 3309, 3310]
    send_request: false
    send_response: false
    max_rows: 100
    max_row_length: 10485760
    processors:
    - include_fields:
        fields: ["name", "tags", "client", "server", "type", "method", "event", "query", "mysql"]
    - add_fields:
        target: ''
        fields:
          cluster_name: ${cluster_name_mysql:mysql}
    - drop_fields:
        fields: ["event.dataset", "event.kind", "event.category"]
  # redis
  - type: redis
    enabled: true
    ports: [6379, 7001, 7002]
    send_request: false
    send_response: false
    processors:
    - include_fields:
        fields: ["name", "tags", "client", "server", "type", "method", "resource", "event", "redis"]
    - add_fields:
        target: ''
        fields:
          cluster_name: ${cluster_name_redis:redis}
    - drop_fields:
        fields: ["event.dataset", "event.kind", "event.category"]

queue.mem:
  events: 4096
  flush.min_events: 256
  flush.timeout: 1s

# output configuration
# file
output:
  file:
    enabled: false
    path: "/data/app/packetbeat/data"
    filename: "packetbeat_file.out"
    number_of_file: 5
    rotate_every_kb: 20480
    #codec.json:
    #  pretty: true
    #codec.format:
    #  string: '%{[@timestamp]} %{[message]}'
# kafka
  kafka:
    enabled: true
    hosts: ["kafka1:9092", "kafka2:9092", "kafka3:9092", "kafka4:9092", "kafka5:9092"]
    topic: "packetbeat_01"
    partition.round_robin:
      reachable_only: true
    metadata:
      refresh_frequency: 5m
      full: false
    #codec.json:
    #  pretty: true
    #codec.format:
    #  string: '%{[@timestamp]} %{[message]}'

pprof:
image

@romber2001

This comment has been minimized.

Copy link
Author

@romber2001 romber2001 commented Jun 25, 2019

i also have a test to output to local file, but problem still exists,
and i found that the cpu usage of the bad node is low(about ),
at the same time other 7 good nodes use 100% cpu(i set the max_procs to 1),
so i don't know why the bad node could not use cpu as much as other servers,
maybe caused by virtual machine?

bad node:
image

goode node:
image

@adriansr adriansr self-assigned this Jun 27, 2019
@adriansr

This comment has been minimized.

Copy link
Member

@adriansr adriansr commented Jun 27, 2019

Hi @romber2001

By looking at the code and the provided pprof output, I think that the logical explanation is that Packetbeat is capturing a lot of Redis requests but not the responses. This will currently cause it to leak memory as it will accumulate the requests indefinitely waiting for a response to be received (that's something that should be fixed). It might be caused by a lot of packet loss / retransmissions in the network.

Can we confirm by having either:
a) the debug output of running with -d redis
b) a pcap of the traffic seen by Packetbeat with --dump redis.pcap.

@adriansr

This comment has been minimized.

Copy link
Member

@adriansr adriansr commented Jun 28, 2019

Hi @romber2001, thanks for the files, they confirm my suspicions, I will work on a fix.

@elastic elastic deleted a comment from romber2001 Jun 28, 2019
@romber2001

This comment has been minimized.

Copy link
Author

@romber2001 romber2001 commented Jun 29, 2019

@adriansr , thanks for your help, is this caused by there are only slaves on the server?

@adriansr

This comment has been minimized.

Copy link
Member

@adriansr adriansr commented Jul 1, 2019

@romber2001 as I understand from the pcap, what causes the memory usage, in this case, is the replication streams. These consist of the master sending requests to the slave and there is never a response. The redis processor in Packetbeat is not prepared for this.

@romber2001

This comment has been minimized.

Copy link
Author

@romber2001 romber2001 commented Jul 1, 2019

@adriansr , is there any workaround to set a hard limit of memory usage of packetbeat?
should i have to wait for releasing the new version?

adriansr added a commit to adriansr/beats that referenced this issue Jul 1, 2019
This patch limits the amount of memory that can be used by outstanding
requests by adding two new configuration options to the Redis protocol:

- max_queue_size: Limits the total size of the queued requests (in bytes).
- max_queue_length: Limits the number of requests queued.

These limits apply to individual connections. The defaults are to queue
up to 1MB or 20.000 requests. This is enough to limit the currently
unbounded memory used by replication streams while at the same time
allow for pipelining requests.

Closes elastic#12657
adriansr added a commit to adriansr/beats that referenced this issue Jul 1, 2019
This patch limits the amount of memory that can be used by outstanding
requests by adding two new configuration options to the Redis protocol:

- max_queue_size: Limits the total size of the queued requests (in bytes).
- max_queue_length: Limits the number of requests queued.

These limits apply to individual connections. The defaults are to queue
up to 1MB or 20.000 requests. This is enough to limit the currently
unbounded memory used by replication streams while at the same time
allow for pipelining requests.

Closes elastic#12657
adriansr added a commit to adriansr/beats that referenced this issue Jul 1, 2019
This patch limits the amount of memory that can be used by outstanding
requests by adding two new configuration options to the Redis protocol:

- max_queue_size: Limits the total size of the queued requests (in bytes).
- max_queue_length: Limits the number of requests queued.

These limits apply to individual connections. The defaults are to queue
up to 1MB or 20.000 requests. This is enough to limit the currently
unbounded memory used by replication streams while at the same time
allow for pipelining requests.

Closes elastic#12657
adriansr added a commit to adriansr/beats that referenced this issue Jul 1, 2019
This patch limits the amount of memory that can be used by outstanding
requests by adding two new configuration options to the Redis protocol:

- max_queue_size: Limits the total size of the queued requests (in bytes).
- max_queue_length: Limits the number of requests queued.

These limits apply to individual connections. The defaults are to queue
up to 1MB or 20.000 requests. This is enough to limit the currently
unbounded memory used by replication streams while at the same time
allow for pipelining requests.

Closes elastic#12657
@adriansr

This comment has been minimized.

Copy link
Member

@adriansr adriansr commented Jul 1, 2019

@romber2001 I've submitted a patch. I don't have a workaround other than suggesting you set a bpf_filter that ignores traffic between servers (replication)

Something like this (untested!):

packetbeat.interfaces.bpf_filter: '(port 6379 or port 7001 or port 7002) and (not src host XXXX.146 or not dst host XXXXX.45) and (not src host XXXXX.18 or not dst host XXXXX.146)'

adriansr added a commit that referenced this issue Jul 2, 2019
This patch limits the amount of memory that can be used by outstanding
requests by adding two new configuration options to the Redis protocol:

- max_queue_size: Limits the total size of the queued requests (in bytes).
- max_queue_length: Limits the number of requests queued.

These limits apply to individual connections. The defaults are to queue
up to 1MB or 20.000 requests. This is enough to limit the currently
unbounded memory used by replication streams while at the same time
allow for pipelining requests.

Closes #12657
adriansr added a commit to adriansr/beats that referenced this issue Jul 2, 2019
This patch limits the amount of memory that can be used by outstanding
requests by adding two new configuration options to the Redis protocol:

- max_queue_size: Limits the total size of the queued requests (in bytes).
- max_queue_length: Limits the number of requests queued.

These limits apply to individual connections. The defaults are to queue
up to 1MB or 20.000 requests. This is enough to limit the currently
unbounded memory used by replication streams while at the same time
allow for pipelining requests.

Closes elastic#12657

(cherry picked from commit 1275ee8)
adriansr added a commit to adriansr/beats that referenced this issue Jul 2, 2019
This patch limits the amount of memory that can be used by outstanding
requests by adding two new configuration options to the Redis protocol:

- max_queue_size: Limits the total size of the queued requests (in bytes).
- max_queue_length: Limits the number of requests queued.

These limits apply to individual connections. The defaults are to queue
up to 1MB or 20.000 requests. This is enough to limit the currently
unbounded memory used by replication streams while at the same time
allow for pipelining requests.

Closes elastic#12657

(cherry picked from commit 1275ee8)
adriansr added a commit to adriansr/beats that referenced this issue Jul 2, 2019
This patch limits the amount of memory that can be used by outstanding
requests by adding two new configuration options to the Redis protocol:

- max_queue_size: Limits the total size of the queued requests (in bytes).
- max_queue_length: Limits the number of requests queued.

These limits apply to individual connections. The defaults are to queue
up to 1MB or 20.000 requests. This is enough to limit the currently
unbounded memory used by replication streams while at the same time
allow for pipelining requests.

Closes elastic#12657

(cherry picked from commit 1275ee8)
adriansr added a commit to adriansr/beats that referenced this issue Jul 2, 2019
This patch limits the amount of memory that can be used by outstanding
requests by adding two new configuration options to the Redis protocol:

- max_queue_size: Limits the total size of the queued requests (in bytes).
- max_queue_length: Limits the number of requests queued.

These limits apply to individual connections. The defaults are to queue
up to 1MB or 20.000 requests. This is enough to limit the currently
unbounded memory used by replication streams while at the same time
allow for pipelining requests.

Closes elastic#12657

(cherry picked from commit 1275ee8)
adriansr added a commit to adriansr/beats that referenced this issue Jul 2, 2019
This patch limits the amount of memory that can be used by outstanding
requests by adding two new configuration options to the Redis protocol:

- max_queue_size: Limits the total size of the queued requests (in bytes).
- max_queue_length: Limits the number of requests queued.

These limits apply to individual connections. The defaults are to queue
up to 1MB or 20.000 requests. This is enough to limit the currently
unbounded memory used by replication streams while at the same time
allow for pipelining requests.

Closes elastic#12657

(cherry picked from commit 1275ee8)
adriansr added a commit to adriansr/beats that referenced this issue Jul 2, 2019
This patch limits the amount of memory that can be used by outstanding
requests by adding two new configuration options to the Redis protocol:

- max_queue_size: Limits the total size of the queued requests (in bytes).
- max_queue_length: Limits the number of requests queued.

These limits apply to individual connections. The defaults are to queue
up to 1MB or 20.000 requests. This is enough to limit the currently
unbounded memory used by replication streams while at the same time
allow for pipelining requests.

Closes elastic#12657

(cherry picked from commit 1275ee8)
@romber2001

This comment has been minimized.

Copy link
Author

@romber2001 romber2001 commented Jul 3, 2019

@adriansr , thanks for your effort!!!
all i need to do is to use the new 7.2.0 version, right?

@adriansr

This comment has been minimized.

Copy link
Member

@adriansr adriansr commented Jul 3, 2019

@romber2001 nope, this was too late for 7.2.0. Expect it in 7.2.1 and 7.3.0

adriansr added a commit to adriansr/beats that referenced this issue Jul 3, 2019
This patch limits the amount of memory that can be used by outstanding
requests by adding two new configuration options to the Redis protocol:

- max_queue_size: Limits the total size of the queued requests (in bytes).
- max_queue_length: Limits the number of requests queued.

These limits apply to individual connections. The defaults are to queue
up to 1MB or 20.000 requests. This is enough to limit the currently
unbounded memory used by replication streams while at the same time
allow for pipelining requests.

Closes elastic#12657

(cherry picked from commit 1275ee8)
@romber2001

This comment has been minimized.

Copy link
Author

@romber2001 romber2001 commented Jul 3, 2019

@adriansr got it, i'll patch it to the v7.2.0 and have a test.
and what does per-session mean for the two new parameter?
packetbeat process or redis instance or redis client connection?

@adriansr

This comment has been minimized.

Copy link
Member

@adriansr adriansr commented Jul 3, 2019

@romber2001 it's a limit per redis connection.

@romber2001

This comment has been minimized.

Copy link
Author

@romber2001 romber2001 commented Jul 3, 2019

@adriansr so if there are 1000 clients connected to redis server, queue_max_bytes is 1MB, and packetbeat will use up to 1GB RAM?

@adriansr

This comment has been minimized.

Copy link
Member

@adriansr adriansr commented Jul 3, 2019

It's an upper limit @romber2001. This queue will only fill in the case of replication connections. For regular clients the queue will usually be empty.

@romber2001

This comment has been minimized.

Copy link
Author

@romber2001 romber2001 commented Jul 4, 2019

@adriansr thanks for your explain, i finally got the point, thanks again.

adriansr added a commit that referenced this issue Jul 4, 2019
This patch limits the amount of memory that can be used by outstanding
requests by adding two new configuration options to the Redis protocol:

- max_queue_size: Limits the total size of the queued requests (in bytes).
- max_queue_length: Limits the number of requests queued.

These limits apply to individual connections. The defaults are to queue
up to 1MB or 20.000 requests. This is enough to limit the currently
unbounded memory used by replication streams while at the same time
allow for pipelining requests.

Closes #12657

(cherry picked from commit 1275ee8)
adriansr added a commit that referenced this issue Jul 4, 2019
…eplication (#12752)

This patch limits the amount of memory that can be used by outstanding
requests by adding two new configuration options to the Redis protocol:

- max_queue_size: Limits the total size of the queued requests (in bytes).
- max_queue_length: Limits the number of requests queued.

These limits apply to individual connections. The defaults are to queue
up to 1MB or 20.000 requests. This is enough to limit the currently
unbounded memory used by replication streams while at the same time
allow for pipelining requests.

Closes #12657

(cherry picked from commit 1275ee8)
@romber2001

This comment has been minimized.

Copy link
Author

@romber2001 romber2001 commented Jul 5, 2019

@adriansr hi again, just want to give a feedback to you and other guys who may keep an eye on this issue that i've run nearly 24 hours with new patch and memory usage does not grow anymore, problem solved.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.