Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Memory leak in TS 8.0.6 #6614

Open
jvgutierrez opened this issue Apr 3, 2020 · 5 comments
Open

Memory leak in TS 8.0.6 #6614

jvgutierrez opened this issue Apr 3, 2020 · 5 comments
Labels
Leak

Comments

@jvgutierrez
Copy link
Member

jvgutierrez commented Apr 3, 2020 •
edited
Loading

We are seeing what it looks like a memory leak in TS 8.0.6:
image
In the screenshot you can see cp1081 being affected by it, beginning on April 2nd at 09:28.
From the logs I cannot see anything relevant at that time:

Apr 02 08:11:59 cp1081 traffic_ctl[42743]: NOTE: using command line path as RUNROOT
Apr 02 08:11:59 cp1081 traffic_ctl[42743]: NOTE: using command line path as RUNROOT
Apr 02 08:11:59 cp1081 traffic_manager[43724]: [Apr  2 08:11:59.769] traffic_manager NOTE: User has changed config file ssl_multicert.config
Apr 02 08:11:59 cp1081 systemd[1]: Reloaded trafficserver-tls.service.
Apr 02 08:33:59 cp1081 traffic_manager[43724]: [Apr  2 08:33:59.876] [ET_NET 64] ERROR: HTTP/2 connection error client_ip=REDACTED session_id=REDACTED stream_id=1 recv headers compression error
Apr 02 10:02:17 cp1081 traffic_manager[43724]: [Apr  2 10:02:17.702] [ET_NET 8] ERROR: HTTP/2 connection error client_ip=REDACTED session_id=REDACTED stream_id=0 priority 0 stream_id
Apr 02 13:50:29 cp1081 traffic_manager[43724]: [Apr  2 13:50:29.757] [ET_NET 63] ERROR: HTTP/2 connection error client_ip=REDACTED session_id=REDACTED stream_id=1 reset frame bad id stream not found

Configuration for cp1081:
vgutierrez@cp1081:~$ sudo -i cat /srv/trafficserver/tls/etc/remap.config
map wss://etherpad.wikimedia.org ws://10.64.16.24:3120
map / http://10.64.16.24:3120

vgutierrez@cp1081:~$ sudo -i cat /srv/trafficserver/tls/etc/ssl_multicert.config
dest_ip=* ssl_cert_name=acmecerts/unified/live/rsa-2048.chained.crt,acmecerts/unified/live/ec-prime256v1.chained.crt ssl_key_name=acmecerts/unified/live/rsa-2048.key,acmecerts/unified/live/ec-prime256v1.key ssl_ocsp_name=acmecerts/unified/live/rsa-2048.ocsp,acmecerts/unified/live/ec-prime256v1.ocsp
ssl_cert_name=acmecerts/wikiworkshop/live/rsa-2048.chained.crt,acmecerts/wikiworkshop/live/ec-prime256v1.chained.crt ssl_key_name=acmecerts/wikiworkshop/live/rsa-2048.key,acmecerts/wikiworkshop/live/ec-prime256v1.key ssl_ocsp_name=acmecerts/wikiworkshop/live/rsa-2048.ocsp,acmecerts/wikiworkshop/live/ec-prime256v1.ocsp

vgutierrez@cp1081:~$ sudo -i traffic_ctl --run-root=/srv/trafficserver/tls config diff
NOTE: using command line path as RUNROOT
proxy.config.http.server_ports has changed
	Current Value   : 443:ssl 443:ipv6:ssl
	Default Value   : 8080 8080:ipv6
proxy.config.admin.user_id has changed
	Current Value   : trafficserver
	Default Value   : root
proxy.config.http.insert_request_via_str has changed
	Current Value   : 0
	Default Value   : 1
proxy.config.http.insert_client_ip has changed
	Current Value   : 0
	Default Value   : 1
proxy.config.http.response_server_enabled has changed
	Current Value   : 2
	Default Value   : 1
proxy.config.url_remap.pristine_host_hdr has changed
	Current Value   : 1
	Default Value   : 0
proxy.config.disable_configuration_modification has changed
	Current Value   : 1
	Default Value   : 0
proxy.config.net.connections_throttle has changed
	Current Value   : 0
	Default Value   : 30000
proxy.config.net.sock_option_flag_in has changed
	Current Value   : 13
	Default Value   : 5
proxy.config.net.default_inactivity_timeout has changed
	Current Value   : 300
	Default Value   : 86400
proxy.config.http2.max_priority_frames_per_minute has changed
	Current Value   : 0
	Default Value   : 120
proxy.config.http2.accept_no_activity_timeout has changed
	Current Value   : 180
	Default Value   : 120
proxy.config.http2.no_activity_timeout_in has changed
	Current Value   : 180
	Default Value   : 120
proxy.config.http2.stream_error_rate_threshold has changed
	Current Value   : 1.000000
	Default Value   : 0.100000
proxy.config.http.connect_attempts_timeout has changed
	Current Value   : 3
	Default Value   : 30
proxy.config.http.post_connect_attempts_timeout has changed
	Current Value   : 3
	Default Value   : 1800
proxy.config.http.transaction_no_activity_timeout_in has changed
	Current Value   : 200
	Default Value   : 30
proxy.config.http.transaction_no_activity_timeout_out has changed
	Current Value   : 180
	Default Value   : 30
proxy.config.http.accept_no_activity_timeout has changed
	Current Value   : 180
	Default Value   : 120
proxy.config.http.send_100_continue_response has changed
	Current Value   : 1
	Default Value   : 0
proxy.config.http.max_post_size has changed
	Current Value   : 104857600
	Default Value   : 0
proxy.config.http.server_session_sharing.match has changed
	Current Value   : ip
	Default Value   : both
proxy.config.http.auth_server_session_private has changed
	Current Value   : 0
	Default Value   : 1
proxy.config.http.cache.http has changed
	Current Value   : 0
	Default Value   : 1
proxy.config.http.cache.max_open_read_retries has changed
	Current Value   : 50
	Default Value   : -1
proxy.config.http.cache.max_open_write_retries has changed
	Current Value   : 150
	Default Value   : 1
proxy.config.diags.output.status has changed
	Current Value   : O
	Default Value   : L
proxy.config.diags.output.note has changed
	Current Value   : O
	Default Value   : L
proxy.config.diags.output.warning has changed
	Current Value   : O
	Default Value   : L
proxy.config.diags.output.error has changed
	Current Value   : E
	Default Value   : L
proxy.config.diags.output.fatal has changed
	Current Value   : E
	Default Value   : L
proxy.config.diags.output.alert has changed
	Current Value   : E
	Default Value   : L
proxy.config.diags.output.emergency has changed
	Current Value   : E
	Default Value   : L
proxy.config.log.max_space_mb_headroom has changed
	Current Value   : 0
	Default Value   : 1000
proxy.config.log.rolling_enabled has changed
	Current Value   : 0
	Default Value   : 1
proxy.config.log.max_secs_per_buffer has changed
	Current Value   : 0
	Default Value   : 5
proxy.config.log.file_stat_frequency has changed
	Current Value   : 1
	Default Value   : 32
proxy.config.log.log_buffer_size has changed
	Current Value   : 147456
	Default Value   : 9216
proxy.config.log.max_line_size has changed
	Current Value   : 147456
	Default Value   : 9216
proxy.config.ssl.TLSv1 has changed
	Current Value   : 0
	Default Value   : 1
proxy.config.ssl.TLSv1_1 has changed
	Current Value   : 0
	Default Value   : 1
proxy.config.ssl.TLSv1_3 has changed
	Current Value   : 0
	Default Value   : 1
proxy.config.ssl.server.cipher_suite has changed
	Current Value   : -ALL:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES128-SHA:DHE-RSA-AES128-SHA
	Default Value   : ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-DSS-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA256:DHE-RSA-AES128-SHA256:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA:DHE-DSS-AES256-SHA:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA
proxy.config.ssl.server.groups_list has changed
	Current Value   : X25519:prime256v1
	Default Value   : NULL
proxy.config.ssl.server.cert.path has changed
	Current Value   : /srv/trafficserver/tls/etc/tls
	Default Value   : /etc/trafficserver
proxy.config.ssl.server.private_key.path has changed
	Current Value   : /srv/trafficserver/tls/etc/tls
	Default Value   : /etc/trafficserver
proxy.config.ssl.max_record_size has changed
	Current Value   : 16383
	Default Value   : 0
proxy.config.ssl.server.dhparams_file has changed
	Current Value   : /etc/ssl/dhparam.pem
	Default Value   : NULL
proxy.config.ssl.ocsp.enabled has changed
	Current Value   : 1
	Default Value   : 0
proxy.config.ssl.ocsp.response.path has changed
	Current Value   : /srv/trafficserver/tls/etc/tls
	Default Value   : /etc/trafficserver
proxy.config.ssl.session_cache.timeout has changed
	Current Value   : 7200
	Default Value   : 0
proxy.config.ssl.session_cache.size has changed
	Current Value   : 512000
	Default Value   : 102400
proxy.config.ssl.session_cache.num_buckets has changed
	Current Value   : 32768
	Default Value   : 256
proxy.config.ssl.cert.load_elevated has changed
	Current Value   : 1
	Default Value   : 0
proxy.config.ssl.server.session_ticket.enable has changed
	Current Value   : 0
	Default Value   : 1
proxy.config.ssl.handshake_timeout_in has changed
	Current Value   : 60
	Default Value   : 0
proxy.config.body_factory.template_sets_dir has changed
	Current Value   : /srv/trafficserver/tls/etc/error_template
	Default Value   : body_factory```
@randall randall added the Leak label Apr 3, 2020
@jvgutierrez
Copy link
Member Author

Things have improved a lot by backporting #5697, I'm not sure it's the only one, but that's definitely helping:)

@jvgutierrez
Copy link
Member Author

The issue is still there sadly :
image

@jvgutierrez
Copy link
Member Author

perf top points to an issue related to HTTP/2:

10.12%  traffic_server  [.] Http2Stream::signal_write_event
10.01%  traffic_server  [.] Http2Stream::update_write_request
 9.98%  traffic_server  [.] Http2ConnectionState::send_a_data_frame

@github-actions
Copy link

This issue has been automatically marked as stale because it has not had recent activity. Marking it stale to flag it for further consideration by the community.

@github-actions github-actions bot added the Stale label Jun 18, 2021
@bryancall
Copy link
Contributor

@jvgutierrez are you still seeing memory leaks? What version of ATS are you running?

@github-actions github-actions bot removed the Stale label Jun 6, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Leak
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@bryancall @randall @jvgutierrez