nginx, php process crashes due to high memory usage, tls_ua_fingerprint.php

[pfsiedl]

Important notices
Before you add a new report, we ask you kindly to acknowledge the following:

• [ x ] I have read the contributing guide lines at https://github.com/opnsense/plugins/blob/master/CONTRIBUTING.md
• [ x ] I have searched the existing issues, open and closed, and I'm convinced that mine is new.
• [ x ] The title contains the plugin to which this issue belongs

Describe the bug
PHP process crashes due to excessive memory usage when writing the file /var/log/nginx/handshakes.json by the script tls_ua_fingerprint.php.

To Reproduce
Not a clear solution to reproduce the behavior, just some facts.
/var/log/nginx/handshakes.json got a filesize of 234MB.
According to the traffic statistics, there have been over 1,4 Mio. requests in 20 days uptime, of which 320.000 requests have been accepted.

Expected behavior
No crash due to an more memory efficient way to write the file.

Relevant log files
[21-Feb-2024 22:20:02 Europe/Vienna] PHP Fatal error: Allowed memory size of 1073741824 bytes exhausted (tried to allocate 216010752 bytes) in /usr/local/opnsense/scripts/nginx/tls_ua_fingerprint.php on line 135

Additional context
We already have two independent firewalls with crashes of this type.
If you need any further logs or informations, I try my best to provide you with the additional informations.

Environment

FreeBSD 13.2-RELEASE-p7 stable/23.7-n254871-d5ec322cffc SMP amd64
OPNsense 23.10.2 212ca5a53
Plugins os-OPNBEcore-1.3 os-acme-client-3.20 os-git-backup-1.0_3 os-nginx-1.32.2 os-theme-cicada-1.34_4
Time Tue, 12 Mar 2024 17:14:38 +0100
OpenSSL 1.1.1w
Python 3.9.18
PHP 8.2.14

[kulikov-a]

Hi
how old is your nginx setup? fingerprints db looks quiet oversized to me

[pfsiedl]

Hello,
the creation time of handshakes.json is October 2021, so quite old.

Should we prune this file via cronjob every now and then?

[kulikov-a]

@pfsiedl There were changes in the tls_ua_fingerprint.php that were supposed to reduce the number of unique fingerprints received. but that was not that long ago)
I think you can just delete the handshakes.json file, it will start being created again automatically

[pfsiedl]

@kulikov-a I've deleted the handshakes.json file and let it rebuild. Seems to work for the time being.

Is there any description behind the mechanism of handshakes.json / tls_handshakes and why the plugin stores them separately in a json file? Is this only for visuals and ban-features via GUI?

[kulikov-a]

@pfsiedl the general idea is described in the docs (https://docs.opnsense.org/manual/how-tos/nginx_tls_fingerprints.html)
so tls_handshake.log is a log created by the nginx and containing the agent name, ciphers and curves. Once every 10 minutes, the data in this log is merged with the data in the permanent fingerprint db (handshakes.json).
You can see handshakes.json content at the SERVICES: NGINX: TLS FINGERPRINTS and use it to save the agent’s fingerprint and try to control possible traffic interception (if the fingerprint in request differs from the reference one). X-TLS-Client-Intercepted header is used to pass this info to the upstream

[pfsiedl]

@kulikov-a we resolved the issue with cleanup of handshakes.json.

[pfsiedl]

We again running into the issue, that handshakes.json file is quite large and therefore the php process runs into an oom exception. Current size if handshakes.json is 259MB.

Can't we have this file deleted once a week using a cronjob until you implement a longer-term fix?
Can I provide you with more information?

[kulikov-a]

hi! I'll try to take a look at the end of the week. a little busy right now, sorry

[kulikov-a]

@pfsiedl
Hi!
Sorry, I don't know the devs' plans for merging the previous pr (#3678). Therefore, I don’t understand whether it makes sense to try to add something there (for example, the ability to disable handshake logging for the server (there is a feeling that few people use the ability to use fingerprints for mitm detect. and adding the ability to disable it would be appropriate)).
In the meantime, I think you could delete this file yourself or add a configuration file (like http://kb.unixservertech.com/other/networking/opnsense/cron-jobs) that allows you to add a cron job through the GUI (tell me if you need help with this file).
However, i think that the handshakes db file size seems unusually large to me (on my production setup with a dozen sites, the file size does not exceed several megabytes. and I haven't encountered any complaints about this issue before). It would be interesting to look at the contents of your file - perhaps this will help us understand the root of the problem.

[pfsiedl]

@kulikov-a
I think I've found the culprit. We are using a groupware that alters the User-Agent header every request from their application to the server, and thats proxied via nginx.

The creation of the cronjob worked flawless and is a good workaround.

We would really appreciate an option to disable the handshake logging per server and hope your PR gets accepted asap to continue working on this plugin. Thank you for your effort.

[kulikov-a]

@pfsiedl
Glad to hear that you managed to resolve the issue and imho this is definitely a reason to add a 'Log TLS handshake' checkbox. I'll try to add this to the request)