Downloading specific file results in "Terminating connection (out of memory)!" even when unencrypted

[matoro]

I've encountered a very strange issue where attempting to download this very small file Module-Implementation-0.09.tar.gz from any host, encrypted or unencrypted, over IPv4 or IPv6, results in sslsplit killing the connection with the error "Terminating connection (out of memory)!". There is more than sufficient memory on the host (64GB) and the file in question is only a few KB in size anyway. I haven't observed the issue with any other similar files (tarballs from CPAN), but it is perfectly reproducible with this specific file, even when sslsplit is compiled with no optimization.

This file is available at both of the following URLs:

• http://distfiles.gentoo.org/distfiles/41/Module-Implementation-0.09.tar.gz
• https://cpan.metacpan.org/authors/id/D/DR/DROLSKY/Module-Implementation-0.09.tar.gz

Note that the first URL is unencrypted. When attempting to download this file, here are the client-side and server-side logs:

$ curl -vL 'http://distfiles.gentoo.org/distfiles/41/Module-Implementation-0.09.tar.gz'
*   Trying [2a02:6ea0:c400::11]:80...
* Connected to distfiles.gentoo.org (2a02:6ea0:c400::11) port 80 (#0)
> GET /distfiles/41/Module-Implementation-0.09.tar.gz HTTP/1.1
> Host: distfiles.gentoo.org
> User-Agent: curl/7.88.1
> Accept: */*
> 
* Empty reply from server
* Closing connection 0
curl: (52) Empty reply from server

# /usr/bin/sslsplit -f /etc/sslsplit/sslsplit.conf -D
SSLsplit 0.5.5 (built 2023-03-15)
Copyright (c) 2009-2019, Daniel Roethlisberger <daniel@roe.ch>
https://www.roe.ch/SSLsplit
Build info: V:DIR N:83c4edf
Features: buildpkg pid-sandbox news nostrip compress-build-logs binpkg-logs sfperms sandbox usersandbox usersync fixlafiles installsources binpkg-dostrip xattr ebuild-locks preserve-libs buildpkg-live distlocks unknown-features-warn userfetch userpriv binpkg-docompress unmerge-logs parallel-install network-sandbox assume-digests qa-unresolved-soname-deps ipc-sandbox merge-sync compressdebug unmerge-orphans config-protect-if-modified splitdebug strict protect-owned multilib-strict
NAT engines: netfilter* tproxy
netfilter: IP_TRANSPARENT IP6T_SO_ORIGINAL_DST
Local process info support: no
compiled against OpenSSL 3.0.8 7 Feb 2023 (30000080)
rtlinked against OpenSSL 3.0.8 7 Feb 2023 (30000080)
OpenSSL has support for TLS extensions
TLS Server Name Indication (SNI) supported
OpenSSL is thread-safe with THREADID
OpenSSL has engine support
Using SSL_MODE_RELEASE_BUFFERS
SSL/TLS protocol availability: tls10 tls11 tls12 
SSL/TLS algorithm availability: !SHA0 RSA DSA ECDSA DH ECDH EC
OpenSSL option availability: SSL_OP_NO_COMPRESSION SSL_OP_NO_TICKET SSL_OP_ALLOW_UNSAFE_LEGACY_RENEGOTIATION SSL_OP_DONT_INSERT_EMPTY_FRAGMENTS SSL_OP_NO_SESSION_RESUMPTION_ON_RENEGOTIATION SSL_OP_TLS_ROLLBACK_BUG
compiled against libevent 2.1.12-stable
rtlinked against libevent 2.1.12-stable
compiled against libnet 1.2
rtlinked against libnet 1.2
compiled against libpcap n/a
rtlinked against libpcap 1.10.3 (with TPACKET_V3)
16 CPU cores detected
Generated 2048 bit RSA key for leaf certs.
SSL/TLS protocol: tls12
proxyspecs:
- [REDACTED]:8444 ssl|http netfilter
- [REDACTED]:8443 ssl|http netfilter
- [REDACTED]:8081 tcp|http netfilter
- [REDACTED]:8080 tcp|http netfilter
Loaded CA: '/CN=REDACTED'
SSL/TLS leaf certificates taken from:
- Generated on the fly
Privsep fastpath disabled
Created self-pipe [r=5,w=6]
Created chld-pipe [r=7,w=8]
Created socketpair 0 [p=9,c=10]
Created socketpair 1 [p=11,c=12]
Created socketpair 2 [p=13,c=14]
Created socketpair 3 [p=15,c=16]
Created socketpair 4 [p=17,c=18]
Created socketpair 5 [p=19,c=20]
Privsep parent pid 4353
Privsep child pid 4355
Using libevent backend 'epoll'
Event base supports: edge yes, O(1) yes, anyfd no
Received privsep req type 03 sz 9 on srvsock 9
Received privsep req type 03 sz 9 on srvsock 9
Received privsep req type 03 sz 9 on srvsock 9
Received privsep req type 03 sz 9 on srvsock 9
Received privsep req type 00 sz 1 on srvsock 9
Dropped privs to user daemon group - chroot -
Received privsep req type 00 sz 1 on srvsock 11
Received privsep req type 00 sz 1 on srvsock 13
Received privsep req type 00 sz 1 on srvsock 15
Received privsep req type 00 sz 1 on srvsock 17
Received privsep req type 00 sz 1 on srvsock 19
Inserted events:
  0xaaaae76f9a38 [fd  5] Read Persist Internal
  0xaaaae76f9c10 [fd  7] Read Persist Internal
  0xaaaae76f65d8 [fd  8] Read Persist
  0xaaaae76f6788 [fd  9] Read Persist
  0xaaaae76f7368 [fd  11] Read Persist
  0xaaaae76f8438 [fd  13] Read Persist
  0xaaaae76e9bd0 [sig 1] Signal Persist
  0xaaaae76d9860 [sig 2] Signal Persist
  0xaaaae76f62d0 [sig 3] Signal Persist
  0xaaaae76f6510 [sig 10] Signal Persist
  0xaaaae76f7180 [sig 13] Signal Persist
  0xaaaae76f6d20 [sig 15] Signal Persist
  0xaaaae76f6980 [fd  -1] Persist Timeout=1678935180.127959
Active events:
Initialized 32 connection handling threads
Started 32 connection handling threads
Starting main event loop.
Connecting to [2a02:6ea0:c400::11]:80
TCP connected to [2a02:6ea0:c400::11]:80
TCP connected from [REDACTED]:58662
Terminating connection (out of memory)!

When downloading the encrypted file:

$ curl -vL 'https://cpan.metacpan.org/authors/id/D/DR/DROLSKY/Module-Implementation-0.09.tar.gz'
*   Trying [2a04:4e42:45::729]:443...
* Connected to cpan.metacpan.org (2a04:4e42:45::729) port 443 (#0)
* ALPN: offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN: server did not agree on a protocol. Uses default.
* Server certificate:
*  subject: CN=*.metacpan.org
*  start date: Mar 15 02:53:07 2023 GMT
*  expire date: Mar 14 02:53:07 2024 GMT
*  subjectAltName: host "cpan.metacpan.org" matched cert's "*.metacpan.org"
*  issuer: CN=REDACTED
*  SSL certificate verify ok.
* using HTTP/1.x
> GET /authors/id/D/DR/DROLSKY/Module-Implementation-0.09.tar.gz HTTP/1.1
> Host: cpan.metacpan.org
> User-Agent: curl/7.88.1
> Accept: */*
> 
* TLSv1.2 (IN), TLS alert, close notify (256):
* Empty reply from server
* Closing connection 0
* TLSv1.2 (OUT), TLS alert, close notify (256):
curl: (52) Empty reply from server

# /usr/bin/sslsplit -f /etc/sslsplit/sslsplit.conf -D                                                                                                                                                                           
SSLsplit 0.5.5 (built 2023-03-15)                                                                                                                                                                                                                
Copyright (c) 2009-2019, Daniel Roethlisberger <daniel@roe.ch>                                                                                                                                                                                   
https://www.roe.ch/SSLsplit                                                                                                                                                                                                                      
Build info: V:DIR N:83c4edf                                                                                                                                                                                                                      
Features: buildpkg pid-sandbox news nostrip compress-build-logs binpkg-logs sfperms sandbox usersandbox usersync fixlafiles installsources binpkg-dostrip xattr ebuild-locks preserve-libs buildpkg-live distlocks unknown-features-warn userfetc
h userpriv binpkg-docompress unmerge-logs parallel-install network-sandbox assume-digests qa-unresolved-soname-deps ipc-sandbox merge-sync compressdebug unmerge-orphans config-protect-if-modified splitdebug strict protect-owned multilib-stri
ct                                                                                                                                                                                                                                               
NAT engines: netfilter* tproxy                                                                                                                                                                                                                   
netfilter: IP_TRANSPARENT IP6T_SO_ORIGINAL_DST                                                                                                                                                                                                   
Local process info support: no                                                                                                                                                                                                                   
compiled against OpenSSL 3.0.8 7 Feb 2023 (30000080)                                                                                                                                                                                             
rtlinked against OpenSSL 3.0.8 7 Feb 2023 (30000080)                                                                                                                                                                                             
OpenSSL has support for TLS extensions                                                                                                                                                                                                           
TLS Server Name Indication (SNI) supported                                                                                                                                                                                                       
OpenSSL is thread-safe with THREADID                                                                                                                                                                                                             
OpenSSL has engine support                                                                                                                                                                                                                       
Using SSL_MODE_RELEASE_BUFFERS
SSL/TLS protocol availability: tls10 tls11 tls12 
SSL/TLS algorithm availability: !SHA0 RSA DSA ECDSA DH ECDH EC
OpenSSL option availability: SSL_OP_NO_COMPRESSION SSL_OP_NO_TICKET SSL_OP_ALLOW_UNSAFE_LEGACY_RENEGOTIATION SSL_OP_DONT_INSERT_EMPTY_FRAGMENTS SSL_OP_NO_SESSION_RESUMPTION_ON_RENEGOTIATION SSL_OP_TLS_ROLLBACK_BUG
compiled against libevent 2.1.12-stable
rtlinked against libevent 2.1.12-stable
compiled against libnet 1.2
rtlinked against libnet 1.2
compiled against libpcap n/a
rtlinked against libpcap 1.10.3 (with TPACKET_V3)
16 CPU cores detected
Generated 2048 bit RSA key for leaf certs.
SSL/TLS protocol: tls12
proxyspecs:
- [REDACTED]:8444 ssl|http netfilter
- [REDACTED]:8443 ssl|http netfilter
- [REDACTED]:8081 tcp|http netfilter
- [REDACTED]:8080 tcp|http netfilter
Loaded CA: '/CN=REDACTED'
SSL/TLS leaf certificates taken from:
- Generated on the fly
Privsep fastpath disabled
Created self-pipe [r=5,w=6]
Created chld-pipe [r=7,w=8]
Created socketpair 0 [p=9,c=10]
Created socketpair 1 [p=11,c=12]
Created socketpair 2 [p=13,c=14]
Created socketpair 3 [p=15,c=16]
Created socketpair 4 [p=17,c=18]
Created socketpair 5 [p=19,c=20]
Privsep parent pid 6080
Privsep child pid 6082
Using libevent backend 'epoll'
Event base supports: edge yes, O(1) yes, anyfd no
Received privsep req type 03 sz 9 on srvsock 9
Received privsep req type 03 sz 9 on srvsock 9
Received privsep req type 03 sz 9 on srvsock 9
Received privsep req type 03 sz 9 on srvsock 9
Received privsep req type 00 sz 1 on srvsock 9
Dropped privs to user daemon group - chroot -
Received privsep req type 00 sz 1 on srvsock 11
Received privsep req type 00 sz 1 on srvsock 13
Received privsep req type 00 sz 1 on srvsock 15
Received privsep req type 00 sz 1 on srvsock 17
Received privsep req type 00 sz 1 on srvsock 19
Inserted events:
  0xaaaae0895a38 [fd  5] Read Persist Internal
  0xaaaae0895c10 [fd  7] Read Persist Internal
  0xaaaae08925d8 [fd  8] Read Persist
  0xaaaae0892788 [fd  9] Read Persist
  0xaaaae0893368 [fd  11] Read Persist
  0xaaaae0894438 [fd  13] Read Persist
  0xaaaae0885bd0 [sig 1] Signal Persist
  0xaaaae0875860 [sig 2] Signal Persist
  0xaaaae0889ad0 [sig 3] Signal Persist
  0xaaaae0892510 [sig 10] Signal Persist
  0xaaaae0893180 [sig 13] Signal Persist
  0xaaaae0892d20 [sig 15] Signal Persist
  0xaaaae0892980 [fd  -1] Persist Timeout=1678935243.207318
Active events:
Initialized 32 connection handling threads
Started 32 connection handling threads
Starting main event loop.
SNI peek: [cpan.metacpan.org] [complete]
Connecting to [2a04:4e42:45::729]:443
===> Original server certificate:
Subject DN: /CN=*.metacpan.org
Common Names: *.metacpan.org/*.metacpan.org
Fingerprint: 32:30:8C:C8:4F:DB:56:97:D7:31:AF:08:9F:4A:28:32:45:11:11:9B
Certificate cache: MISS
===> Forged server certificate:
Subject DN: /CN=*.metacpan.org
Common Names: *.metacpan.org/*.metacpan.org
Fingerprint: 30:2D:A1:D8:87:B3:60:1F:57:DE:E0:F6:A9:D6:E8:11:29:78:29:67
SSL connected to [2a04:4e42:45::729]:443 TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256
CLIENT_RANDOM 9C179C5014DDB988E0AAEC209C13D1DDCE5464E7DF24D5F429A5065B502ECE24 3FF822A755A800D2F3A480296E0F88F20E5A16B54AA9731280977DA5751A0E63FECAF31A98E1F7EC4AA2B2DBB5B64EC3
SSL session cache: MISS
Certificate cache: KEEP (SNI match or target mode)
SSL connected from [REDACTED]:46496 TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384
CLIENT_RANDOM 71071A6E6121DD9BAE5373DAF2D352AF34CD71100ACB1B914FB3B292944844D2 C1A8DB624405AF99A1CEE4B185EE493CDD5F5C2B8CC55BA0AF4E80D6AC96CCBC8EFBEC962AEF23F51194843897EAAA66
Terminating connection (out of memory)!
SSL_free() in state 00000001 = 0001 = SSLOK (SSL negotiation finished successfully) [connect socket]
SSL_free() in state 00000001 = 0001 = SSLOK (SSL negotiation finished successfully) [accept socket]

Backtrace:

# gdb -q --args /usr/bin/sslsplit -f /etc/sslsplit/sslsplit.conf -D
Reading symbols from /usr/bin/sslsplit...
(gdb) set follow-fork-mode child
(gdb) b pxy_conn_terminate_free
Breakpoint 1 at 0x260e4: file pxyconn.c, line 1724.
(gdb) r
Thread 2.4 "sslsplit" hit Breakpoint 1, pxy_conn_terminate_free (ctx=0xaaaaaab6bc90, is_requestor=1) at pxyconn.c:1724
1724    {
(gdb) bt full
#0  pxy_conn_terminate_free (ctx=0xaaaaaab6bc90, is_requestor=1) at pxyconn.c:1724
No locals.
#1  0x0000aaaaaaac683c in pxy_bev_readcb (bev=0xfffff0000c50, arg=0xaaaaaab6bc90) at pxyconn.c:1875
        ctx = 0xaaaaaab6bc90
        other = 0xaaaaaab6bca8
        inbuf = 0xfffff0000e70
        outbuf = 0xaaaaaab6c170
#2  0x0000fffff7965d08 in bufferevent_run_deferred_callbacks_locked () from /usr/lib64/libevent-2.1.so.7
No symbol table info available.
#3  0x0000fffff79746ac in event_process_active_single_queue () from /usr/lib64/libevent-2.1.so.7
No symbol table info available.
#4  0x0000fffff7974aa0 in event_process_active.lto_priv () from /usr/lib64/libevent-2.1.so.7
No symbol table info available.
#5  0x0000fffff799bc28 in event_base_loop.constprop () from /usr/lib64/libevent-2.1.so.7
No symbol table info available.
#6  0x0000aaaaaaac8560 in pxy_thrmgr_thr (arg=0xaaaaaab56640) at pxythrmgr.c:88
        ctx = 0xaaaaaab56640
        timer_delay = {tv_sec = 60, tv_usec = 0}
        ev = 0xfffff0000b70
#7  0x0000fffff777e200 in start_thread () from /lib64/libc.so.6
No symbol table info available.
#8  0x0000fffff77e421c in thread_start () from /lib64/libc.so.6
No symbol table info available.

sslsplit server:

Linux REDACTED 6.1.18-gentoo-dist #1 SMP PREEMPT_DYNAMIC Sun Mar 12 22:29:53 EDT 2023 aarch64 GNU/Linux

Config file:

$ egrep '^[^#]' /etc/sslsplit/sslsplit.conf
CACert REDACTED
CAKey REDACTED
DenyOCSP yes
Passthrough no
SSLCompression yes
ForceSSLProto tls12
User daemon
PidFile /var/run/sslsplit.pid
MirrorIf dummy0
VerifyPeer yes
ProxySpec http REDACTED 8080
ProxySpec http REDACTED 8081
ProxySpec https REDACTED 8443
ProxySpec https REDACTED 8444

[sonertari]

Thanks for the detailed report.

I see that you use version v0.5.5. Since that release we've made a couple of important changes on the develop branch. Particularly relevant here are the ones on watermarking and underlying buffer events. I haven't checked the size of that specific file you download, but if it is large enough, it may cause watermarking to kick in.

So, can you fetch the develop branch, and try it instead of v0.5.5, and report back please?

[matoro]

Thanks for the detailed report.

I see that you use version v0.5.5. Since that release we've made a couple of important changes on the develop branch. Particularly relevant here are the ones on watermarking and underlying buffer events. I haven't checked the size of that specific file you download, but if it is large enough, it may cause watermarking to kick in.

So, can you fetch the develop branch, and try it instead of v0.5.5, and report back please?

Thank you for looking at this. Which commit(s) specifically are you referring to? I was reviewing 0.5.5...develop to try and get an understanding of the changes but could not see any changes with a functional impact (besides the PR for mirroring to dummy interface, which I already backported and am using).

The file is only 19KB, which is why it's so surprising! Everything works fine with files many orders of magnitude larger.

[sonertari]

Ah, sorry, it's on the underlying-bevs-issue#303 branch: develop...underlying-bevs-issue#303

But, if the file is just19KB, then I don't think that's the issue, as you guessed too.

I should try to reproduce this issue myself too.

You can also try sslproxy with a split mode proxyspec, to see if the issue is specific to sslsplit only.