liboverture.so memory leak

[bangbang93]

Environment

• Android version: 8.0.0 (47.1.A.8.49)
• Device: Sony Xperia XZ P (G8142)
• Shadowsocks version: 4.4.6
• Last version that did not exhibit the issue (if applicable):

Configuration

Put an x inside the [ ] that applies.

• IPv4 server address
• IPv6 server address
• Client IPv4 availability
• Client IPv6 availability
• Encrypt method: chacha20-ietf-poly1305
• Route
  • All
  • Bypass LAN
  • Bypass China
  • Bypass LAN & China
  • GFW List
  • China List
  • Custom rules
• IPv6 route
• Apps VPN mode
  • Bypass mode
• Remote DNS: 8.8.8.8
• DNS Forwarding
• Plugin configuration (if applicable):
• Auto Connect
• TCP Fast Open
• If you're not using VPN mode, please supply more details here:

What did you do?

Nothing specially.Firstly I found my phone cannot hold application on background. Everytime I switch apps between weibo and wechat or qq, app will reinit from splash screen.
Then when I playing some game. The game screen always freeze.
So I use adb shell top -s 10 to view the memory usage, and found that liboverture.so is using 800MB+ of RAM (There is 4GB of RAM in Xperia XZP).

What did you expect to see?

Less memory usage

What did you see instead?

[Mygod]

I used the same configuration and performed 5000 DNS queries using for i in $(seq 0 10000); do ping test$i & done and I didn't observe memory leak (overture RAM usage = 11M).

CPU time on your side is suspicious as well. On my side there is only ~9s after 5k queries.

[bangbang93]

My phone is running for serval days with shadowsocks running in background.
And now

6808 u0_a256 20 0 354M 80M 1.0M S 0.0 2.1 0:30.25 liboverture.so -c overture.conf -V

It's seems ok for now, I will keep tracking with this.

[wongsyrone]

liboverture.so is compiled by go1.9, you may want to upgrade to 1.10

[Mygod]

Is there a known mem leak issue in go 1.9? If there is it should have been fixed long before 1.10 methinks.

[wongsyrone]

Despite the mem leak, you can always benefit from the new version. e.g. new instructions

[bangbang93]

31252 u0_a256 20 0 1.2G 400M 1.2M S 0.3 10.3 16:25.89 liboverture.so -c overture.conf -V
I have kept connection for 2 days. 119MB TX and 170MB RX

My phone has not root. So I cannot dump the process memory

G8142:/proc/31252 $ cat status
Name:	liboverture.so
State:	S (sleeping)
Tgid:	31252
Pid:	31252
PPid:	8117
TracerPid:	0
Uid:	10256	10256	10256	10256
Gid:	10256	10256	10256	10256
Ngid:	0
FDSize:	1024
Groups:	3003 9997 20256 50256
VmPeak:	 1290608 kB
VmSize:	 1290608 kB
VmLck:	       0 kB
VmPin:	       0 kB
VmHWM:	  410164 kB
VmRSS:	  394320 kB
VmData:	 1239604 kB
VmStk:	     132 kB
VmExe:	    3092 kB
VmLib:	   41568 kB
VmPTE:	    2540 kB
VmPMD:	      28 kB
VmSwap:	   87308 kB
Threads:	276
SigQ:	0/14075
SigPnd:	0000000000000000
ShdPnd:	0000000000000000
SigBlk:	0000000000001204
SigIgn:	0000000000000000
SigCgt:	fffffffe7fc1feff
CapInh:	0000000000000000
CapPrm:	0000000000000000
CapEff:	0000000000000000
CapBnd:	0000000000000000
CapAmb:	0000000000000000
Seccomp:	2
Cpus_allowed:	ff
Cpus_allowed_list:	0-7
Mems_allowed:	1
Mems_allowed_list:	0
voluntary_ctxt_switches:	879
nonvoluntary_ctxt_switches:	1587

[madeye]

Threads:	276

It means there are a lot of queries waiting for responses.

Could you make sure that shadowsocks works well on your device?

[bangbang93]

Yes, I'm using this to playing game, chatting by telegram or twitter. All of them works fine.

[madeye]

Try this build: https://www.dropbox.com/s/8tsvs8k0j4at6z3/shadowsocks-nightly-18-2-24.apk

[Mygod]

276 threads taking 1 giga still sounds crazy. If those threads are just listening for DNS responses, they shouldn't take a lot of memory unless go is terribly inefficient.

[madeye]

It seems a memory leak in regexp of golang. According to the profiling result:

       flat  flat%   sum%        cum   cum%
  262.57MB 95.11% 95.11%   262.57MB 95.11%  regexp.(*bitState).reset /usr/local/Cellar/go/1.9.2/libexec/src/regexp/backtrack.go
   10.50MB  3.80% 98.91%    10.50MB  3.80%  regexp.progMachine /usr/local/Cellar/go/1.9.2/libexec/src/regexp/exec.go
       2MB  0.73% 99.64%        2MB  0.73%  regexp/syntax.(*compiler).inst /usr/local/Cellar/go/1.9.2/libexec/src/regexp/syntax/compile.go (inline)
         0     0% 99.64%   273.07MB 98.91%  github.com/miekg/dns.(*ServeMux).ServeDNS /Users/max/Develop/workspace/shadowsocks-android/core/src/overture/src/github.com/miekg/dns/server.go
         0     0% 99.64%   273.07MB 98.91%  github.com/miekg/dns.(*Server).serve /Users/max/Develop/workspace/shadowsocks-android/core/src/overture/src/github.com/miekg/dns/server.go

[Mygod]

golang/go#19573 might be relevant but fix has already been merged into 1.9.

[madeye]

(pprof) list regexp.progMachine
Total: 28300
ROUTINE ======================== regexp.progMachine in /usr/local/Cellar/go/1.9.2/libexec/src/regexp/exec.go
      9626       9626 (flat, cum) 34.01% of Total
         .          .     68:	return &m.inputReader
         .          .     69:}
         .          .     70:
         .          .     71:// progMachine returns a new machine running the prog p.
         .          .     72:func progMachine(p *syntax.Prog, op *onePassProg) *machine {
      5462       5462     73:	m := &machine{p: p, op: op}
         .          .     74:	n := len(m.p.Inst)
      2048       2048     75:	m.q0 = queue{make([]uint32, n), make([]entry, 0, n)}
      2116       2116     76:	m.q1 = queue{make([]uint32, n), make([]entry, 0, n)}
         .          .     77:	ncap := p.NumCap
         .          .     78:	if ncap < 2 {
         .          .     79:		ncap = 2
         .          .     80:	}
         .          .     81:	if op == notOnePass {

It looks golang's regexp is not that memory efficient.

[Mygod]

Apparently it isn't. golang/go#11646

[madeye]

@bangbang93 Try https://github.com/shadowsocks/shadowsocks-android/releases/tag/v4.5.0

[bangbang93]

30859 u0_a256 20 0 443M 35M 1.5M S 0.0 0.9 120:44.17 liboverture.so -c overture.conf -V
After one day usage, seems resolved

[Mygod]

Wait. @madeye So if I get this right, the compiled regexp is preventing all matchers from being GC-ed? Should we report this to upstream?

[madeye]

@Mygod Yes, I think so. But I don't know the root cause.

[Mygod]

@madeye Did you open an issue to upstream?

[madeye]

@Mygod Nope. Actually, I plan to use pcre directly in golang.