Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

segfaults #13

Open
Abraxas0 opened this issue Feb 24, 2019 · 11 comments
Open

segfaults #13

Abraxas0 opened this issue Feb 24, 2019 · 11 comments

Comments

@Abraxas0
Copy link

@Abraxas0 Abraxas0 commented Feb 24, 2019

Hi!

I love mmproxy, it helps me a lot hiding my SSH server behind a HA proxy. Unfortunately, multiple times a day the mmproxy process crashes with a segfault.

I have no experience reporting segfaults, this is my first report :) ... so I send you the information I think you need - please ask for further information, if needed:

OS: CentOS 7.5.1804
Kernel: 3.10.0-862.el7.x86_64

Log lines:

Feb 24 03:41:28 server kernel: [10342272.180498] mmproxy[59252]: segfault at 2b3e510 ip 0000000000444934 sp 00007fff29837dd0 error 4 in mmproxy[400000+fd000]
Feb 24 03:41:39 server kernel: [10342283.488181] mmproxy[17168]: segfault at 12bc510 ip 0000000000444934 sp 00007fff93723c00 error 4 in mmproxy[400000+fd000]
Feb 24 03:43:32 server kernel: [10342395.810838] mmproxy[17218]: segfault at 135e510 ip 0000000000444934 sp 00007fff070719e0 error 4 in mmproxy[400000+fd000]
Feb 24 03:43:42 server kernel: [10342406.580925] mmproxy[17583]: segfault at 2902510 ip 0000000000444934 sp 00007ffc2cd3c780 error 4 in mmproxy[400000+fd000]

Any thoughts on that?

Thank you for that great piece of software!

@SERIY1337

This comment has been minimized.

Copy link

@SERIY1337 SERIY1337 commented Mar 4, 2019

Hello. I have same problem with segmentation fault. When i try telnet mmproxy host, on host mmproxy crash and wrote in stdout - "Segmentation fault".

@majek

This comment has been minimized.

Copy link
Contributor

@majek majek commented May 24, 2019

I'm afraid I will need to see gdb backtrace. run mmproxy under gdb and type "bt" once it crashes.

$ gdb ./mmproxy
r --help --other-options
<when it crashes>
bt 
@Abraxas0

This comment has been minimized.

Copy link
Author

@Abraxas0 Abraxas0 commented May 24, 2019

I'm afraid I will need to see gdb backtrace. run mmproxy under gdb and type "bt" once it crashes.

Thank you! I will do so and commit the backtrace. Please stand by.

@Abraxas0

This comment has been minimized.

Copy link
Author

@Abraxas0 Abraxas0 commented May 24, 2019

Hello! I did what you have asked for and was able to reproduce a crash, but unfortunately with no backtrace (see output from gdb):

[root ~]# gdb /usr/sbin/mmproxy
GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-114.el7
...
(gdb) r --verbose --mark 256 --table 100 --allowed-networks /etc/mmproxy/ips.acl --force-real-client-port -l 10.128.3.136:2222 -4 127.0.0.1:2222 -6 [::1]:2222
...
Program terminated with signal SIGSYS, Bad system call.
The program no longer exists.
(gdb)
(gdb) bt
No stack.
(gdb)
@majek

This comment has been minimized.

Copy link
Contributor

@majek majek commented May 28, 2019

Oh! Interesting! This might be a seccomp bug. Please disable seccomp with "--do-not-sandbox", and run "strace -C" on the running mmproxy for a while and show the output.
Basically - did we forget to whitelist some syscall?

https://github.com/cloudflare/mmproxy/blob/master/src/main.c#L564-L601

@Abraxas0

This comment has been minimized.

Copy link
Author

@Abraxas0 Abraxas0 commented Jun 2, 2019

Hello! I'm back from holyday - sorry that you had to wait for my reply.

I did what you have asked for and was not able to reproduce a crash with the --do-not-sandbox parameter, but I think I remember that I have tried this parameter before and the process also crashed after a while. Unfortunately, the process sometimes crashes after a few seconds/minutes and sometimes after hours.

This is what I did now:

[root ~]# gdb /usr/sbin/mmproxy
GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-114.el7
...
(gdb) r --do-not-sandbox --verbose --mark 256 --table 100 --allowed-networks /etc/mmproxy/ips.acl --force-real-client-port -l 10.128.3.136:2222 -4 127.0.0.1:2222 -6 [::1]:2222

Find the output of "strace -C" (running for 5 minutes with multiple concurrent connections) attached:

strace.txt

Thank you for your help! Your suppor is really appriciated!

@pbennett

This comment has been minimized.

Copy link

@pbennett pbennett commented Jun 3, 2019

Another possibility is that bad data is being passed in the proxy headers. I saw this during some initial testing where the proxy headers I created had bad data in them (I let comma delimited data slip through in the IP address). The validation of 'bad' data in the mmproxy code is fairly limited from what I saw.

@Abraxas0

This comment has been minimized.

Copy link
Author

@Abraxas0 Abraxas0 commented Jun 20, 2019

Tiny update from my side:

Despite I was not able to reproduce the behaviour manually by running mmproxy with gdb within an adequate time, the mmproxy still keeps crashing when running as a daemon and while using the --do-not-sandbox parameter for multiple times a day.

Any thoughts on that or anything I can do to sort that out?

Your help is very appriciated! Thank's in advance!

@Rar349

This comment has been minimized.

Copy link

@Rar349 Rar349 commented Jun 21, 2019

I am having the same issue and can not seem to find a cause or a solution. I have tried everything mentioned here without any luck. The software is very useful but I can not continue using it for much longer if I can't get this issue nipped.

I would try to diagnose it myself but I do not really have much experience with software like this.

I would really appreciate any more possible solutions that anyone can suggest. I am still unsure if it's a problem with the software or just user error on my end.

@lindafarley

This comment has been minimized.

Copy link

@lindafarley lindafarley commented Jun 26, 2019

I am in the same boat as @Rar349. I found another version that seems to work at https://github.com/path-network/go-mmproxy if it helps anyone.

@patrickbsf

This comment has been minimized.

Copy link

@patrickbsf patrickbsf commented Jun 26, 2019

That looks fantastic and I'm a huge Go fan. Thank you @lindafarley !!! It's significantly faster as well.
What strikes me as odd is that CloudFlare is a big go user and this C code even uses a library that tries to emulate go channels/goroutines. I'm not sure why they didn't just write it in Go.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
7 participants
You can’t perform that action at this time.