New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Invisible Glutton crash/panic #59

Closed
glaslos opened this Issue Mar 28, 2017 · 11 comments

Comments

Projects
None yet
3 participants
@glaslos
Member

glaslos commented Mar 28, 2017

I'm running Glutton directly on my server. Every 1-2 days the server becomes unreachable. I assume Glutton crashed and didn't clean up the iptables rules, locking me out from the machine. I have to reboot the box in order to access it again. I tail stderr to a file to see if there is any output, so far without luck. I assume we never execute https://github.com/kung-foo/freki/blob/master/freki.go#L245 on a panic.

@glaslos glaslos added the bug label Mar 28, 2017

@glaslos glaslos closed this Jun 6, 2017

@t3chn0m4g3

This comment has been minimized.

Show comment
Hide comment
@t3chn0m4g3

t3chn0m4g3 Apr 13, 2018

Contributor

I have the same behaviour when using glutton with network_mode: "host". Starting the container works fine and everything works as expected. Gracefully stopping (docker-compose down -v) results in the machine being unreachable.

Thoughts?

Contributor

t3chn0m4g3 commented Apr 13, 2018

I have the same behaviour when using glutton with network_mode: "host". Starting the container works fine and everything works as expected. Gracefully stopping (docker-compose down -v) results in the machine being unreachable.

Thoughts?

@glaslos glaslos reopened this Apr 13, 2018

@t3chn0m4g3

This comment has been minimized.

Show comment
Hide comment
@t3chn0m4g3

t3chn0m4g3 Apr 13, 2018

Contributor

I increased the graceful shutdown period to 120 seconds while monitoring the debug output, which basically prints the following lines for 120 seconds:

{"level":"info","ts":1523658022.4271533,"caller":"glutton/system.go:34","msg":"[system  ] running Go routines: 19 and open files: 9","sensorID":"2db13527-0def-4544-8727-4d02dee24a96"}
{"level":"info","ts":1523658032.4269712,"caller":"glutton/system.go:34","msg":"[system  ] running Go routines: 19 and open files: 9","sensorID":"2db13527-0def-4544-8727-4d02dee24a96"}
{"level":"info","ts":1523658042.4279432,"caller":"glutton/system.go:34","msg":"[system  ] running Go routines: 19 and open files: 9","sensorID":"2db13527-0def-4544-8727-4d02dee24a96"}
{"level":"info","ts":1523658052.427788,"caller":"glutton/system.go:34","msg":"[system  ] running Go routines: 19 and open files: 9","sensorID":"2db13527-0def-4544-8727-4d02dee24a96"}
{"level":"info","ts":1523658062.4275613,"caller":"glutton/system.go:34","msg":"[system  ] running Go routines: 19 and open files: 9","sensorID":"2db13527-0def-4544-8727-4d02dee24a96"}
{"level":"info","ts":1523658072.427133,"caller":"glutton/system.go:34","msg":"[system  ] running Go routines: 19 and open files: 9","sensorID":"2db13527-0def-4544-8727-4d02dee24a96"}
{"level":"info","ts":1523658082.427789,"caller":"glutton/system.go:34","msg":"[system  ] running Go routines: 19 and open files: 9","sensorID":"2db13527-0def-4544-8727-4d02dee24a96"}
{"level":"info","ts":1523658092.427497,"caller":"glutton/system.go:34","msg":"[system  ] running Go routines: 19 and open files: 9","sensorID":"2db13527-0def-4544-8727-4d02dee24a96"}
{"level":"info","ts":1523658102.4270444,"caller":"glutton/system.go:34","msg":"[system  ] running Go routines: 19 and open files: 9","sensorID":"2db13527-0def-4544-8727-4d02dee24a96"}
{"level":"info","ts":1523658112.4276683,"caller":"glutton/system.go:34","msg":"[system  ] running Go routines: 19 and open files: 9","sensorID":"2db13527-0def-4544-8727-4d02dee24a96"}

Once the container is stopped any open (and also by configuration set to pass through SSH) connection will be cut off, however restarting the container also results in the connections being revived.

iptables while container is active:

iptables -t raw -L
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination         
NFQUEUE    tcp  --  anywhere             anywhere             NFQUEUE num 0
NFQUEUE    udp  --  anywhere             anywhere             NFQUEUE num 0

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
NFQUEUE    tcp  --  anywhere             anywhere             NFQUEUE num 0
NFQUEUE    udp  --  anywhere             anywhere             NFQUEUE num 0

And after stopping the iptables raw still remain:
image

Deleting them with iptables -t raw -F restores connectivity to the machine.

Contributor

t3chn0m4g3 commented Apr 13, 2018

I increased the graceful shutdown period to 120 seconds while monitoring the debug output, which basically prints the following lines for 120 seconds:

{"level":"info","ts":1523658022.4271533,"caller":"glutton/system.go:34","msg":"[system  ] running Go routines: 19 and open files: 9","sensorID":"2db13527-0def-4544-8727-4d02dee24a96"}
{"level":"info","ts":1523658032.4269712,"caller":"glutton/system.go:34","msg":"[system  ] running Go routines: 19 and open files: 9","sensorID":"2db13527-0def-4544-8727-4d02dee24a96"}
{"level":"info","ts":1523658042.4279432,"caller":"glutton/system.go:34","msg":"[system  ] running Go routines: 19 and open files: 9","sensorID":"2db13527-0def-4544-8727-4d02dee24a96"}
{"level":"info","ts":1523658052.427788,"caller":"glutton/system.go:34","msg":"[system  ] running Go routines: 19 and open files: 9","sensorID":"2db13527-0def-4544-8727-4d02dee24a96"}
{"level":"info","ts":1523658062.4275613,"caller":"glutton/system.go:34","msg":"[system  ] running Go routines: 19 and open files: 9","sensorID":"2db13527-0def-4544-8727-4d02dee24a96"}
{"level":"info","ts":1523658072.427133,"caller":"glutton/system.go:34","msg":"[system  ] running Go routines: 19 and open files: 9","sensorID":"2db13527-0def-4544-8727-4d02dee24a96"}
{"level":"info","ts":1523658082.427789,"caller":"glutton/system.go:34","msg":"[system  ] running Go routines: 19 and open files: 9","sensorID":"2db13527-0def-4544-8727-4d02dee24a96"}
{"level":"info","ts":1523658092.427497,"caller":"glutton/system.go:34","msg":"[system  ] running Go routines: 19 and open files: 9","sensorID":"2db13527-0def-4544-8727-4d02dee24a96"}
{"level":"info","ts":1523658102.4270444,"caller":"glutton/system.go:34","msg":"[system  ] running Go routines: 19 and open files: 9","sensorID":"2db13527-0def-4544-8727-4d02dee24a96"}
{"level":"info","ts":1523658112.4276683,"caller":"glutton/system.go:34","msg":"[system  ] running Go routines: 19 and open files: 9","sensorID":"2db13527-0def-4544-8727-4d02dee24a96"}

Once the container is stopped any open (and also by configuration set to pass through SSH) connection will be cut off, however restarting the container also results in the connections being revived.

iptables while container is active:

iptables -t raw -L
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination         
NFQUEUE    tcp  --  anywhere             anywhere             NFQUEUE num 0
NFQUEUE    udp  --  anywhere             anywhere             NFQUEUE num 0

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
NFQUEUE    tcp  --  anywhere             anywhere             NFQUEUE num 0
NFQUEUE    udp  --  anywhere             anywhere             NFQUEUE num 0

And after stopping the iptables raw still remain:
image

Deleting them with iptables -t raw -F restores connectivity to the machine.

@glaslos

This comment has been minimized.

Show comment
Hide comment
@glaslos

glaslos Apr 14, 2018

Member

Hm, seems like some sort of deadlock.

Member

glaslos commented Apr 14, 2018

Hm, seems like some sort of deadlock.

@t3chn0m4g3

This comment has been minimized.

Show comment
Hide comment
@t3chn0m4g3

t3chn0m4g3 Apr 15, 2018

Contributor

Docker sends by default a SIGHUP to end processes, or does glutton need a different signal to trigger a clean up before exiting?

Contributor

t3chn0m4g3 commented Apr 15, 2018

Docker sends by default a SIGHUP to end processes, or does glutton need a different signal to trigger a clean up before exiting?

@glaslos

This comment has been minimized.

Show comment
Hide comment
@glaslos

glaslos Apr 15, 2018

Member

Ah, that might be something, I am expecting a SIGINT

Member

glaslos commented Apr 15, 2018

Ah, that might be something, I am expecting a SIGINT

@glaslos

This comment has been minimized.

Show comment
Hide comment
@glaslos

glaslos Apr 15, 2018

Member

I think it makes most sense to handle SIGHUP in Glutton as well. Let me add that line..

Member

glaslos commented Apr 15, 2018

I think it makes most sense to handle SIGHUP in Glutton as well. Let me add that line..

@glaslos

This comment has been minimized.

Show comment
Hide comment
@glaslos

glaslos Apr 15, 2018

Member

I added SIGHUP to the interrupt handler, check if that works now.

Member

glaslos commented Apr 15, 2018

I added SIGHUP to the interrupt handler, check if that works now.

@t3chn0m4g3

This comment has been minimized.

Show comment
Hide comment
@t3chn0m4g3

t3chn0m4g3 Apr 15, 2018

Contributor

Just tested it and it works if CMD in Dockerfile is in exec form:

CMD ["bin/server", "-i", "ens33", "-l", "/var/log/glutton.log", "-d", "true"]

... shell form does not work, which I am guessing is caused by SIGHUP going to the shell which probably terminates by different a signal:

CMD bin/server -i ens33 -l /var/log/glutton/glutton.log -d true

Thank you @glaslos

Contributor

t3chn0m4g3 commented Apr 15, 2018

Just tested it and it works if CMD in Dockerfile is in exec form:

CMD ["bin/server", "-i", "ens33", "-l", "/var/log/glutton.log", "-d", "true"]

... shell form does not work, which I am guessing is caused by SIGHUP going to the shell which probably terminates by different a signal:

CMD bin/server -i ens33 -l /var/log/glutton/glutton.log -d true

Thank you @glaslos

@glaslos

This comment has been minimized.

Show comment
Hide comment
@glaslos

glaslos Apr 15, 2018

Member

No sweat, thanks for reporting!

Member

glaslos commented Apr 15, 2018

No sweat, thanks for reporting!

@glaslos glaslos closed this Apr 15, 2018

@t3chn0m4g3

This comment has been minimized.

Show comment
Hide comment
@t3chn0m4g3

t3chn0m4g3 Apr 15, 2018

Contributor

@glaslos For the record, the tests this morning were showing SIGHUPs for whatever reason, but I got this wrong. Checked the FAQs and docker sends by default a SIGTERM.

Contributor

t3chn0m4g3 commented Apr 15, 2018

@glaslos For the record, the tests this morning were showing SIGHUPs for whatever reason, but I got this wrong. Checked the FAQs and docker sends by default a SIGTERM.

@t3chn0m4g3

This comment has been minimized.

Show comment
Hide comment
@t3chn0m4g3

t3chn0m4g3 Apr 15, 2018

Contributor

If anyone else needs the docker shell form instead of the exec form exec will do the trick by replacing the shell process resulting in SIGTERM reaching glutton:

CMD exec bin/server -i ens33 -l /var/log/glutton/glutton.log -d true
Contributor

t3chn0m4g3 commented Apr 15, 2018

If anyone else needs the docker shell form instead of the exec form exec will do the trick by replacing the shell process resulting in SIGTERM reaching glutton:

CMD exec bin/server -i ens33 -l /var/log/glutton/glutton.log -d true
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment