MTR does not play nice with nohup

[svdasein]

I've come across two instances - one on an older ubuntu intel and on an a very old sparc solaris - where attempting to run a loop around "mtr -c 1" in a backgrounded nohupped job with output & err > /dev/null and input < /dev/null leads to mtr sitting in the background doing apparently nothing (save for I think periodically checking a filehandle with a select call). I've tried all kinds of permutations and tricks & am becoming really puzzled by this. Am I doing something wrong or is there something keeping it from working in that context?

[svdasein]

Additional detail:

I tried this on a new distribution - same problem.

Replicate with:

nohup mtr -c 1 --split 8.8.8.8

This will never return. Strace shows:

select(10, [0 6 8 9], [], NULL, {0, 0}) = 1 (in [0], left {0, 0})
[repeating - very fast]

If you run:

nohup mtr -c 1 --split 8.8.8.8 < /dev/null > /dev/null 2>&1 &

you get the same strace pattern and it never returns.

[rewolff]

I've fixed a bug that might present itself just like this "for macos". But I suspect I may have fixed it for you as well. Are you running the latest version?

When select returns '1 (in[0])' the code can only do one thing, and that is issue a READ to filedescriptor zero.
read (0, "", 1) = -1 Exxx
You're seeing those as well? Right?

[rewolff]

I've looked at the "split.c" code, and it had a similar bug to what caused problems on macos. Fixed now. (I like single-line-fixes).

Still I can't reproduce your failure mode with the new or before-fix code.

[svdasein]

There was a circumstance in which I was also seeing a read(0...) but I can't seem to recreate that at the moment. I've tried with two versions of mtr: 0.86 and 0.92.25-fd41 (which is latest). The fast loop around a select is a constant though.

[svdasein]

I'll spin something fresh up & see if I can reproduce on that - if I can I'll send you creds to log in.

[svdasein]

I have set up a vm on which I've been able to re-create the problem with both the latest (from git) code and with the distribution's default package.

The older version is just a fast loop around:

select(10, [0 6 8 9], [], NULL, {0, 0}) = 1 (in [0], left {0, 0})

The latest version presents with two pids - one a child of another. The one that's taking a bunch of cpu is doing this:

select(8, [0 5 7], [], NULL, {0, 0}) = 1 (in [0], left {0, 0})
select(1, [0], NULL, NULL, {0, 0}) = 1 (in [0], left {0, 0})
read(0, "", 1) = 0

I have set up a login on the host for you with sudo. I will send a private message with credentials. Feel free to login at your convenience.

[svdasein]

Hm ok I guess I can't send a private message - is there some other way?

Edit/answer by REW: If you do a git clone, you can look in the commit messages to find Email addresses.

[svdasein]

rewolff - if you're on freenode, I'm dparker & I'm in #mtr right now - If you go there I'll send you the creds in a pm

[svdasein]

I was just looking at split.c - if that's where this is looping, I think maybe what's required is a new command line switch to just run without any option for interaction.

The use case is this:

I need to run a daemon (background, nohup) that repeatedly traces to an IP (-c 1). I'm taking the stdout from that and sending to logstash and from there to elasticsearch. This helped me track down a problem with thrashing routes and probably will be a useful tool going forward. However the inability to run mtr under nohup etc keeps me from being able to fire and forget - I have to leave a console open.

[rewolff]

Can you try again with the LATEST git. :-)
(as of 3 seconds ago. Has not even passed the "it compiles ship it" test. )

[svdasein]

I pulled and rebuilt:

rewolff@mtrtest:~$ /opt/mtr/sbin/mtr --version
mtr 0.92.26-ebdb

Foreground:

rewolff@mtrtest:~$ /opt/mtr/sbin/mtr -c 1 --split 8.8.8.8 > test.txt 
rewolff@mtrtest:~$ cat test.txt
1 23.92.24.3 0 1 1 0 0 0
2 173.230.159.2 0 1 1 0 0 0
3 206.197.187.50 0 1 1 4 4 4
3 as15169.sfmix.org 0 1 1 4 4 4
4 108.170.242.81 0 1 1 3 3 3
5 216.239.63.65 0 1 1 5 5 5
6 8.8.8.8 0 1 1 4 4 4
6 google-public-dns-a.google.com 0 1 1 4 4 4

nohup foreground:

rewolff@mtrtest:~$ nohup /opt/mtr/sbin/mtr -c 1 --split 8.8.8.8 > test.txt 
nohup: ignoring input and redirecting stderr to stdout
rewolff@mtrtest:~$ jobs
rewolff@mtrtest:~$ cat test.txt
rewolff@mtrtest:~$ ps auxw | grep mtr
rewolff  20102  0.0  0.0  14224   944 pts/0    S+   16:34   0:00 grep --color=auto mtr

nohup background:

rewolff@mtrtest:~$ nohup /opt/mtr/sbin/mtr -c 1 --split 8.8.8.8 > test.txt  2>&1  < /dev/null &
[1] 20114
rewolff@mtrtest:~$ 
[1]+  Done                    nohup /opt/mtr/sbin/mtr -c 1 --split 8.8.8.8 > test.txt 2>&1 < /dev/null
rewolff@mtrtest:~$ 
rewolff@mtrtest:~$ cat test.txt
rewolff@mtrtest:~$ 

[svdasein]

rewolff I sent you an email

[yvs2014]

$ nohup /opt/mtr/sbin/mtr -c 1 --split 8.8.8.8 > test.txt
$ nohup /opt/mtr/sbin/mtr -c 1 --split 8.8.8.8 > test.txt 2>&1 < /dev/null &

In split-mode, mtr monitors the input for some commands, and you are trying to make it read from an unreadable source. If it was supposed to get the output in a file, you would try that in report/raw modes.

[svdasein]

yvs2014 ah ok - I guess that was what I was missing (it doesn't really say that in the man page). I'll rework my stuff to use the raw format. Thanks - closing.

[rewolff]

WAIT WAIT!

You, if I analyzed this correctly, made MTR hang instead of doing the right thing.

The way I fixed it, I think it will now quit, correct? Are you seeing that too? Even if you're going to make use of a different mode, I would like this bug fixed anyway.

[svdasein]

rewolff - it did in fact quit, yes. The series of tests I showed illustrate that. So yes I think with your change it's now doing the right thing.

[AlexTan-b-z]

Hi, did you solve this problem? How?
I have the same problem.
Can you help me? Thank you!

[svdasein]

I converted everything to the --raw format. --raw is really the only option mtr has for non-interactive operation. --report was coded with the idea in mind that it's running underneath some interactive thing.

…

--raw is kind of a pita - you have to get your head into what's going on a bit more than you do with --report, but you could probably say that falls into "with great power comes great responsibility". Good luck.

On Sat, Jan 6, 2018 at 2:17 AM, AlexTan-b-z ***@***.***> wrote: Hi, did you solve this problem? How? I have the same problem. Can you help me? Thank you! — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <#231 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAfPIdNnHh4u99MyOq3WeFB7-4cjr5AFks5tHywxgaJpZM4Qk-wf> .

[rewolff]

As far as I know, --report is meant to run an mtr scan say every hour from cron. So why are you saying otherwise?

[AlexTan-b-z]

Thank you very much!