Ncat 7.94 "breaks udptunnel autopkgtest" on Debian.

[fyodor]

Debian Bug #1039613 reports that:

With a recent upload of nmap the autopkgtest of udptunnel fails in testing when that autopkgtest is run with the binary packages of nmap from unstable. It passes when run with only packages from testing....Currently this regression is blocking the migration of nmap to testing.

The Nmap version in testing is "7.94+dfsg1-1". A further comment on the Debian bug notes:

I think this is likely to be a bug in nmap ncat, when I remove
udptunnel from the script by connecting ncat directly to ncat,
then the failure still happens with the autopkgtest.

I was able to reproduce this issue outside a chroot. I have attached a
pair of scripts; bad fails and good succeeds. In the good case, I use
sleep to hold stdin of ncat -l open for 2s, in the bad case I let
it use the stdin provided by the shell (a GNOME terminal pty here).

Looking at the diff between the verbose logs is interesting.
In the good case one select fd is ready and one becomes ready later,
the latter one gets used, but in the bad case both fds appear to be
ready immediately but neither of them get used.

-NCAT DEBUG: select returned 1 fds ready
-NCAT DEBUG: fd 3 is ready
-NCAT DEBUG: selecting, fdmax 3
-NCAT DEBUG: select returned 1 fds ready
+NCAT DEBUG: select returned 2 fds ready

So the issue can be worked around in the udptunnel script by piping
sleep to the ncat listener instead of launching ncat and then sleeping,
or the ncat change that caused this can get bisected and fixed.

They say that version of Ncat in testing (which seems to be 7.93+dfsg1-1) still works. And given that we did make several Ncat changes in 7.94, we should probably investigate and fix whatever might have caused this apparent regression.

[samueloph]

I was about to open a bug report now that I confirmed it's an issue, thanks!

This regression was introduced on 4e6c8fe, I confirm the commit previous to this one does not trigger the issue.

I noticed that this only happens when running inside a script, I can't reproduce it manually/interactively.
To be more precise, if I open a terminal window, start an ncat server (same parameters), then send data from another terminal window, the data is written to the file.

This is the script I've used to git bisect the issue, it has some boilerplate that you might not need, but at least it can be used straight away for reproduction (assuming your cwd is the locally checked git repo).

#!/bin/bash

# Bail out and tell git this commit is untestable.
untestable() {
    git reset --hard
    exit 125
}

# Build ncat locally.
autoconf || untestable
./configure --build=x86_64-linux-gnu --includedir=\${prefix}/include --disable-option-checking --disable-silent-rules --libdir=\${prefix}/lib/x86_64-linux-gnu --disable-maintainer-mode --disable-dependency-tracking --with-liblua=/usr --with-liblinear --without-zenmap --without-ndiff --enable-ipv6 STRIP=/bin/true || untestable
make -j16 build-ncat || untestable

# Add extra verbosity and fail on any errors.
set -euo pipefail
set -x

# Variables.
UDPSERVER_PORT=5500
TX_FILE="tx.data"
RX_FILE="rx.data"

# Point to locally built ncat.
ncat() {
    ./ncat/ncat "$@"
}

# Double check we're running the right version.
ncat --version

# Start UDP server.
ncat -u -l 127.0.0.1 $UDPSERVER_PORT -o $RX_FILE &
sleep 3

# Create TX_FILE.
# Trailing whitespaces were left on purpose.
cat <<EOF > $TX_FILE
THIS  
 IS  
  THE  
   DATA 
    SEND 
     OVER 
      UDPTunnel 
EOF

# Send UDP data to the server.
cat $TX_FILE | ncat -u 127.0.0.1 $UDPSERVER_PORT

# Check if RX_FILE contains the data sent.
if ! diff $TX_FILE $RX_FILE; then
    echo "RX_FILE is not equals TX_FILE"
    # Cleanup git repo, for git bisect.
    git reset --hard
    exit 1
else
    # If we got here, then it worked.
    git reset --hard
    exit 0
fi

[dmiller-nmap]

The issue at the core here is what to do when EOF is received on STDIN. Previously, Ncat did not quit in this case for SOCK_DGRAM (i.e. UDP) listen mode, but it does do so for non-TCP SOCK_STREAM (i.e. SCTP). The commit which introduced this issue unified the listen mode code but did not handle the UDP case, which fell under the "non-TCP" part of the logic. The fix, which will be synchronized shortly, is for listen mode to only handle EOF on STDIN specially in the case of SOCK_STREAM (TCP and SCTP), restoring the previous behavior of UDP mode.

A helpful enhancement to Ncat would be to support the -q option as implemented by both Traditional netcat and OpenBSD netcat. This instructs netcat to wait for some time after receiving EOF on STDIN before quitting, which would accomplish the same thing as the "sleep | ncat" workaround mentioned in the Debian bug report.