tty output: May contain 'more data' than cc-runtime

[grahamwhaley]

Running:

 docker run -ti --runtime=cc-runtime ubuntu dmesg | wc
   1445   11185  110388

gives a significantly different answer than:

docker run -ti --runtime=kata-runtime ubuntu dmesg | wc
   1364   10516  917504

The line and word counts look similar enough, but the byte count is very different.
Now, it may be that kata is passing some of the control characters out from the dmesg more verbosely or similar, but I've not figured out why yet. If anybody has any ideas, please chip in.

Found this when running https://github.com/clearcontainers/tests/blob/master/metrics/time/launch_times.sh#L101, which failed to find the 'end' of the kernel boot - I suspect as the output is so much larger we may have simply run out of buffer space somewhere in the script/shell.

[jodh-intel]

Confirmed - I'm seeing a lot of nul bytes and what appears to be occasional corruption in the output.

/cc @sboeuf.

[jodh-intel]

Actually, no corruption - just lots of nul byte padding ;)

[grahamwhaley]

A little more info:

The extra NULL bytes appear on 'the end' of each buffer full/line of output from the containers.
The output does not have to come from dmesg - a simple echo will show it.

If you run:

$ docker run --rm -ti --runtime=kata-runtime busybox echo hello > crud
$ ls -la crud
 -rw-rw-r-- 1 guest guest 32768 Jan 29 14:42 crud

then you can see the file size is not the 7 bytes I might expect. And, that 32768 is a nicely suspicious binary size.
I suspect we have a buffer somewhere, and we are taking and transmitting the length of that buffer instead of the length of the valid contents of the buffer :-)
I had a peek around the agent, proxy and shim code, and I have a feeling this might be buried somewhere in the grpc code, maybe. Before I burn time annotating and digging, @amshinde @sboeuf - any obvious ideas or places you can have a quick peek to check?

[sboeuf]

Nothing obvious come to my mind but I'll take a look.

[sboeuf]

@grahamwhaley here is the fix: kata-containers/agent#124

[grahamwhaley]

Just confirming this does look fixed with kata-containers/agent#124, but now I see the possibly related kata-containers/agent#125