Version 2.8.6 of the Docker container breaks with Linux kernel 4.17

[pkgw]

If I try to use Decktape with my particular setup, version 2.8.5 of the Docker container works fine, but version 2.8.6 fails.

Invocation. For version 2.8.5 the only change is the version number in the Docker command:

exec docker run \
     --rm \
     --net=host \
     -v $(pwd):/slides:rw,Z \
     --shm-size 2G \
     astefanutti/decktape:2.8.6 \
     -s 1600x1200 \
     reveal http://localhost:23232/slides.html export.pdf

Error message with 2.8.6:

Error: Failed to launch chrome!


TROUBLESHOOTING: https://github.com/GoogleChrome/puppeteer/blob/master/docs/troubleshooting.md

    at onClose (/decktape/node_modules/puppeteer/lib/Launcher.js:285:14)
    at Interface.helper.addEventListener (/decktape/node_modules/puppeteer/lib/Launcher.js:274:50)
    at Interface.emit (events.js:185:15)
    at Interface.close (readline.js:379:8)
    at Socket.onend (readline.js:152:10)
    at Socket.emit (events.js:185:15)
    at endReadableNT (_stream_readable.js:1106:12)
    at process._tickCallback (internal/process/next_tick.js:178:19)

If I add -e DEBUG="puppeteer:*" to the command line, I don't get any additional output.

For completeness' sake, here's the output from version 2.8.5:

Loading page http://localhost:23232/slides.html ...
Loading page finished with status: 200
Reveal JS plugin activated
Printing slide #/3      (4/4) ...
Printed 4 slides

[astefanutti]

I'm unable to reproduce locally unfortunately.

Aside from the Puppeteer / Chromium version that has changed from 2.8.5 to 2.8.6, the --disable-dev-shm-usage flag has been added by default. So maybe you could try running the command without the --shm-size option.

There is a dumpio option exposed by Puppeteer, that would be helpful to set. Unfortunately, there is no way to set it from the command line currently:

decktape/decktape.js

Line 183 in 4b31b5d

// dumpio : true,

If rebuilding the image locally is an option for you, that would help. Otherwise I will see to expose this option from DeckTape CLI.

[pkgw]

I tried removing the SHM flag but that didn't fix things, unfortunately.

I built a new version of the container with dumpio set to true, and the output I get is exactly the same. (With the DEBUG environment variable or not.) Is there another way to get more info? I guess Puppeteer is dying before just about any of its subsystems get initialized.

[astefanutti]

Any that I'm aware of. What I do in such situation is to run a shell and try starting Chromium manually, like:

$ docker run --rm -it --entrypoint=sh astefanutti/decktape:2.8.6
$ chromium-browser --no-sandbox --disable-gpu --disable-software-rasterizer --headless --mute-a
udio --hide-scrollbars

That may give some more information.

[pkgw]

Uh oh. If I rebuild the Docker container off of the v2.8.5 tag, I still get the same problem. That implies that some downloaded dependency has changed in a way that breaks things on my machine. Fun times.

[astefanutti]

Uh oh. If I rebuild the Docker container off of the v2.8.5 tag, I still get the same problem. That implies that some downloaded dependency has changed in a way that breaks things on my machine. Fun times.

Yes, that is the Chromium version then. The Dockerfile picks the latest Chromium build for Alpine. v2.8.5 was based on Chromium 64, whereas v2.8.6 uses Chromium 68.

[pkgw]

Instant segfault when running Chromium manually as suggested.

[astefanutti]

What is your environment? Docker version, host OS...

Any useful information logged before segfaulting?

I've just found https://bugs.alpinelinux.org/issues/9161, that may be related.

[pkgw]

Yes, I wouldn't be surprised if that's related.

Machine is running Fedora 28, kernel 4.17.11-200, 8-core x86_64 Dell XPS 15 laptop. Docker 18.06.0-ce build 0ffa825.

No output before the segfault. I get some stuff if I strace it, but nothing that jumps out as incriminating.

Thanks for helping track this down!

[astefanutti]

I've just upgraded the Docker image to Alpine 3.8. Would you be able to give it a spin once it's been built on Docker Hub (or you get a local build)?

[pkgw]

Built it locally, still instant segfault, alas.

[pkgw]

Holy cow:

$ ldd /usr/lib/chromium/chrome
	/lib/ld-musl-x86_64.so.1 (0x7f5454c43000)
Segmentation fault

I.e., ldd segfaults! This doesn't happen for a sample of other binaries in the container. I believe that ldd does some magic where it launches the executable with some environment variables set but basically things seem really messed up ... on my particular setup.

[astefanutti]

I've just tried the following:

$ docker run --rm -it alpine:3.8 sh
/ # apk add --no-cache chromium
/ # ldd /usr/lib/chromium/chrome
	/lib/ld-musl-x86_64.so.1 (0x7fb05f748000)
Segmentation fault

However, I cannot reproduce the segmentation fault while running chromium:

/ # chromium-browser --no-sandbox --disable-gpu --disable-software-rasterizer --headless --mute-a
udio --hide-scrollbars
[0810/085819.395117:WARNING:dns_config_service_posix.cc(333)] Failed to read DnsConfig.
[0810/085819.397899:ERROR:gpu_process_transport_factory.cc(1016)] Lost UI shared context.

Could you try running the commands above? At least that would rule the DeckTape image out and help narrowing the issue.

[pkgw]

I get what you might expect: segfaults for both the ldd command, and running chromium-browser.

I noticed some error messages in my kernel logs:

Aug 13 14:48:28 lambda.localdomain audit[11537]: ANOM_ABEND auid=4294967295 uid=0 gid=0 ses=4294967295 subj=system_u:system_r:spc_t:s0 pid=11537 comm="ldd" exe="/lib/ld-musl-x86_64.so.1" sig=11 res=1
Aug 13 14:48:28 lambda.localdomain kernel: ldd[11537]: segfault at 7faf383e9010 ip 00007faf40845771 sp 00007ffdb7d1cce8 error 7 in ld-musl-x86_64.so.1[7faf407ed000+8d000]
Aug 13 14:48:43 lambda.localdomain audit[11582]: ANOM_ABEND auid=4294967295 uid=0 gid=0 ses=4294967295 subj=system_u:system_r:spc_t:s0 pid=11582 comm="chrome" exe="/usr/lib/chromium/chrome" sig=11 res=1
Aug 13 14:48:43 lambda.localdomain kernel: 11582 (chrome): Uhuuh, elf segment at 0000563c6d98d000 requested but the memory is mapped already

This bug is using a very different environment, but it involves chromium giving this error on Linux kernel 4.17, which is what I'm running.

[pkgw]

Other potentially related items: manjaro.org, r/linuxquestions, r/archlinux, kernel.org, chromium.org.

I am willing to say that whatever is going on here, the cause is very unrelated to decktape. Feel free to close if you'd like, although maybe people will come looking here for workarounds.

[astefanutti]

Thanks a lot for the useful information. Let's leave that issue open until we see progress on the above issues.

[astefanutti]

I've upgraded Chromium in 3dfea7b and it seems I cannot reproduce the segmentation fault.

Let me close this speculatively. Do not hesitate to reopen if you still have the issue with Decktape version 2.9.1.