[Reporting] Kibana does not scan stderr from the Chromium process to diagnose OS issues

[tsullivan]

Kibana version: 5.6.4
Server OS version: CentOS 7
Original install method (e.g. download page, yum, from source, etc.): Minimal CentOS on Vagrant, Kibana installed via .tar.gz

When trying to run a Reporting job, Kibana should keep watch over the stderr logs from Chromium to try to detect messages about missing dependencies so it can be shown to the user in Kibana logs. See in source: https://github.com/elastic/kibana/blob/master/x-pack/plugins/reporting/server/browsers/chromium/driver_factory/index.js#L98-L106 Apparently that code is not working.

Describe the bug:
A minimal CentOS environment will have a lot of missing dependencies needed for the Chromium headless binary ([See CentOS dependencies in Puppeteer documentation](CentOS Dependencies)).
When I try to run a Reporting job in this environment, the job fails with Error spawning Chromium browser: Error: socket hang up, and there is not really any useful information in the logs.

When I try to run the headless binary manually with this command:

./headless_shell \
--disable-translate \
--disable-extensions \
--disable-background-networking \
--safebrowsing-disable-auto-update \
--disable-sync \
--metrics-recording-only \
--disable-default-apps \
--mute-audio \
--no-first-run \
--user-data-dir=. \
--disable-gpu \
--headless \
--hide-scrollbars \
--enable-logging \
--v=1

The stderr logs show: No usable sandbox! Update your kernel... This log is explicitly handled by the Kibana reporting code.

Why this might not be working, @kobelb wrote this in a chat:

I wonder if there's a race condition to us reading that from the [stderr] before puppeteer barfs because it can't connect on that port

[tsullivan]

@joelgriffith found that when Puppeteer encounters an error and decides to shut down Chromium, either gracefully or forcefully, it stops all the event listeners on the browser client. This is what turns off the listener we added for reading the error output.

We might be out of options for the idea to capture error output. The self-test feature added in #28001 might offer more help on the kind of troubleshooting we sought to do with the error output capturing.

[crankedguy]

Hi, I wanted to pick that up again as I also have the problems mentioned in the closed #28123 running Kibana 7.1.0 in a CentOS 7.6 server install.
I am getting the error

[0527/212719.493942:ERROR:egl_util.cc(59)] Failed to load GLES library: /var/lib/kibana/headless_shell-linux/swiftshader/libGLESv2.so: cannot open shared object file: No such file or directory

until I install the package chromium-libs.x86_64-73.0.3683.86-2.el7 which provides the libgles dependency. The problem for me is this installs 64 other packages as dependencies and messes up a more or less clean install. I will never use these dependencies for anything else...
Did you find another solution for this up until now?

The location where it is looked for in the error message is empty, there is no swiftshader directory, only the binary headless_shell

[root@ms1 filebeat]# ll /var/lib/kibana/headless_shell-linux/
total 111148
-rw-r--r--. 1 kibana kibana     18030 May 27 21:27 chrome_debug.log
-rwxr-xr-x. 1 kibana kibana 113780392 May 27 19:45 headless_shell
drwx------. 2 kibana kibana      4096 May 27 19:45 locales

Thanks

[joelgriffith]

Hey @crankedguy, we're definitely aware of this issue, however the workarounds are pretty tough and not well documented. I have it on my personal agenda to address this somehow.

[crankedguy]

@joelgriffith Thanks for the heads-up on this. I already thought of extracting this lib out of an rpm solely and installing it but this would only work out if there are no other dependencies etc. of course. From you it sounds there will be a lot more work to do. Hope you can sort this somehow as the solution of installing so many unneeded deps is not the most beautiful. Will keep an eye on it here, thanks!

[asgoth]

@crankedguy I had similar issues. For now, I "solved" it by picking the missing swiftshader directory from the chromium package:

# we only need chromium-headless, but there are missing files only available in chromium package
yum install -y chromium && \
cp -r /usr/lib64/chromium-browser/swiftshader /tmp && \
yum autoremove -y chromium && \
yum install -y chromium-headless fontconfig && \
cp -r /tmp/swiftshader /usr/lib64/chromium-browser

[tsullivan]

The problem of losing the stderr/stdout logs is due to: #43295

[tsullivan]

Hi all, there is a workaround for getting verbose logging from Chromium.

In our troubleshooting guide, we've recently added a section on accessing the debug logs from Puppeteer. The method is to start Kibana in an environment that has the DEBUG variable set, like so:

env DEBUG="puppeteer:*" ./bin/kibana