How do I make envoy log to the systemd journal other than using /dev/stdout?

[Remz-Jay]

Description:

I'm trying to make Envoy output access logs to my Systemd Journal rather than to a file.

Following #1291 i tried setting envoy.http_connection_manager.access_log.config.path to /dev/stdout.
This works when I manually run Envoy using /usr/bin/envoy --config-path /etc/envoy/config.yaml, but fails when started through systemctl with error:

unable to open file '/dev/stdout': No such device or address

Turns out that stdout is a symlink on my system:

envoy # ls -lsa /dev/stdout
0 lrwxrwxrwx 1 root root 15 16 sep 20:23 /dev/stdout -> /proc/self/fd/1
envoy # ls -lsa /proc/self/fd/1
0 lrwx------ 1 root root 64 19 sep 17:31 /proc/self/fd/1 -> /dev/pts/1
envoy # ls -lsa /dev/pts/1
0 crw--w---- 1 remco tty 136, 1 19 sep 17:31 /dev/pts/1

I tried setting /dev/pts/1 as path, but that outputs the logs to my current console rather than to the systemd journal.

As such, how do I get Envoy to output logs to my journal rather than to a file, if /dev/stdout is not usable?

Version info:

envoy  version: 36cf26b9d6ac32f5671c24b92f9ea22ab70bc6c1/1.12.0-dev/Clean/RELEASE/BoringSSL

Manually compiled from source, running on Gentoo with Systemd, kernel 4.19.72 (so not in Docker).

Systemd unit:

# /etc/systemd/system/envoy.service
[Unit]
Description=Envoy Proxy
Documentation=https://www.envoyproxy.io/
After=network-online.target
Requires=envoy-auth-server.service
Wants=nginx.service

[Service]
User=root
Restart=on-failure
ExecStart=/usr/bin/envoy --config-path /etc/envoy/config.yaml
[Install]
WantedBy=multi-user.target

[stale]

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or other activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically closed because it has not had activity in the last 37 days. If this issue is still valid, please ping a maintainer and ask them to label it as "help wanted". Thank you for your contributions.

[cetanu]

Did you manage to figure out how to do this? I'm also looking into it.

[cstrahan]

When a systemd unit starts your process, it sets your process's file descriptor 1 (i.e. stdout) to an open Unix domain socket. You can observe this if you were to do this within your systemd unit:

readlink /dev/stdout
readlink -f /dev/stdout

You should see something like this:

/proc/self/fd/1
/proc/29290/fd/socket:[994577]

This shows that /dev/stdout is a symlink to /proc/self/fd/1, which is in turn a symlink to /proc/29290/fd/socket:[994577]. As you can see, your process's stdin is not a pipe, but rather an open Unix domain socket.

This behavior is actually documented in the systemd.exec documentation under the StandardOutput= section (emphasis mine):

[...]
If the standard output (or error output, see below) of a unit is connected to the journal or the kernel log buffer, the unit will implicitly gain a dependency of type After= on systemd-journald.socket (also see the "Implicit Dependencies" section above). Also note that in this case stdout (or stderr, see below) will be an AF_UNIX stream socket, and not a pipe or FIFO that can be re-opened. This means when executing shell scripts the construct echo "hello" > /dev/stderr for writing text to stderr will not work. To mitigate this use the construct echo "hello" >&2 instead, which is mostly equivalent and avoids this pitfall.
[...]

When your envoy config has access_log.config.path set to /dev/stdout, you're telling Envoy to open(2) /dev/stdout to subsequently write logs to. The problem here is that, within your systemd unit, if you follow the /dev/stdout symlink, that points to Unix domain socket: open(2) does not support opening UDS. If you look at that man page, you'll see there's an error code that open(2) will return if you try to give it a path to a UDS:

ENXIO The file is a UNIX domain socket.

To work around this (Linux) limitation, you could change your unit like so:

ExecStart=bash -c '/usr/bin/envoy --config-path /etc/envoy/config.yaml | tee'

That would make envoy's /dev/stdout (i.e. file descriptor 1) be one end of a pipe, which envoy will have no problem opening (again, if you refer to the man page, open(2) does support opening a link to a pipe). From there, tee will hand off envoy's output directly to its file descriptor 1, which is the one set up by systemd in order to log to the journal. tee will have no difficulty doing so, because it directly uses file descriptor 1, instead of trying to open /dev/stdout.

Alternatively (to using tee in this way), you can see if systemd-cat will work for your needs.

[pelov]

My feeling is that this problem is still not properly resolved as it requires running envoy under bash.

Since the issue was opened there have been some changes in the access logging configuration of envoy by introducing explicit envoy.access_loggers.stdout and envoy.access_loggers.stderr loggers. As far as I understand, they were trying to solve the problem of printing logs to stdout and stderr under windows (#13964). I was hoping that they will also solve the current problem but they haven't because the posix implementation still tries to open /dev/stdout and /dev/stderr which are missing under systemd and I am still getting the error:

unable to open file '/dev/stdout': No such device or address

My question is can't the posix implementation of stdout and stderr loggers also open directly fd 1 and 2 as the windows implementation and wouldn't that finally fix the problem?

I am also not sure if I should open a new issue (which essentially is the same problem as described here) or keep discussing it here.

[cetanu]

@dio (sorry, not sure who else to ping)
Could we get this re-opened, or should we submit a new issue to track the work to support stdout logging on linux?

[man-htsts]

To work around this (Linux) limitation, you could change your unit like so:

ExecStart=bash -c '/usr/bin/envoy --config-path /etc/envoy/config.yaml | tee'

pipefail option and sterr redirection to stdout can be also usefull here:

ExecStart=bash -o pipefail -c '/usr/bin/envoy --config-path /etc/envoy/config.yaml 2>&1 | tee'