prometheus could not log to syslog

[veksh]

Prometheus 2.3.2 lost an option for syslog logging (it was looking like
--log.format=logger:syslog?appname=prometheus&local=5), and now has only --log.level
and stderr logging.

[krasi-georgiev]

Which was the last version this worked on?
Prometheus switched logging packages few versions back.

[veksh]

Cannot remember exact version (log.format=logger.syslog is still OK with e.g. node_exporter). So this change is intentional? It is a pity then, having syslog was great for general troubleshooting. Some operational and configuration errors are hard to catch w/o logs (e.g. rule evaluation errors).

[simonpasquier]

The flag has been dropped since 2.0.0 (see #3219). There's prometheus/common#136 to add JSON logging to the common log library and eventually to Prometheus. Once that's in, it might be possible to enable back Syslog (no commitment though).

[veksh]

JSON would be nice too, but it is syslog that I miss. For systemd-based Linux that is less of an issue, but for other platforms there is no options to catch Prometheus messages except redirecting stdout to a file, which is a bit less flexible than standard syslog.

[simonpasquier]

redirecting stdout to a file, which is a bit less flexible than standard syslog.

Agreed. What I wanted to say is that adding JSON logging will bring back the log.format flag. Since the underlying log library already supports Syslog, it shouldn't be too complicated to put it back. Again just my thoughts.

[brian-brazil]

The decision thus far is that supporting stdout is sufficient, as the user can then hook that in however they like whether that be syslog or something else.

[ramato-procon]

currently logs are written to a log file in /tmp folder additionally. Writing logs to /tmp in a docker container is an antipattern IMHO. Therefore there should be an option to disable file logging at all.
JSON logging is a must for ELK stack.

[brian-brazil]

Prometheus does not write logs to a file, only to stdout. It is something in your setup that is doing that.

[ramato-procon]

Yes, you're right. The logs were created by OpenShift. Excuse me.

Nevertheless JSON logging is needed for ELK.

[ntindall]

I wrote this to convert logfmt to json .... https://github.com/ntindall/logfmt2json
but can we just merge prometheus/common#136 to bring back the json support?

[dlazerka]

@brian-brazil

Prometheus does not write logs to a file, only to stdout.

Actually, prometheus 2.2.1 writes to STDERR, not STDOUT. How to make it write to stdout when I don't have shell access? E.g. I'm running official vanilla prom/prometheus docker image on k8s. And by default, my logging collectors mark all the normal logs as errors because STDERR.