All events from Docker Imported as Error Level in Seq

[isaiahlg]

Hi!

Apologies if this is the wrong forum, let me know if there's a better place to post this. I've set up my organization's instance of Seq to ingest Docker logs from an image of NATS Streaming, but every single event imported shows up as an error:

I'm trying to understand the reason for this. The [INF] tag indicates an Information Level, as opposed to [WRN] or [ERR]. What is the logic that this service uses to determine the level of the event? Any help would be much appreciated.

Configurations for Seq and NATS included here...

NATS Configuration:

version: '3'
services:
  nats:
    image: nats-streaming:latest
    command: -m 8222 --cluster_id test-cluster --store SQL --sql_driver postgres --sql_source "${connection_string}"
    container_name: Nats
    ports:
      - "4222:4222"
      - "8222:8222"
    network_mode: host
    logging:
      driver: "gelf"
      options:
        gelf-address: "udp://localhost:12201"
    restart: always

Seq:

version: '2'

services:
  seq:
    image: datalust/seq:latest
    container_name: Seq_Server
    volumes:
      - seq:/data
    ports:
      - "5341:80"
    environment:
      - ACCEPT_EULA=Y
    mem_limit: 3g
    memswap_limit: 3g
    restart: always

  seq-input-gelf:
    image: datalust/seq-input-gelf:latest
    ports:
      - "12201:12201/udp"
    network_mode: host
    environment:
      SEQ_ADDRESS: "http://localhost:5341"
      restart: unless-stopped
    depends_on:
      - seq

volumes:
  seq:

[nblumhardt]

Hi Isaiah, thanks for dropping by! We'll dig into this and let you know what we find 👍

[KodrAus]

Hi @isaiahlg 👋

The server uses the GELF messages level property to determine what level to forward the event as and doesn't consider any content in the message itself. The level property should follow the standard syslog integer values, with 1 being alert, 3 being error, and 6 being info. I have noticed some logging frameworks give different meanings to these numbers though (I think it was fluentd I was looking at, which is Ruby).

I've merged #75, which is published on Docker Hub under the image datalust/sqelf-ci:2.0.290-dev. Is it possible to update your seq-input-gelf compose to something like:

 seq-input-gelf:
    image: datalust/sqelf-ci:2.0.290-dev
    ports:
      - "12201:12201/udp"
    network_mode: host
    environment:
      SEQ_ADDRESS: "http://localhost:5341"
      GELF_ENABLE_DIAGNOSTICS="True"
      restart: unless-stopped
    depends_on:
      - seq

Then, when events are logged in Seq you should see an extra raw_payload property attached that should give a clue of what level the message is being sent with.

[andymac4182]

I am having a similar issue with containers that write to stderr. eg postgres

{
	"version":"1.1",
	"host":"docker-desktop",
	"short_message":"2020-08-24 07:54:10.273 UTC [1] LOG:  database system is ready to accept connections",
	"timestamp":1598255650.273,
	"level":3,
	"_command":"docker-entrypoint.sh postgres",
	"_container_id":"8328b64fc67d229c87ff6cddca94e57cd23436e3f961cfd91f95e1221748ef27",
	"_container_name":"a9-automation-tool-server_postgres_1",
	"_created":"2020-08-24T07:45:12.2754228Z",
	"_image_id":"sha256:62473370e7eeea08c9ad60273081a32452c5ba461314490b893c11021e01aefd",
	"_image_name":"postgres",
	"_tag":"postgres"
}

[KodrAus]

Hey @andymac4182!

The culprit will be that "level":3, which corresponds to syslog's Error level. That's interesting. So your postgres container is logging to stdout and Docker's own log driver is forwarding those as GELF?

[andymac4182]

That was from the start up logs from postgres:latest I hadn't done anything other to set any options around logging. I am seeing the same behavior with a node library that is logging to stderr not stdout. In this case the offending library is https://www.npmjs.com/package/debug . I don't control the library that is using debug so I am looking into options to set config from the outside.

By default debug will log to stderr

[KodrAus]

I wouldn't go as far as to say that integer-based levels are bad but... integer-based levels are bad 🙂

This is something we could try and fix in the input here itself with configuration, but I'm a little concerned about accumulating too many ad-hoc pipeline-esc features (we've already got a few). I think the best way forward would be to use something like fluentbit to massage your logs on their way through and then treat the GELF input as a sink. It's unfortunately more machinery, but is shared infrastructure and already suitable for transforming logs from any shape into any other.