Severity level missing in stackdriver in logging-winston 0.11.0 on Kubernetes container level logs

[darshan-willow]

winstion 3.2.1
google-cloud/logging-winston 0.11.0

Severity level is info even for logger.error in Kuberenets container logs but if the resources type is set to global on GCP Stackdriver, it works just fine. Is it a exprected behaviour or am I missing some thing? How to get the severity to work even at a container level logs?

[DominicKramer]

Thanks for opening this issue. This looks like a bug, but to help investigate could you share a snippet of code showing how you create and configure your LoggingWinston transport and Winston logger? Thanks.

[ofrobots]

@darshan-willow can you provide a repro and information as requested?

[DominicKramer]

I'm going to close this issue for now. If you have more info, just let us know and we can re-open it. Thanks.

[dgoerdes]

Seems like we currently have the same issue.

The used log level has no impact on the severity within the Stackdriver logs.
It is always logged with severity INFO.

Here is a reduced snippet of the used logging code and the resulting logs of Stackdriver.

Logging Snippet

import * as winston from 'winston';
import { LoggingWinston } from '@google-cloud/logging-winston';

const gcpLogging = new LoggingWinston({
    serviceContext: {
        service: 'my-service',
        version: 'qa',
    },
});

const logger = winston.createLogger({
    level: 'info',
    transports: [new winston.transports.Console(), gcpLogging],
});

logger.warn('A warning');
logger.error('An error');

Resulting log entries in Stackdriver

{
    "insertId": "d881dype9cnsk4viz",
    "jsonPayload": {
        "level": "warn",
        "message": "A warning"
    },
    "labels": {
        "k8s-pod/app": "project-api-qa",
        "k8s-pod/pod-template-hash": "7669f4d75d"
    },
    "logName": "projects/project-245310/logs/stdout",
    "receiveTimestamp": "2019-08-15T07:10:17.532550987Z",
    "resource": {
        "labels": {
            "cluster_name": "my-cluster",
            "container_name": "api-sha256",
            "location": "europe-west1-b",
            "namespace_name": "default",
            "pod_name": "project-api-qa-7669f4d75d-kgrs2",
            "project_id": "project-245310"
        },
        "type": "k8s_container"
    },
    "severity": "INFO",
    "timestamp": "2019-08-15T07:10:13.431368006Z"
}

{
    "insertId": "d881dype9cnsk5cwe",
    "jsonPayload": {
        "level": "error",
        "message": "An error"
    },
    "labels": {
        "k8s-pod/app": "project-api-qa",
        "k8s-pod/pod-template-hash": "847585684b"
    },
    "logName": "projects/project-245310/logs/stdout",
    "receiveTimestamp": "2019-08-15T07:52:09.002061295Z",
    "resource": {
        "labels": {
            "cluster_name": "my-cluster",
            "container_name": "api-sha256",
            "location": "europe-west1-b",
            "namespace_name": "default",
            "pod_name": "project-api-qa-847585684b-cgxtn",
            "project_id": "project-245310"
        },
        "type": "k8s_container"
    },
    "severity": "INFO",
    "timestamp": "2019-08-15T07:52:06.986580949Z"
}

[irokhes]

@dgoerdes @darshan-willow Did you manage to fix this issue? I'm having the same problem.

[darshan-willow]

const { LoggingWinston } = require('@google-cloud/logging-winston')

new winston.Logger({
  level: 'debug',
  transports: [
    loggingWinston,
    new (winston.transports.Console)({
      formatter(options) {
        const entry = {
          severity: options.level.toUpperCase(),
          message: options.message,
          metadata: options.meta
        }

        return JSON.stringify(entry).trim()
      }
    })
  ]
});

@irokhes I solved it by adding a formatter

[julienp]

I would love to know if there's a better solution for this?

[julienp]

I would love to know if there's a better solution for this?

Turns out I was looking at the stdout logs ingested via Kubernetes, it seems to work fine when I look at the winston_log.

[vongohren]

@irokhes I solved it by adding a formatter

Iv read that this is potentially a performance issue. As you do upperCase on ALL logggings happening

Iv read about adding your own custom levels, but not getting it to work properly

I have the same issue, and trying out this performance degrading way, because of this comment: winstonjs/winston#1345 (comment)

[hitzam-wartek]

Hi @dgoerdes,
I wonder how can you get these kubernetes informations attached to to the log

"resource": {
        "labels": {
            "cluster_name": "my-cluster",
            "container_name": "api-sha256",
            "location": "europe-west1-b",
            "namespace_name": "default",
            "pod_name": "project-api-qa-7669f4d75d-kgrs2",
            "project_id": "project-245310"
        },
        "type": "k8s_container"
    },

while I only got:

resource: {
	labels: {
		project_id: "myservice-staging"
	}
	type: "global"
}