Skip to content

Commit

Permalink
feat: Add support to print structured logging to STDOUT (#676)
Browse files Browse the repository at this point in the history 
* feat: Add support to print structured logging to STDOUT

* Add newline

* 🦉 Updates from OwlBot post-processor

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

* Address PR comments

* 🦉 Updates from OwlBot post-processor

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

* Rename stackdriver variable

* Rename stackdriverLog in tests

Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>
  • Loading branch information
@losalex @gcf-owl-bot
losalex and gcf-owl-bot[bot] committed Mar 18, 2022
1 parent 077960d commit 76135ca
Show file tree
Hide file tree
Showing 7 changed files with 149 additions and 50 deletions.
29 changes: 26 additions & 3 deletions .readme-partials.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -81,13 +81,13 @@ body: |-
### Error Reporting
Any `Error` objects you log at severity `error` or higher can automatically be picked up by [Stackdriver Error Reporting](https://cloud.google.com/error-reporting/) if you have specified a `serviceContext.service` when instantiating a `LoggingWinston` instance:
Any `Error` objects you log at severity `error` or higher can automatically be picked up by [Error Reporting](https://cloud.google.com/error-reporting/) if you have specified a `serviceContext.service` when instantiating a `LoggingWinston` instance:
```javascript
const loggingWinston = new LoggingWinston({
serviceContext: {
service: 'my-service', // required to report logged errors
// to the Google Cloud Error Reporting
// to the Error Reporting
// console
version: 'my-version'
}
Expand All @@ -102,7 +102,7 @@ body: |-
### Error handling with a default callback
The `LoggingWinston` class creates an instance of `LoggingCommon` which uses the `Log` class from `@google-cloud/logging` package to write log entries.
The `LoggingWinston` class creates an instance of `LoggingCommon` which by default uses the `Log` class from `@google-cloud/logging` package to write log entries.
The `Log` class writes logs asynchronously and there are cases when log entries cannot be written and an error is
thrown - if error is not handled properly, it could crash the application. One possible way to handle the error is to provide a default callback
to the `LoggingWinston` constructor which will be used to initialize `Log` object with that callback like in example below:
Expand Down Expand Up @@ -211,3 +211,26 @@ body: |-
```
![Request log with prefix](https://raw.githubusercontent.com/googleapis/nodejs-logging-winston/master/doc/images/request-log-with-prefix.png)
### Alternative way to ingest logs in Google Cloud managed environments
If you use this library with the Cloud Logging Agent, you can configure the handler to output logs to `process.stdout` using
the [structured logging Json format](https://cloud.google.com/logging/docs/structured-logging#special-payload-fields).
To do this, add `redirectToStdout: true` parameter to the `LoggingWinston` constructor as in sample below.
You can use this parameter when running applications in Google Cloud managed environments such as AppEngine, Cloud Run,
Cloud Function or GKE. The logger agent installed on these environments can capture `process.stdout` and ingest it into Cloud Logging.
The agent can parse structured logs printed to `process.stdout` and capture additional log metadata beside the log payload.
It is recommended to set `redirectToStdout: true` in serverless environments like Cloud Functions since it could
decrease logging record loss upon execution termination - since all logs are written to `process.stdout` those
would be picked up by the Cloud Logging Agent running in Google Cloud managed environment.
```js
// Imports the Google Cloud client library for Winston
const {LoggingWinston} = require('@google-cloud/logging-winston');
// Creates a client that writes logs to stdout
const loggingWinston = new LoggingWinston({
projectId: 'your-project-id',
keyFilename: '/path/to/key.json',
redirectToStdout: true,
});
28 changes: 25 additions & 3 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -158,13 +158,13 @@ main();

### Error Reporting

Any `Error` objects you log at severity `error` or higher can automatically be picked up by [Stackdriver Error Reporting](https://cloud.google.com/error-reporting/) if you have specified a `serviceContext.service` when instantiating a `LoggingWinston` instance:
Any `Error` objects you log at severity `error` or higher can automatically be picked up by [Error Reporting](https://cloud.google.com/error-reporting/) if you have specified a `serviceContext.service` when instantiating a `LoggingWinston` instance:

```javascript
const loggingWinston = new LoggingWinston({
serviceContext: {
service: 'my-service', // required to report logged errors
// to the Google Cloud Error Reporting
// to the Error Reporting
// console
version: 'my-version'
}
Expand All @@ -179,7 +179,7 @@ You may also want to see the [@google-cloud/error-reporting](https://github.com/

### Error handling with a default callback

The `LoggingWinston` class creates an instance of `LoggingCommon` which uses the `Log` class from `@google-cloud/logging` package to write log entries.
The `LoggingWinston` class creates an instance of `LoggingCommon` which by default uses the `Log` class from `@google-cloud/logging` package to write log entries.
The `Log` class writes logs asynchronously and there are cases when log entries cannot be written and an error is
thrown - if error is not handled properly, it could crash the application. One possible way to handle the error is to provide a default callback
to the `LoggingWinston` constructor which will be used to initialize `Log` object with that callback like in example below:
Expand Down Expand Up @@ -289,6 +289,28 @@ logger.debug('test msg');

![Request log with prefix](https://raw.githubusercontent.com/googleapis/nodejs-logging-winston/master/doc/images/request-log-with-prefix.png)

### Alternative way to ingest logs in Google Cloud managed environments
If you use this library with the Cloud Logging Agent, you can configure the handler to output logs to `process.stdout` using
the [structured logging Json format](https://cloud.google.com/logging/docs/structured-logging#special-payload-fields).
To do this, add `redirectToStdout: true` parameter to the `LoggingWinston` constructor as in sample below.
You can use this parameter when running applications in Google Cloud managed environments such as AppEngine, Cloud Run,
Cloud Function or GKE. The logger agent installed on these environments can capture `process.stdout` and ingest it into Cloud Logging.
The agent can parse structured logs printed to `process.stdout` and capture additional log metadata beside the log payload.
It is recommended to set `redirectToStdout: true` in serverless environments like Cloud Functions since it could
decrease logging record loss upon execution termination - since all logs are written to `process.stdout` those
would be picked up by the Cloud Logging Agent running in Google Cloud managed environment.

```js
// Imports the Google Cloud client library for Winston
const {LoggingWinston} = require('@google-cloud/logging-winston');

// Creates a client that writes logs to stdout
const loggingWinston = new LoggingWinston({
projectId: 'your-project-id',
keyFilename: '/path/to/key.json',
redirectToStdout: true,
});


## Samples

Expand Down
71 changes: 50 additions & 21 deletions src/common.ts
View file
Original file line number Diff line number Diff line change
Expand Up @@ -19,10 +19,12 @@ import {
ServiceContext,
SeverityNames,
Log,
LogSync,
} from '@google-cloud/logging';
import {LogSeverityFunctions} from '@google-cloud/logging/build/src/utils/log-common';
import mapValues = require('lodash.mapvalues');
import {Options} from '.';
import {LogEntry} from '@google-cloud/logging/build/src/entry';
import {Entry, LogEntry} from '@google-cloud/logging/build/src/entry';

type Callback = (err: Error | null, apiResponse?: {}) => void;
export type MonitoredResource = protos.google.api.MonitoredResource;
Expand All @@ -42,7 +44,7 @@ export interface Metadata {
[key: string]: any;
}

// Map of npm output levels to Stackdriver Logging levels.
// Map of npm output levels to Cloud Logging levels.
const NPM_LEVEL_NAME_TO_CODE = {
error: 3,
warn: 4,
Expand All @@ -52,8 +54,8 @@ const NPM_LEVEL_NAME_TO_CODE = {
silly: 7,
};

// Map of Stackdriver Logging levels.
const STACKDRIVER_LOGGING_LEVEL_CODE_TO_NAME: {
// Map of Cloud Logging levels.
const CLOUD_LOGGING_LEVEL_CODE_TO_NAME: {
[key: number]: SeverityNames;
} = {
0: 'emergency',
Expand Down Expand Up @@ -110,12 +112,13 @@ export class LoggingCommon {
readonly logName: string;
private inspectMetadata: boolean;
private levels: {[name: string]: number};
stackdriverLog: Log;
cloudLog: LogSeverityFunctions;
private resource: protos.google.api.IMonitoredResource | undefined;
private serviceContext: ServiceContext | undefined;
private prefix: string | undefined;
private labels: object | undefined;
private defaultCallback?: Callback;
redirectToStdout: boolean;
// LOGGING_TRACE_KEY is Cloud Logging specific and has the format:
// logging.googleapis.com/trace
static readonly LOGGING_TRACE_KEY = LOGGING_TRACE_KEY;
Expand All @@ -134,13 +137,19 @@ export class LoggingCommon {
this.logName = options.logName || 'winston_log';
this.inspectMetadata = options.inspectMetadata === true;
this.levels = options.levels || NPM_LEVEL_NAME_TO_CODE;
this.stackdriverLog = new Logging(options).log(this.logName, {
removeCircular: true,
// See: https://cloud.google.com/logging/quotas, a log size of
// 250,000 has been chosen to keep us comfortably within the
// 256,000 limit.
maxEntrySize: options.maxEntrySize || 250000,
});
this.redirectToStdout = options.redirectToStdout ?? false;

if (!this.redirectToStdout) {
this.cloudLog = new Logging(options).log(this.logName, {
removeCircular: true,
// See: https://cloud.google.com/logging/quotas, a log size of
// 250,000 has been chosen to keep us comfortably within the
// 256,000 limit.
maxEntrySize: options.maxEntrySize || 250000,
});
} else {
this.cloudLog = new Logging(options).logSync(this.logName);
}
this.resource = options.resource;
this.serviceContext = options.serviceContext;
this.prefix = options.prefix;
Expand All @@ -163,15 +172,15 @@ export class LoggingCommon {
}

const levelCode = this.levels[level];
const stackdriverLevel = STACKDRIVER_LOGGING_LEVEL_CODE_TO_NAME[levelCode];
const cloudLevel = CLOUD_LOGGING_LEVEL_CODE_TO_NAME[levelCode];

const data: StackdriverData = {};

// Stackdriver Logs Viewer picks up the summary line from the `message`
// Cloud Logs Viewer picks up the summary line from the `message`
// property of the jsonPayload.
// https://cloud.google.com/logging/docs/view/logs_viewer_v2#expanding.
//
// For error messages at severity 'error' and higher, Stackdriver
// For error messages at severity 'error' and higher,
// Error Reporting will pick up error messages if the full stack trace is
// included in the textPayload or the message property of the jsonPayload.
// https://cloud.google.com/error-reporting/docs/formatting-error-messages
Expand All @@ -198,7 +207,7 @@ export class LoggingCommon {
}

// If the metadata contains a httpRequest property, promote it to the
// entry metadata. This allows Stackdriver to use request log formatting.
// entry metadata. This allows Cloud Logging to use request log formatting.
// https://cloud.google.com/logging/docs/reference/v2/rest/v2/LogEntry#HttpRequest
// Note that the httpRequest field must properly validate as HttpRequest
// proto message, or the log entry would be rejected by the API. We no do
Expand Down Expand Up @@ -256,11 +265,31 @@ export class LoggingCommon {
delete data.metadata!.logName;
}

const entry = this.stackdriverLog.entry(entryMetadata, data);
this.stackdriverLog[stackdriverLevel](
entry,
this.defaultCallback ?? callback
);
const entry = this.entry(entryMetadata, data);
// Make sure that both callbacks are called in case if provided
const newCallback: Callback = (err: Error | null, apiResponse?: {}) => {
if (callback) {
callback(err, apiResponse);
}
if (this.defaultCallback) {
this.defaultCallback(err, apiResponse);
}
};
this.cloudLog[cloudLevel](entry, newCallback);
// The LogSync class does not supports callback. However Writable class always
// provides onwrite() callback which needs to be called after each log is written,
// so the stream would remove writing state. Since this.defaultCallback can also be set, we
// should call it explicitly as well.
if (this.redirectToStdout) {
newCallback(null, undefined);
}
}

entry(metadata?: LogEntry, data?: string | {}): Entry {
if (this.redirectToStdout) {
return (this.cloudLog as LogSync).entry(metadata, data);
}
return (this.cloudLog as Log).entry(metadata, data);
}
}

Expand Down
9 changes: 9 additions & 0 deletions src/index.ts
View file
Original file line number Diff line number Diff line change
Expand Up @@ -84,11 +84,20 @@ export interface Options extends LoggingOptions {
labels?: {[key: string]: string};

// An attempt will be made to truncate messages larger than maxEntrySize.
// Please note that this parameter is ignored when redirectToStdout is set.
maxEntrySize?: number;

// A default global callback to be used for {@link LoggingWinston#log} when callback is
// not supplied by caller in function parameters
defaultCallback?: Callback;

/**
* Boolen flag that opts-in redirecting the output to STDOUT instead of ingesting logs to Cloud
* Logging using Logging API. Defaults to {@code false}. Redirecting logs can be used in
* Google Cloud environments with installed logging agent to delegate log ingestions to the
* agent. Redirected logs are formatted as one line Json string following the structured logging guidelines.
*/
redirectToStdout?: boolean;
}

/**
Expand Down
7 changes: 6 additions & 1 deletion src/middleware/express.ts
View file
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@
import {
HttpRequest,
Log,
LogSync,
middleware as commonMiddleware,
} from '@google-cloud/logging';
import {GCPEnv} from 'google-auth-library';
Expand Down Expand Up @@ -64,7 +65,11 @@ export async function makeMiddleware(
logger.add(transport);
}

const auth = transport.common.stackdriverLog.logging.auth;
const auth = (
transport.common.redirectToStdout
? (transport.common.cloudLog as LogSync)
: (transport.common.cloudLog as Log)
).logging.auth;
const [env, projectId] = await Promise.all([
auth.getEnv(),
auth.getProjectId(),
Expand Down

0 comments on commit 76135ca

Please sign in to comment.