Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

create kbn-legacy-logging package #77678

Merged
merged 25 commits into from
Nov 22, 2020
Merged
Show file tree
Hide file tree
Changes from 12 commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
3ad4c93
create kbn-legacy-logging package and start to move things
pgayvallet Sep 16, 2020
73122e5
Merge remote-tracking branch 'upstream/master' into kbn-xxx-legacy-lo…
pgayvallet Sep 17, 2020
87ca167
fix rotator tests
pgayvallet Sep 17, 2020
f6dd86c
fix logging system test mocks
pgayvallet Sep 17, 2020
ba67eb7
move logging format to the package
pgayvallet Sep 17, 2020
e2e1615
move logging setup to package
pgayvallet Sep 17, 2020
bd0c68d
adapt legacy logging server
pgayvallet Sep 17, 2020
76f4f27
remove usage of legacy config in the legacy logging server
pgayvallet Sep 17, 2020
e54b7d9
move legacy logging server to package
pgayvallet Sep 17, 2020
361deda
remove `??` syntax from package
pgayvallet Sep 17, 2020
5d7b70a
update generated doc
pgayvallet Sep 17, 2020
6bd16cf
Merge remote-tracking branch 'upstream/master' into kbn-xxx-legacy-lo…
pgayvallet Sep 22, 2020
8ddc262
Merge remote-tracking branch 'upstream/master' into kbn-xxx-legacy-lo…
pgayvallet Nov 19, 2020
f6c67de
fix a few things due to month old merge
pgayvallet Nov 19, 2020
8382a09
remove typings from project
pgayvallet Nov 19, 2020
56e3ea1
move reconfigureLogging to package
pgayvallet Nov 19, 2020
4b8a661
add basic README file
pgayvallet Nov 19, 2020
5f0471d
update generated doc
pgayvallet Nov 19, 2020
294a05a
remove old typings
pgayvallet Nov 19, 2020
f2766e5
add typing for legacy logging events
pgayvallet Nov 19, 2020
0dd3b23
remove `??` from packages
pgayvallet Nov 19, 2020
5a37913
Merge remote-tracking branch 'upstream/master' into kbn-xxx-legacy-lo…
pgayvallet Nov 20, 2020
a60d262
fix / improve event types usages
pgayvallet Nov 20, 2020
dbbe2e5
remove suffix from tsconfig
pgayvallet Nov 20, 2020
389dfe9
Merge remote-tracking branch 'upstream/master' into kbn-xxx-legacy-lo…
pgayvallet Nov 22, 2020
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -7,5 +7,5 @@
<b>Signature:</b>

```typescript
QueryStringInput: React.FC<Pick<Props, "prepend" | "query" | "size" | "className" | "placeholder" | "onChange" | "onBlur" | "onSubmit" | "isInvalid" | "indexPatterns" | "dataTestSubj" | "screenTitle" | "disableAutoFocus" | "persistedLog" | "bubbleSubmitEvent" | "languageSwitcherPopoverAnchorPosition" | "onChangeQueryInputFocus">>
QueryStringInput: React.FC<Pick<Props, "size" | "className" | "placeholder" | "onChange" | "onBlur" | "onSubmit" | "prepend" | "isInvalid" | "query" | "indexPatterns" | "dataTestSubj" | "screenTitle" | "disableAutoFocus" | "persistedLog" | "bubbleSubmitEvent" | "languageSwitcherPopoverAnchorPosition" | "onChangeQueryInputFocus">>
pgayvallet marked this conversation as resolved.
Show resolved Hide resolved
```
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
<b>Signature:</b>

```typescript
SearchBar: React.ComponentClass<Pick<Pick<SearchBarProps, "query" | "isLoading" | "filters" | "onRefresh" | "onRefreshChange" | "refreshInterval" | "intl" | "indexPatterns" | "dataTestSubj" | "timeHistory" | "customSubmitButton" | "screenTitle" | "showQueryBar" | "showQueryInput" | "showFilterBar" | "showDatePicker" | "showAutoRefreshOnly" | "isRefreshPaused" | "dateRangeFrom" | "dateRangeTo" | "showSaveQuery" | "savedQuery" | "onQueryChange" | "onQuerySubmit" | "onSaved" | "onSavedQueryUpdated" | "onClearSavedQuery" | "indicateNoData" | "onFiltersUpdated">, "query" | "isLoading" | "filters" | "onRefresh" | "onRefreshChange" | "refreshInterval" | "indexPatterns" | "dataTestSubj" | "timeHistory" | "customSubmitButton" | "screenTitle" | "showQueryBar" | "showQueryInput" | "showFilterBar" | "showDatePicker" | "showAutoRefreshOnly" | "isRefreshPaused" | "dateRangeFrom" | "dateRangeTo" | "showSaveQuery" | "savedQuery" | "onQueryChange" | "onQuerySubmit" | "onSaved" | "onSavedQueryUpdated" | "onClearSavedQuery" | "indicateNoData" | "onFiltersUpdated">, any> & {
WrappedComponent: React.ComponentType<Pick<SearchBarProps, "query" | "isLoading" | "filters" | "onRefresh" | "onRefreshChange" | "refreshInterval" | "intl" | "indexPatterns" | "dataTestSubj" | "timeHistory" | "customSubmitButton" | "screenTitle" | "showQueryBar" | "showQueryInput" | "showFilterBar" | "showDatePicker" | "showAutoRefreshOnly" | "isRefreshPaused" | "dateRangeFrom" | "dateRangeTo" | "showSaveQuery" | "savedQuery" | "onQueryChange" | "onQuerySubmit" | "onSaved" | "onSavedQueryUpdated" | "onClearSavedQuery" | "indicateNoData" | "onFiltersUpdated"> & ReactIntl.InjectedIntlProps>;
SearchBar: React.ComponentClass<Pick<Pick<SearchBarProps, "isLoading" | "query" | "filters" | "onRefresh" | "onRefreshChange" | "refreshInterval" | "intl" | "indexPatterns" | "dataTestSubj" | "timeHistory" | "customSubmitButton" | "screenTitle" | "showQueryBar" | "showQueryInput" | "showFilterBar" | "showDatePicker" | "showAutoRefreshOnly" | "isRefreshPaused" | "dateRangeFrom" | "dateRangeTo" | "showSaveQuery" | "savedQuery" | "onQueryChange" | "onQuerySubmit" | "onSaved" | "onSavedQueryUpdated" | "onClearSavedQuery" | "indicateNoData" | "onFiltersUpdated">, "isLoading" | "query" | "filters" | "onRefresh" | "onRefreshChange" | "refreshInterval" | "indexPatterns" | "dataTestSubj" | "timeHistory" | "customSubmitButton" | "screenTitle" | "showQueryBar" | "showQueryInput" | "showFilterBar" | "showDatePicker" | "showAutoRefreshOnly" | "isRefreshPaused" | "dateRangeFrom" | "dateRangeTo" | "showSaveQuery" | "savedQuery" | "onQueryChange" | "onQuerySubmit" | "onSaved" | "onSavedQueryUpdated" | "onClearSavedQuery" | "indicateNoData" | "onFiltersUpdated">, any> & {
WrappedComponent: React.ComponentType<Pick<SearchBarProps, "isLoading" | "query" | "filters" | "onRefresh" | "onRefreshChange" | "refreshInterval" | "intl" | "indexPatterns" | "dataTestSubj" | "timeHistory" | "customSubmitButton" | "screenTitle" | "showQueryBar" | "showQueryInput" | "showFilterBar" | "showDatePicker" | "showAutoRefreshOnly" | "isRefreshPaused" | "dateRangeFrom" | "dateRangeTo" | "showSaveQuery" | "savedQuery" | "onQueryChange" | "onQuerySubmit" | "onSaved" | "onSavedQueryUpdated" | "onClearSavedQuery" | "indicateNoData" | "onFiltersUpdated"> & ReactIntl.InjectedIntlProps>;
}
```
1 change: 1 addition & 0 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -132,6 +132,7 @@
"@kbn/config-schema": "1.0.0",
"@kbn/i18n": "1.0.0",
"@kbn/interpreter": "1.0.0",
"@kbn/legacy-logging": "1.0.0",
"@kbn/logging": "1.0.0",
"@kbn/pm": "1.0.0",
"@kbn/std": "1.0.0",
Expand Down
32 changes: 32 additions & 0 deletions packages/kbn-legacy-logging/package.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
{
"name": "@kbn/legacy-logging",
"version": "1.0.0",
"private": true,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you add a Readme that this package is a temporary workaround until we migrate logging to the KP?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was more planning to keep it there until we totally get rid of the legacy logging system in 8.0 tbh. It will be way easier to delete it with all the code being centralized in this package. Was that what you meant?

edit: oh, I guess you mean the server event logging part. But there is the same issue. We have to keep support of the legacy logging format, meaning that the module won't go away before 8.0.

"license": "Apache-2.0",
"main": "./target/index.js",
"scripts": {
"build": "tsc",
"kbn:bootstrap": "yarn build",
"kbn:watch": "yarn build --watch"
},
"dependencies": {
"@kbn/std": "1.0.0",
"hapi": "^17.5.3",
"@hapi/good-squeeze": "5.2.1",
"joi": "^13.5.2",
"@elastic/good": "8.1.1-kibana2",
"chokidar": "3.2.1",
"lodash": "^4.17.20",
"rxjs": "^6.5.5",
"json-stringify-safe": "5.0.1",
"chalk": "^2.4.2",
"query-string": "5.1.1",
"moment": "^2.24.0",
"moment-timezone": "^0.5.27"
},
"devDependencies": {
"typescript": "4.0.2",
"@types/hapi": "^17.0.18",
"@types/podium": "^1.0.0"
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -18,20 +18,25 @@
*/

import _ from 'lodash';
import { getLoggerStream } from './log_reporter';
import { getLogReporter } from './log_reporter';
import { LegacyLoggingConfig } from './schema';

export default function loggingConfiguration(config) {
const events = config.get('logging.events');
/**
* Returns the `@hapi/good` plugin configuration to be used for the legacy logging
* @param config
*/
export function getLoggingConfiguration(config: LegacyLoggingConfig, opsInterval: number) {
const events = config.events;

if (config.get('logging.silent')) {
if (config.silent) {
_.defaults(events, {});
} else if (config.get('logging.quiet')) {
} else if (config.quiet) {
_.defaults(events, {
log: ['listening', 'error', 'fatal'],
request: ['error'],
error: '*',
});
} else if (config.get('logging.verbose')) {
} else if (config.verbose) {
_.defaults(events, {
log: '*',
ops: '*',
Expand All @@ -47,24 +52,24 @@ export default function loggingConfiguration(config) {
});
}

const loggerStream = getLoggerStream({
const loggerStream = getLogReporter({
config: {
json: config.get('logging.json'),
dest: config.get('logging.dest'),
timezone: config.get('logging.timezone'),
json: config.json,
dest: config.dest,
timezone: config.timezone,

// I'm adding the default here because if you add another filter
// using the commandline it will remove authorization. I want users
// to have to explicitly set --logging.filter.authorization=none or
// --logging.filter.cookie=none to have it show up in the logs.
filter: _.defaults(config.get('logging.filter'), {
filter: _.defaults(config.filter, {
authorization: 'remove',
cookie: 'remove',
}),
},
events: _.transform(
events,
function (filtered, val, key) {
function (filtered: Record<string, string>, val: string, key: string) {
// provide a string compatible way to remove events
if (val !== '!') filtered[key] = val;
},
Expand All @@ -74,7 +79,7 @@ export default function loggingConfiguration(config) {

const options = {
ops: {
interval: config.get('ops.interval'),
interval: opsInterval,
},
includes: {
request: ['headers', 'payload'],
Expand Down
25 changes: 25 additions & 0 deletions packages/kbn-legacy-logging/src/index.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
/*
* Licensed to Elasticsearch B.V. under one or more contributor
* license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright
* ownership. Elasticsearch B.V. licenses this file to you under
* the Apache License, Version 2.0 (the "License"); you may
* not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/

export { LegacyLoggingConfig, legacyLoggingConfigSchema } from './schema';
export { attachMetaData } from './metadata';
export { setupLoggingRotate } from './rotate';
export { setupLogging } from './setup_logging';
export { getLoggingConfiguration } from './get_logging_config';
export { LegacyLoggingServer } from './legacy_logging_server';
Original file line number Diff line number Diff line change
Expand Up @@ -17,40 +17,47 @@
* under the License.
*/

jest.mock('../../../../legacy/server/config');
jest.mock('../../../../legacy/server/logging');
jest.mock('./setup_logging');

import { LogLevel } from '../../logging';
import { LegacyLoggingServer } from './legacy_logging_server';
import { LegacyLoggingServer, LogRecord } from './legacy_logging_server';

test('correctly forwards log records.', () => {
const loggingServer = new LegacyLoggingServer({ events: {} });
const onLogMock = jest.fn();
loggingServer.events.on('log', onLogMock);

const timestamp = 1554433221100;
const firstLogRecord = {
const firstLogRecord: LogRecord = {
timestamp: new Date(timestamp),
pid: 5355,
level: LogLevel.Info,
level: {
id: 'info',
value: 5,
},
context: 'some-context',
message: 'some-message',
};

const secondLogRecord = {
const secondLogRecord: LogRecord = {
timestamp: new Date(timestamp),
pid: 5355,
level: LogLevel.Error,
level: {
id: 'error',
value: 3,
},
context: 'some-context.sub-context',
message: 'some-message',
meta: { unknown: 2 },
error: new Error('some-error'),
};

const thirdLogRecord = {
const thirdLogRecord: LogRecord = {
timestamp: new Date(timestamp),
pid: 5355,
level: LogLevel.Trace,
level: {
id: 'trace',
value: 7,
},
context: 'some-context.sub-context',
message: 'some-message',
meta: { tags: ['important', 'tags'], unknown: 2 },
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -17,29 +17,40 @@
* under the License.
*/

import { ServerExtType } from 'hapi';
import { ServerExtType, Server } from 'hapi';
import Podium from 'podium';
// @ts-expect-error: implicit any for JS file
import { Config } from '../../../../legacy/server/config';
// @ts-expect-error: implicit any for JS file
import { setupLogging } from '../../../../legacy/server/logging';
import { LogLevel, LogRecord } from '../../logging';
import { LegacyVars } from '../../types';

export const metadataSymbol = Symbol('log message with metadata');
export function attachMetaData(message: string, metadata: LegacyVars = {}) {
return {
[metadataSymbol]: {
message,
metadata,
},
};
import { setupLogging } from './setup_logging';
import { attachMetaData } from './metadata';
import { legacyLoggingConfigSchema } from './schema';

// these LogXXX types are duplicated to avoid a cross dependency with the @kbn/logging package.
// typescript will error if they diverge at some point.
type LogLevelId = 'all' | 'fatal' | 'error' | 'warn' | 'info' | 'debug' | 'trace' | 'off';

interface LogLevel {
id: LogLevelId;
value: number;
}

export interface LogRecord {
timestamp: Date;
level: LogLevel;
context: string;
message: string;
error?: Error;
meta?: { [name: string]: any };
pid: number;
}

const isEmptyObject = (obj: object) => Object.keys(obj).length === 0;

function getDataToLog(error: Error | undefined, metadata: object, message: string) {
if (error) return error;
if (!isEmptyObject(metadata)) return attachMetaData(message, metadata);
if (error) {
return error;
}
if (!isEmptyObject(metadata)) {
return attachMetaData(message, metadata);
}
return message;
}

Expand All @@ -50,7 +61,7 @@ interface PluginRegisterParams {
options: PluginRegisterParams['options']
) => Promise<void>;
};
options: LegacyVars;
options: Record<string, any>;
}

/**
Expand Down Expand Up @@ -84,22 +95,19 @@ export class LegacyLoggingServer {

private onPostStopCallback?: () => void;

constructor(legacyLoggingConfig: Readonly<LegacyVars>) {
constructor(legacyLoggingConfig: any) {
// We set `ops.interval` to max allowed number and `ops` filter to value
// that doesn't exist to avoid logging of ops at all, if turned on it will be
// logged by the "legacy" Kibana.
const config = {
logging: {
...legacyLoggingConfig,
events: {
...legacyLoggingConfig.events,
ops: '__no-ops__',
},
const { value: loggingConfig } = legacyLoggingConfigSchema.validate({
...legacyLoggingConfig,
events: {
...legacyLoggingConfig.events,
ops: '__no-ops__',
},
ops: { interval: 2147483647 },
};
});

setupLogging(this, Config.withDefaultSchema(config));
setupLogging((this as unknown) as Server, loggingConfig, 2147483647);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So we expect LegacyLoggingServer won't be called when the package is imported from CLI scripts? How does the final dependency tree look like?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The only import of this package in core (soon in kbn/config instead) is from the legacy appender

constructor(legacyLoggingConfig: Readonly<LegacyVars>) {
this.loggingServer = new LegacyLoggingServer(legacyLoggingConfig);
}

The idea was to change the file to use a dynamic require in the constructor to only load the module when needed. The cli / packages that will use the config service would default to KP logging (probably just defaulting to console appender), that way, the legacy logging (and hapi stuff) is not imported.

Note that the main goal to not load the legacy logging system / hapi when importing the config service was for the apm script instrumentation, which is no longer concerned by the problem, as it will have its own loader to ensure we are not importing any module we want to instrument at all (#77855)

}

public register({ plugin: { register }, options }: PluginRegisterParams): Promise<void> {
Expand Down