From 84e9788e3fb30df32537649e605aaaea42537bed Mon Sep 17 00:00:00 2001 From: Matt Apperson Date: Tue, 20 Aug 2019 14:57:44 -0400 Subject: [PATCH] [Fleet] Add initial config libs and adapters (#43483) * Project init * initial client libs * add initial UI framework * move fleet to legacy * add readme * Add shell ingest plugin * fix typo * update paths for legacy API. cleanup linting in vscode * remove CRUFT * remove more CRUFT * [Ingest] cleanup names and files/paths to confirm to a standard format (#41773) * [Maps] Rename modules for clarity (#41608) * [Docs] Add simple phrase highlighting to Logs UI (#41610) * [Docs] Add simple phrase highlighting to Logs UI * Fix heading level * [DOCS] Timelion cleanup (#41381) * [Canvas] Removes doc links from backticks. (#41601) * Upgrade EUI to 12.4.0 (#41577) * eui 12.4.0 * styled-components ts conflict * update combobox test service to always open on open call * Revert "update combobox test service to always open on open call" This reverts commit 43074e60061afcaf5c87e56ae5782aed2a4b68dc. * scroll combobox into view * scroll before filter * Move CSP config default values to csp module (#41676) This gives us a little more control over the default values of these configurations to help ensure (though not guarantee) that any changes here can be audited by the security team. * Remove notifications plugin (#41674) The notifications functionality has been replaced by the features of the actions plugin. This notifications plugin was never actually used by end-user facing features of Kibana. * [Logs UI] Make column configurations reorderable (#41035) * [Logs UI] Make column configurations reorderable * Improve typing aand memoize callback * Guard against index bounds and rename reorderLogColumns * Fix useCallback memoization * Add functional test for reordering log columns * Use browser.keys instead of Key in functional test * [Maps] populate _id in tooltip (#41684) * [ML] Data Frames - search bar on list page (#41415) * add search and filter to df list table * add mode filter to list table * adds id + description search * type fix * ensure search syntax is valid * ensure types are correct * retain filter on refresh * fix progress bar jump * [DOCS] Changed Visual Builder to TSVB (#39539) * [DOCS] Changed Visual Builder to TSVB * Reorg of interface changes * Content reorg * Updated image * Added task content * Content conslidation * Final clean up * Comments from Gail * [DOCS] Adds missing Timelion link (#41709) * [Infra UI] Fix section mapping bug in node detail page (#41641) * [Infra UI] Fix section mapping bug in node detail page * Fixing filter to match TSVB * Adding an enum for the InfraMetricsQueryType * removing unnecessary change * Change id to InfraMetric to make less error prone * Fixing type in Metrics Explorer * [Infra UI] Add UI to customize Metrics Explorer chart style (#41022) * Add UI to customize Metrics Explorer chart style * Re-order chart options form * Adding chart options to TSVB link * Rename line series to series chart * Fixing chart context menu tests * Adding test for calculate domain * Ensure caclulateDomain returns numbers * fixing typo * Bump backport to 4.6.1 (#41720) * hide top nav menu in full screen mode of maps and dashboard (#41672) * hide top nav menu in full screen mode of maps and dashboard * Fixed dashboard full screen mode and added full screen mode test to maps * improve typing (#41563) * [Code] test colorize before load a file (#41645) * [Code] handle status when repo is not exists (#41643) fix can't switch to head on root path * [Code] implement filtree api by using isogit (#41558) * remove obsolete http secutiry settings (#41569) * core doesn't use Record for public API (#41448) * core contracts don't use unknown to support type assignment limitations of https://github.com/Microsoft/TypeScript/wiki/Breaking-Changes#-k-string-unknown--is-no-longer-a-wildcard-assignment-target * regenereate docs * remove type over-write * Narrow type of PluginDeps to an object (#40846) * Narrow type of PluginDeps to an object * re-generate docs * [APM] Fix "Show trace logs" link (#41570) * [APM] Fix "Show trace logs" link * Add type for infra link items; escape url.domain param for uptime link * Comment out flakey test steps (#41743) This comments out the problematic portions of the functional test, which caused #41717 to occur. * [ML] Fixes model memory limit for metricbeat system module jobs (#41747) * [i18n] fix i18nrc parsing (#41741) * check for translations in file * update template * Add ownership of new platform security plugin to security team. (#41748) * Clean up top nav \ search bar \ query bar directives (#41636) * Move timepicker (to be deprecated) into old kbn_top_nav * Deleted search-bar and query-bar directives! * moved search bar to kibana_react (it's a generic react component, not a service) * translations * Moved superDatePicker directive to kbn_top_nav (to be deprecated) Deleted unused react_component directives call-out and tool-bar-search-box * TS test fix * Delete relative options * [ML] Use NavigationMenu without angularjs wrapper. (#41343) Follow up to #40830 and #41054 in preparation for single metric viewer migration. The previous PR introduced the navigation menu as a React component. This PR moves dependencies down from the angularjs wrapper directive directly to the React component so the component can also be used stand-alone without the angularjs wrapper. For simple angularjs based HTML templates this stand-alone usage is also part of this PR. Unfortunately the PR turned out to be quite big due to: Most page react components had to be wrapped in another to allow the addition of thus leading to large diffs for the components. All component code inside the was not touched though. * [telemetry] Analytics Package (#41113) * kbn-analytics * kbn-analytics * expose provider * add logger * performance and stats reporters * finalize ui stats metric * functional tests * remove readme file for now * update readme * add types file into to tsconfigs * Update packages/kbn-analytics/src/report.ts Co-Authored-By: Josh Dover * fix typechecks * use enum instead of strings for metric types * getUiStatsReporter -> createUiStatsReporter * fix special typo in README * remove unused stop method * fix tests * default debug to false * use chrome.getInjected * add METRIC_TYPE to jest module mocks * mock create fn * handle enabled:false * init ui_metric in test setup env * regenerator runtime * transform-regenerator * update lock file * update babel configs * runtime dep * add regenerator * babel configs * use env-preset * merge conflicts * fix workpad telemetry tests * regeneratorRuntime attempt to fix number 30000 * env targets * remove module config * try again * try without regenerator * use kbn/babel-preset/webpack_preset only * runtime * just use typescript * update tsconfig * Caches trackers by app value for infra useTrackMetric hook * replace all occurences of placeholder in drilldown url template (#41673) * cleanup names and files/paths to confirm to a standard format * tack down WIP code * remove things not or not yet needed * Added Flexmonster Pivot Table to known plugins list (#41655) * Add Flexmonster Pivot Table to known plugins list * Update docs/plugins/known-plugins.asciidoc Co-Authored-By: Larry Gregory * Fix typo (#41705) * turn on filtering tests (#41202) * turn on filtering tests * run x-pack-firefoxSmoke ciGroup 40 times, run dashboard tests 20 times per job * Revert "run x-pack-firefoxSmoke ciGroup 40 times, run dashboard tests 20 times per job" This reverts commit 5ef02cc53ba5085c3f74431cb6ef20be2d876cde. * GoodBye Notifier (#41663) * Begin notifier removal * Remove remaining notifier traces * Remove dead translations * Remove Angular from config listener * Import angular-sanitize explicitly in map * Revert "lock es snapshot to avoid failing CI" (#41539) This reverts commit 4eca0f3383f5f7746f375c4f1559838a3bedb9df. * fix more types, define SO * [SIEM] - Fix Jest test errors and warnings (#41712) * Fixes #41787 (#41791) * [DOCS] Puts Spaces content on single page (#41536) * [DOCS] Puts Spaces content on single page * [DOCS] Incorporates review comments * [DOCS] Incorporated review comments * [SIEM] - Hosts and Network Tables from LoadMore to Paginated (#41532) * [DOCS] Updates Console doc (#41371) * [DOCS] Updates Console doc * [DOCS] Incorporates comments on Console docs * [DOCS] Updated Console images * Fixed unused variables. Added a few methods to the SO adapter Co-authored-by: Nicolas Chaulet * fix type * Revert "Merge branch 'master' of github.com:elastic/kibana into feature-fleet" This reverts commit 997490feadce9e246e5e3c2123570523e30455f8, reversing changes made to db5fc8fa3feac6f67a36cc33a3aef09b291ae789. * removed blank test file * Fix file path * add i18n * initial client libs * cleanup names and files/paths to confirm to a standard format * tack down WIP code * remove things not or not yet needed * fix more types, define SO * Fixed unused variables. Added a few methods to the SO adapter Co-authored-by: Nicolas Chaulet * fix type * removed blank test file * add config adapter (no tests yet) * progress with config lib * working! * tweaks * fix test * remove whitespace * remove isVersionGreater * remove CRUFT from a bad merge --- src/test_utils/kbn_server.ts | 7 +- x-pack/index.js | 2 + .../fleet/public/lib/compose/kibana.ts | 2 +- .../plugins/fleet/public/lib/framework.ts | 24 -- .../plugins/ingest/common/constants/plugin.ts | 2 +- .../ingest/common/utils/is_version_greater.ts | 41 ++++ x-pack/legacy/plugins/ingest/index.ts | 4 + .../configurations.contract.test.ts.snap | 93 ++++++++ .../adapters/configurations/adapter_types.ts | 6 +- .../libs/adapters/configurations/default.ts | 118 +++++++--- .../libs/adapters/configurations/memorized.ts | 218 ++++++++++++++++++ .../server/libs/adapters/framework/default.ts | 23 +- .../libs/adapters/framework/memorized.ts | 61 +++++ .../ingest/server/libs/compose/kibana.ts | 2 +- .../ingest/server/libs/configuration.ts | 173 +++++++++++++- .../libs/configurations.contract.test.ts | 68 ++++++ .../plugins/ingest/server/libs/framework.ts | 26 ++- .../legacy/plugins/ingest/server/mappings.ts | 67 ++++++ .../test_utils/jest/contract_tests/servers.ts | 14 +- 19 files changed, 877 insertions(+), 74 deletions(-) create mode 100644 x-pack/legacy/plugins/ingest/common/utils/is_version_greater.ts create mode 100644 x-pack/legacy/plugins/ingest/server/libs/__memorize_snapshots__/configurations.contract.test.ts.snap create mode 100644 x-pack/legacy/plugins/ingest/server/libs/adapters/configurations/memorized.ts create mode 100644 x-pack/legacy/plugins/ingest/server/libs/adapters/framework/memorized.ts create mode 100644 x-pack/legacy/plugins/ingest/server/libs/configurations.contract.test.ts create mode 100644 x-pack/legacy/plugins/ingest/server/mappings.ts diff --git a/src/test_utils/kbn_server.ts b/src/test_utils/kbn_server.ts index 706a8050a0adb5..0a36915f693c31 100644 --- a/src/test_utils/kbn_server.ts +++ b/src/test_utils/kbn_server.ts @@ -120,8 +120,11 @@ export function createRoot(settings = {}, cliArgs: Partial = {}) { * @param {Object} [settings={}] Any config overrides for this instance. * @returns {Root} */ -export function createRootWithCorePlugins(settings = {}) { - return createRootWithSettings(defaultsDeep({}, settings, DEFAULT_SETTINGS_WITH_CORE_PLUGINS)); +export function createRootWithCorePlugins(settings = {}, cliArgs: Partial = {}) { + return createRootWithSettings( + defaultsDeep({}, settings, DEFAULT_SETTINGS_WITH_CORE_PLUGINS), + cliArgs + ); } /** diff --git a/x-pack/index.js b/x-pack/index.js index 61dba4e40558b6..7f13ed26c34199 100644 --- a/x-pack/index.js +++ b/x-pack/index.js @@ -43,6 +43,7 @@ import { encryptedSavedObjects } from './legacy/plugins/encrypted_saved_objects' import { snapshotRestore } from './legacy/plugins/snapshot_restore'; import { actions } from './legacy/plugins/actions'; import { alerting } from './legacy/plugins/alerting'; +import { ingest } from './legacy/plugins/ingest'; import { advancedUiActions } from './legacy/plugins/advanced_ui_actions'; import { fleet } from './legacy/plugins/fleet'; @@ -87,6 +88,7 @@ module.exports = function (kibana) { snapshotRestore(kibana), actions(kibana), alerting(kibana), + ingest(kibana), advancedUiActions(kibana), fleet(kibana), ]; diff --git a/x-pack/legacy/plugins/fleet/public/lib/compose/kibana.ts b/x-pack/legacy/plugins/fleet/public/lib/compose/kibana.ts index 787d9469c1dd2d..6eb0087e2e5d18 100644 --- a/x-pack/legacy/plugins/fleet/public/lib/compose/kibana.ts +++ b/x-pack/legacy/plugins/fleet/public/lib/compose/kibana.ts @@ -12,7 +12,6 @@ import chrome from 'ui/chrome'; // @ts-ignore not typed yet import { management } from 'ui/management'; import routes from 'ui/routes'; -import { INDEX_NAMES } from '../../../common/constants/index_names'; import { RestAgentAdapter } from '../adapters/agent/rest_agent_adapter'; import { RestElasticsearchAdapter } from '../adapters/elasticsearch/rest'; import { KibanaFrameworkAdapter } from '../adapters/framework/kibana_framework_adapter'; @@ -22,6 +21,7 @@ import { ElasticsearchLib } from '../elasticsearch'; import { FrontendLibs } from '../types'; import { PLUGIN } from '../../../common/constants/plugin'; import { FrameworkLib } from '../framework'; +import { INDEX_NAMES } from '../../../common/constants'; // A super early spot in kibana loading that we can use to hook before most other things const onKibanaReady = chrome.dangerouslyGetActiveInjector; diff --git a/x-pack/legacy/plugins/fleet/public/lib/framework.ts b/x-pack/legacy/plugins/fleet/public/lib/framework.ts index e6ae33168384e9..ff07beaf558cca 100644 --- a/x-pack/legacy/plugins/fleet/public/lib/framework.ts +++ b/x-pack/legacy/plugins/fleet/public/lib/framework.ts @@ -31,30 +31,6 @@ export class FrameworkLib { ); } - public versionGreaterThen(version: string) { - const pa = this.adapter.version.split('.'); - const pb = version.split('.'); - for (let i = 0; i < 3; i++) { - const na = Number(pa[i]); - const nb = Number(pb[i]); - // version is greater - if (na > nb) { - return true; - } - // version is less then - if (nb > na) { - return false; - } - if (!isNaN(na) && isNaN(nb)) { - return true; - } - if (isNaN(na) && !isNaN(nb)) { - return false; - } - } - return true; - } - public currentUserHasOneOfRoles(roles: string[]) { // If the user has at least one of the roles requested, the returnd difference will be less // then the orig array size. difference only compares based on the left side arg diff --git a/x-pack/legacy/plugins/ingest/common/constants/plugin.ts b/x-pack/legacy/plugins/ingest/common/constants/plugin.ts index 48a93fed18229b..523d8fbe3ab0ba 100644 --- a/x-pack/legacy/plugins/ingest/common/constants/plugin.ts +++ b/x-pack/legacy/plugins/ingest/common/constants/plugin.ts @@ -5,6 +5,6 @@ */ export const PLUGIN = { - ID: 'ingest-data', + ID: 'ingest', }; export const CONFIG_PREFIX = 'xpack.ingest-do-not-disable'; diff --git a/x-pack/legacy/plugins/ingest/common/utils/is_version_greater.ts b/x-pack/legacy/plugins/ingest/common/utils/is_version_greater.ts new file mode 100644 index 00000000000000..f1b18b87f16da4 --- /dev/null +++ b/x-pack/legacy/plugins/ingest/common/utils/is_version_greater.ts @@ -0,0 +1,41 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +export function isVersionGreater(v1: string, v2: string): 1 | 0 | -1 { + const v1parts = v1.split('.'); + const v2parts = v2.split('.'); + + function isValidPart(x: string) { + return /^\d+$/.test(x); + } + + if (!v1parts.every(isValidPart) || !v2parts.every(isValidPart)) { + throw new Error('versions are not valid'); + } + + while (v1parts.length < v2parts.length) v1parts.push('0'); + while (v2parts.length < v1parts.length) v2parts.push('0'); + + for (let i = 0; i < v1parts.length; ++i) { + if (v2parts.length === i) { + return 1; + } + + if (v1parts[i] === v2parts[i]) { + continue; + } else if (v1parts[i] > v2parts[i]) { + return 1; + } else { + return -1; + } + } + + if (v1parts.length !== v2parts.length) { + return -1; + } + + return 0; +} diff --git a/x-pack/legacy/plugins/ingest/index.ts b/x-pack/legacy/plugins/ingest/index.ts index 5f6196b8c1da42..78ab35ff453bad 100644 --- a/x-pack/legacy/plugins/ingest/index.ts +++ b/x-pack/legacy/plugins/ingest/index.ts @@ -8,6 +8,7 @@ import { resolve } from 'path'; import { PLUGIN } from './common/constants'; import { CONFIG_PREFIX } from './common/constants/plugin'; import { initServerWithKibana } from './server/kibana.index'; +import { mappings } from './server/mappings'; export const config = Joi.object({ enabled: Joi.boolean().default(true), @@ -20,6 +21,9 @@ export function ingest(kibana: any) { publicDir: resolve(__dirname, 'public'), config: () => config, configPrefix: CONFIG_PREFIX, + uiExports: { + mappings, + }, init(server: any) { initServerWithKibana(server); }, diff --git a/x-pack/legacy/plugins/ingest/server/libs/__memorize_snapshots__/configurations.contract.test.ts.snap b/x-pack/legacy/plugins/ingest/server/libs/__memorize_snapshots__/configurations.contract.test.ts.snap new file mode 100644 index 00000000000000..f997674c5bf1d3 --- /dev/null +++ b/x-pack/legacy/plugins/ingest/server/libs/__memorize_snapshots__/configurations.contract.test.ts.snap @@ -0,0 +1,93 @@ + +exports['Configurations Lib create should create a new configuration - create - {"name":"test","description":"test description","output":"defaut","monitoring_enabled":true,"agent_version":"8.0.0","data_sources":[]} (2)'] = { + "results": { + "id": "8a8874b0-bd51-11e9-a21b-dbec3e1a8be1" + } +} + +exports['Configurations Lib create should create a new configuration - create - {"name":"test","description":"test description","output":"defaut","monitoring_enabled":true,"shared_id":"994528b0-887f-4c71-923e-4ffe5dd302e2","version":0,"agent_version":"8.0.0","data_sources":[]} (2)'] = { + "results": { + "id": "715d5cb0-bd53-11e9-bb4e-fb77f27555ca", + "shared_id": "994528b0-887f-4c71-923e-4ffe5dd302e2", + "version": 0 + } +} + +exports['Configurations Lib create should create a new configuration - create - {"name":"test","description":"test description","output":"defaut","monitoring_enabled":true,"shared_id":"997e6674-d072-475b-89d3-9b9202e0dd99","version":0,"agent_version":"8.0.0","data_sources":[]} (2)'] = { + "results": { + "id": "9cd167f0-c065-11e9-9b54-89c2396bf183", + "shared_id": "997e6674-d072-475b-89d3-9b9202e0dd99", + "version": 0 + } +} + +exports['Configurations Lib create should create a new configuration - create - {"name":"test","description":"test description","output":"defaut","monitoring_enabled":true,"version":0,"agent_version":"8.0.0","data_sources":[],"shared_id":"string"} (2)'] = { + "results": { + "id": "385d2130-c068-11e9-a90f-d9a51a8c04f8", + "shared_id": "de5a13b9-4b80-4983-8e6a-1619e3f97b9a", + "version": 0 + } +} + +exports['Configurations Lib create should create a new configuration - get info (1)'] = { + "results": { + "kibana": { + "version": "8.0.0" + }, + "license": { + "type": "trial", + "expired": false, + "expiry_date_in_millis": 1568580919209 + }, + "security": { + "enabled": true, + "available": true + }, + "watcher": { + "enabled": true, + "available": true + } + } +} + +exports['Configurations Lib create should create a new configuration - get info (2)'] = { + "results": { + "kibana": { + "version": "8.0.0" + }, + "license": { + "type": "trial", + "expired": false, + "expiry_date_in_millis": 1568240322629 + }, + "security": { + "enabled": true, + "available": true + }, + "watcher": { + "enabled": true, + "available": true + } + } +} + +exports['Configurations Lib create should create a new configuration - get info (3)'] = { + "results": { + "kibana": { + "version": "8.0.0" + }, + "license": { + "type": "trial", + "expired": false, + "expiry_date_in_millis": 1568240322629 + }, + "security": { + "enabled": true, + "available": true + }, + "watcher": { + "enabled": true, + "available": true + } + } +} diff --git a/x-pack/legacy/plugins/ingest/server/libs/adapters/configurations/adapter_types.ts b/x-pack/legacy/plugins/ingest/server/libs/adapters/configurations/adapter_types.ts index c29b4c142c8180..fe98b0a6e84a8b 100644 --- a/x-pack/legacy/plugins/ingest/server/libs/adapters/configurations/adapter_types.ts +++ b/x-pack/legacy/plugins/ingest/server/libs/adapters/configurations/adapter_types.ts @@ -10,7 +10,7 @@ export const RuntimeDatasourceInput = t.interface( { id: t.string, meta: t.union([t.undefined, t.string]), - config: t.string, + config_id: t.string, }, 'DatasourceInput' ); @@ -29,6 +29,8 @@ export const NewRuntimeConfigurationFile = t.interface( description: t.string, output: t.string, monitoring_enabled: t.boolean, + shared_id: t.string, + version: t.number, agent_version: t.string, data_sources: t.array(DataSource), }, @@ -51,7 +53,7 @@ const ExistingDocument = t.interface({ id: t.string, shared_id: t.string, version: t.number, - active: t.boolean, + status: t.union(['active', 'locked', 'inactive'].map(s => t.literal(s))), updated_at: t.string, created_by: t.union([t.undefined, t.string]), updated_on: t.string, diff --git a/x-pack/legacy/plugins/ingest/server/libs/adapters/configurations/default.ts b/x-pack/legacy/plugins/ingest/server/libs/adapters/configurations/default.ts index dbbdaa9471f346..a9c5563e7e1eb6 100644 --- a/x-pack/legacy/plugins/ingest/server/libs/adapters/configurations/default.ts +++ b/x-pack/legacy/plugins/ingest/server/libs/adapters/configurations/default.ts @@ -28,7 +28,7 @@ export class ConfigAdapter { } public async get(id: string): Promise { - const config = await this.so.get('configurations', id); + const config = await this.so.get('configurations', id); if (config.error) { throw new Error(config.error.message); @@ -44,11 +44,13 @@ export class ConfigAdapter { } } - public async list(): Promise { + public async list(page: number = 1, perPage: number = 25): Promise { const configs = await this.so.find({ type: 'configurations', search: '*', searchFields: ['shared_id'], + page, + perPage, }); const uniqConfigurationFile = configs.saved_objects .map(config => { @@ -73,15 +75,22 @@ export class ConfigAdapter { return [...uniqConfigurationFile.values()]; } - public async listVersions(sharedID: string, activeOnly = true): Promise { + public async listVersions( + sharedID: string, + activeOnly = true, + page: number = 1, + perPage: number = 25 + ): Promise { const configs = (await this.so.find({ type: 'configurations', search: sharedID, searchFields: ['shared_id'], + page, + perPage, })).saved_objects; if (!activeOnly) { - const backupConfigs = await this.so.find({ + const backupConfigs = await this.so.find({ type: 'backup_configurations', search: sharedID, searchFields: ['shared_id'], @@ -99,56 +108,109 @@ export class ConfigAdapter { } public async update( - sharedID: string, - fromVersion: number, + id: string, configuration: ConfigurationFile ): Promise<{ id: string; version: number }> { + const config = await this.so.update('configurations', id, configuration); + return { - id: 'fsdfsdf', - version: 0, + id: config.id, + version: config.attributes.version || 1, }; } - public async delete( - sharedID: string, - version?: number - ): Promise<{ success: boolean; error?: string }> { + public async delete(id: string): Promise<{ success: boolean }> { + await this.so.delete('configurations', id); return { success: true, }; } public async createBackup( - sharedID: string, - version?: number + configuration: BackupConfigurationFile ): Promise<{ success: boolean; id?: string; error?: string }> { + const newSo = await this.so.create( + 'configurations', + (configuration as any) as ConfigurationFile + ); + return { - success: true, - id: 'k3jh5lk3j4h5kljh43', + success: newSo.error ? false : true, + id: newSo.id, + error: newSo.error ? newSo.error.message : undefined, }; } - public async getBackup(sharedID: string, version?: number): Promise { - return {} as BackupConfigurationFile; + public async getBackup(id: string): Promise { + const config = await this.so.get('backup_configurations', id); + + if (config.error) { + throw new Error(config.error.message); + } + + if (!config.attributes) { + throw new Error(`No backup configuration found with ID of ${id}`); + } + if (RuntimeConfigurationFile.decode(config.attributes).isRight()) { + return config.attributes as BackupConfigurationFile; + } else { + throw new Error(`Invalid BackupConfigurationFile data. == ${config.attributes}`); + } } /** * Inputs sub-domain type */ - public async getInputsById(ids: string[]): Promise { - return [{} as DatasourceInput]; + public async getInputsById( + ids: string[], + page: number = 1, + perPage: number = 25 + ): Promise { + const inputs = await this.so.find({ + type: 'configurations', + search: ids.reduce((query, id, i) => { + if (i === ids.length - 1) { + return `${query} ${id}`; + } + return `${query} ${id} |`; + }, ''), + searchFields: ['id'], + perPage, + page, + }); + + return inputs.saved_objects.map(input => input.attributes); } - public async addInputs( - sharedID: string, - version: number, - dsUUID: string, - input: DatasourceInput - ): Promise { - return 'htkjerhtkwerhtkjehr'; + public async listInputsforConfiguration( + configurationId: string, + page: number = 1, + perPage: number = 25 + ): Promise { + const inputs = await this.so.find({ + type: 'configurations', + search: configurationId, + searchFields: ['config_id'], + perPage, + page, + }); + + return inputs.saved_objects.map(input => input.attributes); + } + + public async addInputs(inputs: DatasourceInput[]): Promise { + const newInputs = []; + for (const input of inputs) { + newInputs.push(await this.so.create('inputs', input)); + } + + return newInputs.map(input => input.attributes.id); } - public async deleteInputs(inputID: string[]): Promise<{ success: boolean; error?: string }> { + public async deleteInputs(inputIDs: string[]): Promise<{ success: boolean }> { + for (const id of inputIDs) { + await this.so.delete('inputs', id); + } return { success: true, }; diff --git a/x-pack/legacy/plugins/ingest/server/libs/adapters/configurations/memorized.ts b/x-pack/legacy/plugins/ingest/server/libs/adapters/configurations/memorized.ts new file mode 100644 index 00000000000000..b4d52fc3ae7f5e --- /dev/null +++ b/x-pack/legacy/plugins/ingest/server/libs/adapters/configurations/memorized.ts @@ -0,0 +1,218 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +import { memorize } from '@mattapperson/slapshot/lib/memorize'; +import { NewConfigurationFile } from './adapter_types'; +import { ConfigurationFile, DatasourceInput, BackupConfigurationFile } from './adapter_types'; +import { ConfigAdapter } from './default'; + +export class MemorizedConfigAdapter { + constructor(private readonly adapter?: ConfigAdapter) {} + + public async create( + configuration: NewConfigurationFile + ): Promise<{ id: string; shared_id: string; version: number }> { + const { shared_id, ...config } = configuration; + return await memorize( + `create - ${JSON.stringify({ ...config, shared_id: 'string' })}`, + async () => { + if (!this.adapter) { + throw new Error('An adapter must be provided when running tests online'); + } + return await this.adapter.create(configuration); + }, + { + pure: false, + } + ); + } + + public async get(id: string): Promise { + return await memorize( + `get - ${JSON.stringify(id)}`, + async () => { + if (!this.adapter) { + throw new Error('An adapter must be provided when running tests online'); + } + return await this.adapter.get(id); + }, + { + pure: false, + } + ); + } + + public async list(page: number = 1, perPage: number = 25): Promise { + return await memorize( + `list - ${JSON.stringify({ page, perPage })}`, + async () => { + if (!this.adapter) { + throw new Error('An adapter must be provided when running tests online'); + } + return await this.adapter.list(page, perPage); + }, + { + pure: false, + } + ); + } + + public async listVersions( + sharedID: string, + activeOnly = true, + page: number = 1, + perPage: number = 25 + ): Promise { + return await memorize( + `listVersions - ${JSON.stringify({ sharedID, activeOnly, page, perPage })}`, + async () => { + if (!this.adapter) { + throw new Error('An adapter must be provided when running tests online'); + } + return await this.adapter.listVersions(sharedID, activeOnly, page, perPage); + }, + { + pure: false, + } + ); + } + + public async update( + id: string, + configuration: ConfigurationFile + ): Promise<{ id: string; version: number }> { + return await memorize( + `update - ${JSON.stringify({ id, configuration })}`, + async () => { + if (!this.adapter) { + throw new Error('An adapter must be provided when running tests online'); + } + return await this.adapter.update(id, configuration); + }, + { + pure: false, + } + ); + } + + public async delete(id: string): Promise<{ success: boolean }> { + return await memorize( + `delete - ${JSON.stringify(id)}`, + async () => { + if (!this.adapter) { + throw new Error('An adapter must be provided when running tests online'); + } + return await this.adapter.delete(id); + }, + { + pure: false, + } + ); + } + + public async createBackup( + configuration: BackupConfigurationFile + ): Promise<{ success: boolean; id?: string; error?: string }> { + return await memorize( + `createBackup - ${JSON.stringify(configuration)}`, + async () => { + if (!this.adapter) { + throw new Error('An adapter must be provided when running tests online'); + } + return await this.adapter.createBackup(configuration); + }, + { + pure: false, + } + ); + } + + public async getBackup(id: string): Promise { + return await memorize( + `getBackup - ${JSON.stringify(id)}`, + async () => { + if (!this.adapter) { + throw new Error('An adapter must be provided when running tests online'); + } + return await this.adapter.getBackup(id); + }, + { + pure: false, + } + ); + } + + /** + * Inputs sub-domain type + */ + public async getInputsById( + ids: string[], + page: number = 1, + perPage: number = 25 + ): Promise { + return await memorize( + `getInputsById - ${JSON.stringify({ ids, page, perPage })}`, + async () => { + if (!this.adapter) { + throw new Error('An adapter must be provided when running tests online'); + } + return await this.adapter.getInputsById(ids, page, perPage); + }, + { + pure: false, + } + ); + } + + public async listInputsforConfiguration( + configurationId: string, + page: number = 1, + perPage: number = 25 + ): Promise { + return await memorize( + `listInputsforConfiguration - ${JSON.stringify({ configurationId, page, perPage })}`, + async () => { + if (!this.adapter) { + throw new Error('An adapter must be provided when running tests online'); + } + return await this.adapter.listInputsforConfiguration(configurationId, page, perPage); + }, + { + pure: false, + } + ); + } + + public async addInputs(inputs: DatasourceInput[]): Promise { + return await memorize( + `addInputs - ${JSON.stringify(inputs)}`, + async () => { + if (!this.adapter) { + throw new Error('An adapter must be provided when running tests online'); + } + return await this.adapter.addInputs(inputs); + }, + { + pure: false, + } + ); + } + + public async deleteInputs(inputIDs: string[]): Promise<{ success: boolean }> { + return await memorize( + `deleteInputs - ${JSON.stringify(inputIDs)}`, + async () => { + if (!this.adapter) { + throw new Error('An adapter must be provided when running tests online'); + } + return await this.adapter.deleteInputs(inputIDs); + }, + { + pure: false, + } + ); + } +} diff --git a/x-pack/legacy/plugins/ingest/server/libs/adapters/framework/default.ts b/x-pack/legacy/plugins/ingest/server/libs/adapters/framework/default.ts index 30d61e9fa27da2..c48ddd27720d54 100644 --- a/x-pack/legacy/plugins/ingest/server/libs/adapters/framework/default.ts +++ b/x-pack/legacy/plugins/ingest/server/libs/adapters/framework/default.ts @@ -29,9 +29,13 @@ export class BackendFrameworkAdapter { private readonly CONFIG_PREFIX?: string ) { const xpackMainPlugin = this.server.plugins.xpack_main; - const thisPlugin = this.server.plugins.ingest; + const thisPlugin = (this.server.plugins as any)[this.PLUGIN_ID]; - mirrorPluginStatus(xpackMainPlugin, thisPlugin); + if (thisPlugin) { + mirrorPluginStatus(xpackMainPlugin, thisPlugin); + } else { + throw new Error('Plugin is not initalized in Kibana'); + } xpackMainPlugin.status.on('green', () => { this.xpackInfoWasUpdatedHandler(xpackMainPlugin.info); @@ -52,12 +56,24 @@ export class BackendFrameworkAdapter { } } + public async waitForStack() { + return new Promise(resolve => { + this.on('xpack.status.green', () => { + resolve(); + }); + }); + } + public getSetting(settingPath: string) { return this.server.config().get(settingPath); } public log(text: string) { - this.server.log(text); + if (this.server) { + this.server.log(text); + } else { + console.log(text); // eslint-disable-line + } } public exposeMethod(name: string, method: () => any) { @@ -128,6 +144,7 @@ export class BackendFrameworkAdapter { `Error parsing xpack info in ${this.PLUGIN_ID}, ${PathReporter.report(assertData)[0]}` ); } + this.info = xpackInfoUnpacked; return { diff --git a/x-pack/legacy/plugins/ingest/server/libs/adapters/framework/memorized.ts b/x-pack/legacy/plugins/ingest/server/libs/adapters/framework/memorized.ts new file mode 100644 index 00000000000000..694add74c25d99 --- /dev/null +++ b/x-pack/legacy/plugins/ingest/server/libs/adapters/framework/memorized.ts @@ -0,0 +1,61 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +import { Request } from 'src/legacy/server/kbn_server'; +import Slapshot from '@mattapperson/slapshot'; +// @ts-ignore +import { mirrorPluginStatus } from '../../../../../../server/lib/mirror_plugin_status'; +import { internalUser, KibanaUser } from './adapter_types'; +import { BackendFrameworkAdapter } from './default'; + +export class MemorizedBackendFrameworkAdapter { + public readonly internalUser = internalUser; + + public get info() { + return Slapshot.memorize( + `get info`, + async () => { + if (!this.adapter) { + throw new Error('An adapter must be provided when running tests online'); + } + return this.adapter.info; + }, + { + pure: false, + } + ); + } + + constructor(private readonly adapter?: BackendFrameworkAdapter) {} + + public on(event: 'xpack.status.green' | 'elasticsearch.status.green', cb: () => void) { + setTimeout(() => { + cb(); + }, 5); + } + + public getSetting(settingPath: string) { + return Slapshot.memorize(`getSetting - ${JSON.stringify(settingPath)}`, () => { + if (!this.adapter) { + throw new Error('An adapter must be provided when running tests online'); + } + return this.adapter.getSetting(settingPath); + }); + } + + public log(text: string) {} + + public exposeMethod(name: string, method: () => any) {} + + public async getUser(request: Request): Promise { + return await Slapshot.memorize(`getUser - ${JSON.stringify(request)}`, async () => { + if (!this.adapter) { + throw new Error('An adapter must be provided when running tests online'); + } + return await this.adapter.getUser(request); + }); + } +} diff --git a/x-pack/legacy/plugins/ingest/server/libs/compose/kibana.ts b/x-pack/legacy/plugins/ingest/server/libs/compose/kibana.ts index 3b3bf058092bc7..6eb4338b243b0b 100644 --- a/x-pack/legacy/plugins/ingest/server/libs/compose/kibana.ts +++ b/x-pack/legacy/plugins/ingest/server/libs/compose/kibana.ts @@ -25,7 +25,7 @@ export function compose(server: KibanaLegacyServer): ServerLibs { const soDatabase = new SODatabaseAdapter(server.savedObjects, server.plugins.elasticsearch); const configAdapter = new ConfigAdapter(soDatabase); - const configuration = new ConfigurationLib(configAdapter); + const configuration = new ConfigurationLib(configAdapter, { framework }); const libs: ServerLibs = { configuration, diff --git a/x-pack/legacy/plugins/ingest/server/libs/configuration.ts b/x-pack/legacy/plugins/ingest/server/libs/configuration.ts index 6dad8c69c5b399..2d48f3a6a014ee 100644 --- a/x-pack/legacy/plugins/ingest/server/libs/configuration.ts +++ b/x-pack/legacy/plugins/ingest/server/libs/configuration.ts @@ -3,16 +3,185 @@ * or more contributor license agreements. Licensed under the Elastic License; * you may not use this file except in compliance with the Elastic License. */ +import { merge, omit } from 'lodash'; +import uuidv4 from 'uuid/v4'; +import uuid from 'uuid/v4'; import { ConfigAdapter } from './adapters/configurations/default'; +import { BackendFrameworkLib } from './framework'; +import { ConfigurationFile } from './adapters/configurations/adapter_types'; export class ConfigurationLib { - constructor(private readonly adapter: ConfigAdapter) {} + constructor( + private readonly adapter: ConfigAdapter, + private readonly libs: { + framework: BackendFrameworkLib; + } + ) {} + public async create(name: string, description?: string) { + const info = await this.libs.framework.info; + if (!info) { + throw new Error('Could not get version information about Kibana from xpack'); + } + + return await this.adapter.create({ + name, + description: description || '', + output: 'defaut', + monitoring_enabled: true, + shared_id: uuid(), + version: 0, + agent_version: info.kibana.version, + data_sources: [], + }); + } + + public async get(id: string): Promise { + const config = await this.adapter.get(id); + return config; + } + + public async list(page: number = 1, perPage: number = 25): Promise { + const configs = await this.adapter.list(page, perPage); + return configs; + } + + public async listVersions( + sharedID: string, + activeOnly = true, + page: number = 1, + perPage: number = 25 + ): Promise { + const configs = await this.adapter.listVersions(sharedID, activeOnly, page, perPage); + return configs; + } + + public async update( + id: string, + configuration: Partial<{ + name: string; + description: string; + output: string; + monitoring_enabled: boolean; + }> + ): Promise<{ id: string; version: number }> { + const invalidKeys = Object.keys(configuration).filter( + key => !['name', 'description', 'output', 'monitoring_enabled'].includes(key) + ); + + if (invalidKeys.length !== -1) { + throw new Error( + `Update was called with configuration paramaters that are not allowed: ${invalidKeys}` + ); + } + const oldConfig = await this.adapter.get(id); + + if (oldConfig.status === 'active') { + throw new Error( + `Config ${oldConfig.id} can not be updated becuase it is ${oldConfig.status}` + ); + } + + const newConfig = await this._update(oldConfig, configuration); + return newConfig; + } + + public async delete(id: string): Promise<{ success: boolean }> { + return await this.adapter.delete(id); + } + + public async createNewConfigFrom(configId: string) { + const { id, data_sources: dataSources, ...oldConfig } = await this.adapter.get(configId); + const newConfig = await this.adapter.create({ ...oldConfig, data_sources: [] }); + + const newDSs: ConfigurationFile['data_sources'] = []; + for (const ds of dataSources) { + // TODO page through vs one large query as this will break if there are more then 10k inputs + // a likely case for uptime + const oldInputs = await this.adapter.getInputsById(ds.inputs, 1, 10000); + const newInputs = await this.adapter.addInputs( + oldInputs.map(input => ({ + ...input, + id: uuidv4(), + config_id: newConfig.id, + })) + ); + + newDSs.push({ ...ds, uuid: uuidv4(), inputs: newInputs }); + } + + await this.adapter.update(newConfig.id, { + id: newConfig.id, + ...oldConfig, + data_sources: newDSs, + }); + // TODO fire events for fleet that update was made + } + + public async upgrade(configId: string, version: string) { + const { id, agent_version: agentVersion, ...oldConfig } = await this.adapter.get(configId); + const newConfig = await this.adapter.create({ ...oldConfig, agent_version: agentVersion }); + + // TODO: ensure new version is greater then old + // TODO: Ensure new version is a valid version number for agent + // TODO: ensure new version works with current ES version + + await this.adapter.update(newConfig.id, { + id: newConfig.id, + ...oldConfig, + agent_version: version, + }); + // TODO fire events for fleet that update was made + } + + public async finishUpdateFrom(configId: string) { + const oldConfig = await this.adapter.get(configId); + await this.adapter.update(configId, { + ...oldConfig, + status: 'inactive', + }); + } public async rollForward(id: string): Promise<{ id: string; version: number }> { - this.adapter.get(id); return { id: 'fsdfsdf', version: 0, }; } + + /** + * request* because in the future with an approval flow it will not directly make the change + */ + public async requestAddDataSource(id: string) { + const oldConfig = await this.adapter.get(id); + + if (oldConfig.status === 'active') { + throw new Error( + `Config ${oldConfig.id} can not be updated becuase it is ${oldConfig.status}` + ); + } + + // const newConfig = await this._update(oldConfig, configuration); + } + + /** + * request* because in the future with an approval flow it will not directly make the change + */ + public async requestDeleteDataSource() { + throw new Error('Not yet implamented'); + } + + public async listDataSources() { + throw new Error('Not yet implamented'); + } + + private async _update(oldConfig: ConfigurationFile, config: Partial) { + const newConfig = await this.adapter.create( + merge({}, omit(oldConfig, ['id']), config) + ); + + // TODO update oldConfig to set status to locked + // TODO fire events for fleet that update was made + + return newConfig; + } } diff --git a/x-pack/legacy/plugins/ingest/server/libs/configurations.contract.test.ts b/x-pack/legacy/plugins/ingest/server/libs/configurations.contract.test.ts new file mode 100644 index 00000000000000..14126dd755898a --- /dev/null +++ b/x-pack/legacy/plugins/ingest/server/libs/configurations.contract.test.ts @@ -0,0 +1,68 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +import { ConfigurationLib } from './configuration'; +import { callWhenOnline } from '@mattapperson/slapshot/lib/call_when_online'; +import { MemorizedConfigAdapter } from './adapters/configurations/memorized'; +import { ConfigAdapter } from './adapters/configurations/default'; +import { SODatabaseAdapter } from './adapters/so_database/default'; +import { BackendFrameworkLib } from './framework'; +import { MemorizedBackendFrameworkAdapter } from './adapters/framework/memorized'; +import { BackendFrameworkAdapter } from './adapters/framework/default'; +import { camelCase } from 'lodash'; +import { PLUGIN } from '../../common/constants'; +import { CONFIG_PREFIX } from '../../common/constants/plugin'; +import { createKibanaServer } from '../../../../../test_utils/jest/contract_tests/servers'; + +describe('Configurations Lib', () => { + let realConfigAdapter: ConfigAdapter; + let servers: any; + let lib: ConfigurationLib; + let realFrameworkAdapter: BackendFrameworkAdapter; + + beforeAll(async () => { + await callWhenOnline(async () => { + servers = await createKibanaServer({ + security: { enabled: true }, + }); + const soAdapter = new SODatabaseAdapter( + servers.kbnServer.savedObjects, + servers.kbnServer.plugins.elasticsearch + ); + realConfigAdapter = new ConfigAdapter(soAdapter); + realFrameworkAdapter = new BackendFrameworkAdapter( + camelCase(PLUGIN.ID), + servers.kbnServer, + CONFIG_PREFIX + ); + await realFrameworkAdapter.waitForStack(); + }); + + const memorizedConfigAdapter = new MemorizedConfigAdapter(realConfigAdapter) as ConfigAdapter; + const memorizedFrameworkAdapter = new MemorizedBackendFrameworkAdapter( + realFrameworkAdapter + ) as BackendFrameworkAdapter; + + const framework = new BackendFrameworkLib(memorizedFrameworkAdapter); + lib = new ConfigurationLib(memorizedConfigAdapter, { framework }); + }); + + afterAll(async () => { + if (servers) { + await servers.shutdown(); + } + }); + + describe('create', () => { + it('should create a new configuration', async () => { + const newConfig = await lib.create('test', 'test description'); + + expect(typeof newConfig.id).toBe('string'); + expect(typeof newConfig.shared_id).toBe('string'); + expect(typeof newConfig.version).toBe('number'); + }); + }); +}); diff --git a/x-pack/legacy/plugins/ingest/server/libs/framework.ts b/x-pack/legacy/plugins/ingest/server/libs/framework.ts index 7dadd94c9a2462..b4da8609ee5967 100644 --- a/x-pack/legacy/plugins/ingest/server/libs/framework.ts +++ b/x-pack/legacy/plugins/ingest/server/libs/framework.ts @@ -5,17 +5,30 @@ */ import { Request } from 'src/legacy/server/kbn_server'; +import { get } from 'lodash'; import { BackendFrameworkAdapter } from './adapters/framework/default'; +import { LicenseType } from '../../common/types/security'; export class BackendFrameworkLib { /** * Expired `null` happens when we have no xpack info */ - public license = { - type: this.adapter.info ? this.adapter.info.license.type : 'unknown', - expired: this.adapter.info ? this.adapter.info.license.expired : null, - }; - public securityIsEnabled = this.adapter.info ? this.adapter.info.security.enabled : false; + public get license() { + return { + type: get(this.adapter, 'info.license.type', 'oss'), + expired: get(this.adapter, 'info.license.expired', null), + }; + } + public get info() { + return this.adapter.info; + } + public get version() { + return get(this.adapter, 'info.kibana.version', null) as string | null; + } + public get securityIsEnabled() { + return get(this.adapter, 'info.security.enabled', false); + } + public log = this.adapter.log; public on = this.adapter.on.bind(this.adapter); public internalUser = this.adapter.internalUser; @@ -31,4 +44,7 @@ export class BackendFrameworkLib { public exposeMethod(name: string, method: () => any) { return this.adapter.exposeMethod(name, method); } + public async waitForStack() { + return await this.adapter.waitForStack(); + } } diff --git a/x-pack/legacy/plugins/ingest/server/mappings.ts b/x-pack/legacy/plugins/ingest/server/mappings.ts new file mode 100644 index 00000000000000..e7a878bfe167c8 --- /dev/null +++ b/x-pack/legacy/plugins/ingest/server/mappings.ts @@ -0,0 +1,67 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +export const mappings = { + configurations: { + properties: { + name: { + type: 'text', + }, + description: { + type: 'text', + }, + output: { + type: 'keyword', + }, + monitoring_enabled: { + type: 'boolean', + }, + agent_version: { + type: 'keyword', + }, + data_sources: { + properties: { + id: { + type: 'keyword', + }, + meta: { + type: 'keyword', + }, + config_id: { + type: 'keyword', + }, + config: { + type: 'keyword', + }, + }, + }, + id: { + type: 'keyword', + }, + shared_id: { + type: 'keyword', + }, + version: { + type: 'integer', + }, + status: { + type: 'keyword', + }, + updated_at: { + type: 'keyword', + }, + created_by: { + type: 'keyword', + }, + updated_on: { + type: 'keyword', + }, + updated_by: { + type: 'keyword', + }, + }, + }, +}; diff --git a/x-pack/test_utils/jest/contract_tests/servers.ts b/x-pack/test_utils/jest/contract_tests/servers.ts index 44bad3ea702cc1..8d85af5b4644f2 100644 --- a/x-pack/test_utils/jest/contract_tests/servers.ts +++ b/x-pack/test_utils/jest/contract_tests/servers.ts @@ -103,11 +103,15 @@ export async function createKibanaServer(xpackOption = {}) { // Allow kibana to start jest.setTimeout(120000); } - const root = kbnTestServer.createRootWithCorePlugins({ - elasticsearch: { ...getSharedESServer() }, - plugins: { paths: [PLUGIN_PATHS] }, - xpack: xpackOption, - }); + + const root = kbnTestServer.createRootWithCorePlugins( + { + elasticsearch: { ...getSharedESServer() }, + plugins: { paths: [PLUGIN_PATHS] }, + xpack: xpackOption, + }, + { oss: false } + ); await root.setup(); await root.start(); const { server } = (root as any).server.legacy.kbnServer;