Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rollup support for TSVB #28762

Merged
merged 67 commits into from
Feb 26, 2019
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
67 commits
Select commit Hold shift + click to select a range
1e39db3
Added a feature of rollup search on the UI side
sulemanof Jan 10, 2019
2e57c94
Rollup Feature - initial commit
alexwizp Jan 14, 2019
f07357f
Revert "Added a feature of rollup search on the UI side"
sulemanof Jan 15, 2019
2b11d84
Remove the 'label' property from the search strategies
alexwizp Jan 15, 2019
934c1a4
Changed search by strategy from the last
sulemanof Jan 15, 2019
bffde5f
add single search request
alexwizp Jan 15, 2019
29eda73
rollup_search_strategy add base implementation of isViable method
alexwizp Jan 15, 2019
51a1507
rollup_search_strategy add base implementation of isViable method -fix
alexwizp Jan 15, 2019
cba722c
Changed requests due to search request type
sulemanof Jan 15, 2019
6e3cfd2
refactoring of import Base classes / remove '../../../../../../
alexwizp Jan 15, 2019
256e5b1
remove extra await
alexwizp Jan 15, 2019
880a095
rollup_search_strategy. Refactoring of isRollupJobExists method
alexwizp Jan 16, 2019
64f872e
remove question
alexwizp Jan 16, 2019
09e4b54
Add support of annotations and table data
sulemanof Jan 16, 2019
33c696b
skeleton for adding Search Strategy restrictions
alexwizp Jan 17, 2019
71454f2
Add rollup search capabilities
sulemanof Jan 18, 2019
bbda41f
apply search strategy for annotations request
alexwizp Jan 18, 2019
edf8627
set fields capabilities for rollup strategy
alexwizp Jan 18, 2019
cb1cfe7
add timezone, interval into SearchCapabilities
alexwizp Jan 21, 2019
9424155
Add fields from capabilities
sulemanof Jan 21, 2019
666c7f3
add timezone, interval into SearchCapabilities
alexwizp Jan 21, 2019
dfbf14e
fix default timezone
alexwizp Jan 21, 2019
b75e266
Merging of two Rollup Jobs was removed
alexwizp Jan 21, 2019
d996886
move getFieldsForWildcard to searchStrategy
alexwizp Jan 22, 2019
ba42e59
Fix TSVB search requests should have a timeout
sulemanof Jan 16, 2019
2541786
Add unit test
sulemanof Jan 16, 2019
0460e3a
apply getEsShardTimeout for annorations/get_request_params, series/g…
alexwizp Jan 22, 2019
e069f65
rename metrics -> tsvb
alexwizp Jan 22, 2019
5ef7a84
search_strategies_register refactoring: move 'add' method from class
alexwizp Jan 23, 2019
a50a67e
Add merge rollup capabilities with fields
sulemanof Jan 23, 2019
69ba70d
Add merge rollup capabilities with fields - small fixes
alexwizp Jan 23, 2019
c2cfb01
Add support of 'Everything' aggregation for Rollup Search
alexwizp Jan 24, 2019
3f5508e
Return back metrics plugin
sulemanof Jan 24, 2019
18e79ba
remove 'metrics' from the X-pack\rollup require
alexwizp Jan 25, 2019
b4290f7
Fix test cases
sulemanof Jan 28, 2019
2a855ed
fix broken test: fail: "apis InfraOps GraphQL Endpoints metrics shoul…
alexwizp Jan 28, 2019
705ee14
rollup search - split by terms is not working
alexwizp Jan 28, 2019
60cc12a
Add count metric
sulemanof Jan 29, 2019
eb3b3db
/get_bucket_size.js. Add support of 'auto' interval, Add support of g…
alexwizp Jan 29, 2019
cc3cda4
fix build_request_body test
alexwizp Jan 29, 2019
1540961
[Rollup] [Phase 1] Error handling - rollup search errors should be mo…
alexwizp Jan 30, 2019
2656503
[Rollup] [Phase 1] Table View - research the query to ES - sorting is…
alexwizp Jan 30, 2019
e9650ae
Merge #26006 into rollup
alexwizp Jan 31, 2019
712fe25
Add table view support
sulemanof Jan 31, 2019
be2ba07
fix broken build
alexwizp Jan 31, 2019
2db4672
fix broken build
alexwizp Feb 1, 2019
6c05813
Merge remote-tracking branch 'upstream/master' into rollup
alexwizp Feb 1, 2019
68e666d
[Rollup] [Phase 1] - write new tests (rollup_search_request, rollup_…
alexwizp Feb 1, 2019
751d0c0
[Rollup] [Phase 1] - write tests for rollup_search_capabilities
alexwizp Feb 1, 2019
1bd3076
Add test on default_search_capabilities, abstract_search_strategy, se…
sulemanof Feb 1, 2019
bb8bf24
Merge remote-tracking branch 'upstream/master' into rollup
alexwizp Feb 4, 2019
8d4d112
Add test cases for search_requests folder
sulemanof Feb 4, 2019
f3312eb
[Rollup] [Phase 1] - write tests for rollup_search_strategy
alexwizp Feb 4, 2019
86df57c
Merge remote-tracking branch 'upstream/master' into rollup
sulemanof Feb 7, 2019
97f5315
FIx broken build
sulemanof Feb 7, 2019
a3f0de8
Merge remote-tracking branch 'upstream/master' into rollup
alexwizp Feb 11, 2019
5cc93c6
Merge remote-tracking branch 'upstream/master' into rollup
sulemanof Feb 12, 2019
97b6f56
remove todo
alexwizp Feb 15, 2019
095ce67
Merge remote-tracking branch 'upstream/master' into rollup
alexwizp Feb 15, 2019
de4b27c
fix calculation of interval value for rollup search
alexwizp Feb 17, 2019
e3434da
add unit tests
alexwizp Feb 18, 2019
e4d92c9
Remove default exports
sulemanof Feb 18, 2019
06650f4
Merge remote-tracking branch 'upstream/master' into rollup
alexwizp Feb 19, 2019
789a67c
fix PR comments
alexwizp Feb 19, 2019
623b900
fix calendar intervals
alexwizp Feb 19, 2019
64cb9e9
Merge remote-tracking branch 'upstream/master' into rollup
alexwizp Feb 20, 2019
85b61fc
Merge branch 'master' into rollup
alexwizp Feb 26, 2019
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 5 additions & 5 deletions src/legacy/core_plugins/metrics/index.js
Original file line number Diff line number Diff line change
Expand Up @@ -21,14 +21,15 @@ import { resolve } from 'path';

import fieldsRoutes from './server/routes/fields';
import visDataRoutes from './server/routes/vis';
import { SearchStrategiesRegister } from './server/lib/search_strategies/search_strategies_register';

export default function (kibana) {
return new kibana.Plugin({
require: ['kibana', 'elasticsearch'],

uiExports: {
visTypes: [
'plugins/metrics/kbn_vis_types'
'plugins/metrics/kbn_vis_types',
],
styleSheetPaths: resolve(__dirname, 'public/index.scss'),
},
Expand All @@ -37,16 +38,15 @@ export default function (kibana) {
return Joi.object({
enabled: Joi.boolean().default(true),
chartResolution: Joi.number().default(150),
minimumBucketSize: Joi.number().default(10)
minimumBucketSize: Joi.number().default(10),
}).default();
},


init(server) {
fieldsRoutes(server);
visDataRoutes(server);
}


SearchStrategiesRegister.init(server);
},
});
}
31 changes: 16 additions & 15 deletions src/legacy/core_plugins/metrics/public/components/_error.scss
Original file line number Diff line number Diff line change
@@ -1,15 +1,16 @@
.tvbError__additional,
.tvbError__stack {
margin-top: $euiSizeS;
}

// EUITODO: Convert to EuiCodeBlock
.tvbError__stack {
padding: $euiSizeS;
background: $euiCodeBlockBackgroundColor;
color: $euiCodeBlockColor;
line-height: $euiLineHeight;
font-family: $euiCodeFontFamily;
font-weight: $euiFontWeightRegular;
white-space: pre-wrap;
}
.tvbError__title,
.tvbError__additional,
.tvbError__stack {
margin-top: $euiSizeS;
}

// EUITODO: Convert to EuiCodeBlock
.tvbError__stack {
padding: $euiSizeS;
background: $euiCodeBlockBackgroundColor;
color: $euiCodeBlockColor;
line-height: $euiLineHeight;
font-family: $euiCodeFontFamily;
font-weight: $euiFontWeightRegular;
white-space: pre-wrap;
}
8 changes: 7 additions & 1 deletion src/legacy/core_plugins/metrics/public/components/error.js
Original file line number Diff line number Diff line change
Expand Up @@ -23,17 +23,23 @@ import React from 'react';
import _ from 'lodash';
import { FormattedMessage } from '@kbn/i18n/react';

const guidPattern = /\[[[a-f\d-\\]{36}\]/g;

function ErrorComponent(props) {
const { error } = props;
let additionalInfo;
const type = _.get(error, 'error.caused_by.type');
const type = _.get(error, 'error.caused_by.type') || _.get(error, 'error.type');
let reason = _.get(error, 'error.caused_by.reason');
const title = _.get(error, 'error.caused_by.title');

if (!reason) {
reason = _.get(error, 'message');
}

if (['runtime_exception', 'illegal_argument_exception'].includes(type)) {
reason = _.get(error, 'error.reason').replace(guidPattern, ``);
}

if (type === 'script_exception') {
const scriptStack = _.get(error, 'error.caused_by.script_stack');
reason = _.get(error, 'error.caused_by.caused_by.reason');
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ import topN from './vis_types/top_n/vis';
import table from './vis_types/table/vis';
import gauge from './vis_types/gauge/vis';
import markdown from './vis_types/markdown/vis';
import Error from './error';
import ErrorComponent from './error';
import NoData from './no_data';

const types = {
Expand All @@ -46,7 +46,7 @@ function Visualization(props) {
if (error) {
return (
<div className={props.className}>
<Error error={error} />
<ErrorComponent error={error} />
</div>
);
}
Expand Down
13 changes: 8 additions & 5 deletions src/legacy/core_plugins/metrics/server/lib/get_fields.js
Original file line number Diff line number Diff line change
Expand Up @@ -16,13 +16,16 @@
* specific language governing permissions and limitations
* under the License.
*/

import { SearchStrategiesRegister } from './search_strategies/search_strategies_register';
import { uniq } from 'lodash';

export async function getFields(req) {
const { indexPatternsService } = req.pre;
const index = req.query.index || '*';
const resp = await indexPatternsService.getFieldsForWildcard({ pattern: index });
const fields = resp.filter(field => field.aggregatable);
const indexPattern = req.query.index || '*';
const { searchStrategy, capabilities } = await SearchStrategiesRegister.getViableStrategy(req, indexPattern);

const fields = (await searchStrategy
.getFieldsForWildcard(req, indexPattern, capabilities))
.filter(field => field.aggregatable);

return uniq(fields, field => field.name);
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,58 @@
/*
* Licensed to Elasticsearch B.V. under one or more contributor
* license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright
* ownership. Elasticsearch B.V. licenses this file to you under
* the Apache License, Version 2.0 (the "License"); you may
* not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/
import { convertIntervalToUnit, parseInterval } from '../vis_data/helpers/unit_to_seconds';

const getTimezoneFromRequest = request => {
return request.payload.timerange.timezone;
};

export class DefaultSearchCapabilities {
constructor(request, batchRequestsSupport, fieldsCapabilities = {}) {
this.request = request;
this.batchRequestsSupport = batchRequestsSupport;
this.fieldsCapabilities = fieldsCapabilities;
}

get defaultTimeInterval() {
return null;
}

get searchTimezone() {
return getTimezoneFromRequest(this.request);
}

parseInterval(interval) {
return parseInterval(interval);
}

convertIntervalToUnit(intervalString, unit) {
const parsedInterval = this.parseInterval(intervalString);

if (parsedInterval.unit !== unit) {
return convertIntervalToUnit(intervalString, unit);
}

return parsedInterval;
}

getValidTimeInterval(intervalString) {
// Default search capabilities doesn't have any restrictions for the interval string
return intervalString;
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,106 @@
/*
* Licensed to Elasticsearch B.V. under one or more contributor
* license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright
* ownership. Elasticsearch B.V. licenses this file to you under
* the Apache License, Version 2.0 (the "License"); you may
* not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/
import { DefaultSearchCapabilities } from './default_search_capabilities';

describe('DefaultSearchCapabilities', () => {
let defaultSearchCapabilities;
let batchRequestsSupport;
let req;

beforeEach(() => {
req = {};
batchRequestsSupport = true;
defaultSearchCapabilities = new DefaultSearchCapabilities(req, batchRequestsSupport);
});

test('should init default search capabilities', () => {
expect(defaultSearchCapabilities.request).toBe(req);
expect(defaultSearchCapabilities.batchRequestsSupport).toBe(batchRequestsSupport);
expect(defaultSearchCapabilities.fieldsCapabilities).toEqual({});
});

test('should return defaultTimeInterval', () => {
expect(defaultSearchCapabilities.defaultTimeInterval).toBe(null);
});

test('should return Search Timezone', () => {
defaultSearchCapabilities.request = {
payload: {
timerange: {
timezone: 'UTC'
}
}
};

expect(defaultSearchCapabilities.searchTimezone).toEqual('UTC');
});

test('should return a valid time interval', () => {
expect(defaultSearchCapabilities.getValidTimeInterval('20m')).toBe('20m');
});

test('should parse interval', () => {
expect(defaultSearchCapabilities.parseInterval('120s')).toEqual({
value: 120,
unit: 's'
});

expect(defaultSearchCapabilities.parseInterval('20m')).toEqual({
value: 20,
unit: 'm'
});

expect(defaultSearchCapabilities.parseInterval('1y')).toEqual({
value: 1,
unit: 'y'
});
});

test('should convert interval string into different unit', () => {
expect(defaultSearchCapabilities.convertIntervalToUnit('120s', 's')).toEqual({
value: 120,
unit: 's'
});

expect(defaultSearchCapabilities.convertIntervalToUnit('60m', 'h')).toEqual({
value: 1,
unit: 'h'
});

expect(defaultSearchCapabilities.convertIntervalToUnit('4w', 'M')).toEqual({
value: 1,
unit: 'M'
});

expect(defaultSearchCapabilities.convertIntervalToUnit('1y', 'w')).toEqual({
value: 48,
unit: 'w'
});

expect(defaultSearchCapabilities.convertIntervalToUnit('60s', 'm')).toEqual({
value: 1,
unit: 'm'
});

expect(defaultSearchCapabilities.convertIntervalToUnit('1s', 'ms')).toEqual({
value: 1000,
unit: 'ms'
});
});
});
Original file line number Diff line number Diff line change
Expand Up @@ -17,10 +17,4 @@
* under the License.
*/

export default [
'std_deviation',
'variance',
'sum_of_squares'
];


export { SearchStrategiesRegister } from './search_strategies_register';
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
/*
* Licensed to Elasticsearch B.V. under one or more contributor
* license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright
* ownership. Elasticsearch B.V. licenses this file to you under
* the Apache License, Version 2.0 (the "License"); you may
* not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/
import { AbstractSearchStrategy } from './strategies/abstract_search_strategy';
import { AbstractSearchRequest } from './searh_requests/abstract_request';
import { DefaultSearchStrategy } from './strategies/default_search_strategy';
import { DefaultSearchCapabilities } from './default_search_capabilities';

const strategies = [];

const addStrategy = searchStrategy => {
if (searchStrategy instanceof AbstractSearchStrategy) {
strategies.unshift(searchStrategy);
}
return strategies;
};

export class SearchStrategiesRegister {
static init(server) {
server.expose('AbstractSearchStrategy', AbstractSearchStrategy);
server.expose('AbstractSearchRequest', AbstractSearchRequest);
server.expose('DefaultSearchCapabilities', DefaultSearchCapabilities);
server.expose('addSearchStrategy', searchStrategy => addStrategy(searchStrategy));

addStrategy(new DefaultSearchStrategy(server));
}

static async getViableStrategy(req, indexPattern) {
for (const searchStrategy of strategies) {
const { isViable, capabilities } = await searchStrategy.checkForViability(req, indexPattern);

if (isViable) {
return {
searchStrategy,
capabilities,
};
}
}
}
}
Loading