Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Logs UI] Adapt log entry rate data visualisations #47558

Merged
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
81 commits
Select commit Hold shift + click to select a range
4adc91d
Amend outer page / panel structure
Kerry350 Oct 3, 2019
62c7a6b
Remove unused imports
Kerry350 Oct 3, 2019
9793f4c
Merge remote-tracking branch 'upstream/master' into 47201-adapt-log-e…
Kerry350 Oct 7, 2019
0cefd23
Add anomalies section and overall chart
Kerry350 Oct 7, 2019
b5f3bf8
Merge remote-tracking branch 'upstream/master' into 47201-adapt-log-e…
Kerry350 Oct 8, 2019
7524569
Add legend
Kerry350 Oct 8, 2019
4119a05
Rename dataSet to partition
weltenwort Oct 8, 2019
8d887a8
Add max bucket anomaly score
weltenwort Oct 8, 2019
bb157fd
Merge branch '47201-adapt-log-entry-rate-data-vis' of github.com:Kerr…
Kerry350 Oct 8, 2019
7811629
Add table / expanded row charts
Kerry350 Oct 8, 2019
ae059f0
Add bucket span text
Kerry350 Oct 8, 2019
a3820b7
Add stat section to expanded rows
Kerry350 Oct 8, 2019
1e8678d
Amend annotations on overall graph
Kerry350 Oct 8, 2019
427105d
Add rule to account for EuiFlexItem edge case
Kerry350 Oct 9, 2019
123084b
Move functions that handle derivations of data to a new file
Kerry350 Oct 9, 2019
737a25d
Tweak data points fetched
Kerry350 Oct 9, 2019
ded5117
Style bars in grey for anomalies charts
Kerry350 Oct 9, 2019
94fdb76
Add severity scoring to annotations
Kerry350 Oct 9, 2019
25d6e54
Fix default
Kerry350 Oct 9, 2019
7be35d2
Remove decimal places from anomaly score representations
Kerry350 Oct 9, 2019
a3d81da
Show all partitions and overall anomaly score in annotation tooltip f…
Kerry350 Oct 9, 2019
2f7855e
Handle 'unknown' to workaround lack of '' suuport in tables
Kerry350 Oct 10, 2019
78750b3
Add stats section to overall anomalies section
Kerry350 Oct 10, 2019
7e72fe4
Base x-domain off the series so that certain buckets aren't omitted
Kerry350 Oct 10, 2019
cb3d525
Tweak colours and DRY up annotation rendering
Kerry350 Oct 10, 2019
9db1834
Add sorting to table
Kerry350 Oct 10, 2019
a0908da
Merge remote-tracking branch 'upstream/master' into 47201-adapt-log-e…
Kerry350 Oct 10, 2019
fd26252
Add "number of logs" to API results and render in UI stats
Kerry350 Oct 10, 2019
87074d1
Fix api integration tests
weltenwort Oct 11, 2019
e453e07
Merge remote-tracking branch 'upstream/master' into 47201-adapt-log-e…
Kerry350 Oct 14, 2019
3e181b0
Use "pretty" numbers for "Number of logs" stats
Kerry350 Oct 14, 2019
bb01766
Add toolbar text
Kerry350 Oct 14, 2019
6100b51
Format all y axis values to 3 digits
Kerry350 Oct 14, 2019
36c7cb9
Remove "Overall anomaly score" and change all wording / calculations …
Kerry350 Oct 14, 2019
af7833c
Merge branch '47201-adapt-log-entry-rate-data-vis' of github.com:Kerr…
Kerry350 Oct 14, 2019
3031b69
Sort anomaly maximum scores for the overall chart tooltip
Kerry350 Oct 14, 2019
fbe41af
Remove unused translations
Kerry350 Oct 14, 2019
5f85146
Use white text with badge in toolbar
Kerry350 Oct 14, 2019
af01a97
Merge remote-tracking branch 'upstream/master' into 47201-adapt-log-e…
Kerry350 Oct 15, 2019
40dbb59
Amend key
Kerry350 Oct 15, 2019
8e215d3
Use Math.round and introduce a formatAnomalyScore helper function
Kerry350 Oct 15, 2019
19ef762
Update x-pack/legacy/plugins/infra/public/pages/logs/analysis/section…
Kerry350 Oct 15, 2019
32833e6
Update x-pack/legacy/plugins/infra/public/pages/logs/analysis/section…
Kerry350 Oct 15, 2019
e732f5d
Format y-axis of log entry rate chart the same as anomalies charts
Kerry350 Oct 15, 2019
1548e14
Update x-pack/legacy/plugins/infra/public/pages/logs/analysis/section…
Kerry350 Oct 15, 2019
b1a743a
Merge branch '47201-adapt-log-entry-rate-data-vis' of github.com:Kerr…
Kerry350 Oct 15, 2019
b76bf02
Remove grow prop
Kerry350 Oct 15, 2019
1cb4143
Update x-pack/legacy/plugins/infra/common/http_api/log_analysis/resul…
Kerry350 Oct 15, 2019
cadd37f
Update x-pack/legacy/plugins/infra/common/http_api/log_analysis/resul…
Kerry350 Oct 15, 2019
79c4e21
Update x-pack/legacy/plugins/infra/server/routes/log_analysis/results…
Kerry350 Oct 15, 2019
63aca0b
Update x-pack/legacy/plugins/infra/public/pages/logs/analysis/page_re…
Kerry350 Oct 15, 2019
7bd5708
Update x-pack/legacy/plugins/infra/public/pages/logs/analysis/section…
Kerry350 Oct 15, 2019
ee60082
Update x-pack/legacy/plugins/infra/public/pages/logs/analysis/section…
Kerry350 Oct 15, 2019
157ddd8
Update x-pack/legacy/plugins/infra/server/routes/log_analysis/results…
Kerry350 Oct 15, 2019
e4ed1d1
Update x-pack/legacy/plugins/infra/public/pages/logs/analysis/section…
Kerry350 Oct 15, 2019
eb0b34a
Update x-pack/legacy/plugins/infra/public/pages/logs/analysis/section…
Kerry350 Oct 15, 2019
a685a43
Update x-pack/legacy/plugins/infra/public/pages/logs/analysis/section…
Kerry350 Oct 15, 2019
ad530c7
Update x-pack/legacy/plugins/infra/public/pages/logs/analysis/section…
Kerry350 Oct 15, 2019
4c3f3f0
Update x-pack/legacy/plugins/infra/public/pages/logs/analysis/section…
Kerry350 Oct 15, 2019
ad81285
Update x-pack/legacy/plugins/infra/public/pages/logs/analysis/section…
Kerry350 Oct 15, 2019
0ce5bf6
Update x-pack/legacy/plugins/infra/public/pages/logs/analysis/section…
Kerry350 Oct 15, 2019
1c587f1
Update x-pack/legacy/plugins/infra/public/pages/logs/analysis/section…
Kerry350 Oct 15, 2019
8b7535c
Update x-pack/legacy/plugins/infra/public/pages/logs/analysis/section…
Kerry350 Oct 15, 2019
b52d9c1
Update x-pack/legacy/plugins/infra/public/pages/logs/analysis/section…
Kerry350 Oct 15, 2019
4bdfcfb
Update x-pack/legacy/plugins/infra/server/lib/log_analysis/log_analys…
Kerry350 Oct 15, 2019
a32f8e5
Update x-pack/legacy/plugins/infra/server/lib/log_analysis/log_analys…
Kerry350 Oct 15, 2019
8e31a6c
Update x-pack/legacy/plugins/infra/server/routes/log_analysis/results…
Kerry350 Oct 15, 2019
721b7f8
Update x-pack/legacy/plugins/infra/server/routes/log_analysis/results…
Kerry350 Oct 15, 2019
914790e
Merge branch '47201-adapt-log-entry-rate-data-vis' of github.com:Kerr…
Kerry350 Oct 15, 2019
a491bff
Change path
Kerry350 Oct 15, 2019
f7787bd
Amend property name
Kerry350 Oct 15, 2019
305cdf4
Use proper default value argument
Kerry350 Oct 15, 2019
7f468b4
Use Kibana dateFormat setting for toolbar formatting
Kerry350 Oct 15, 2019
7fa26ed
Change logic for calculating severity score categories
Kerry350 Oct 15, 2019
d01e5f1
Add missing translations
Kerry350 Oct 15, 2019
a1493ce
Add interface to top of file
Kerry350 Oct 15, 2019
8197183
Add no-wrap to tooltip
Kerry350 Oct 15, 2019
5bcad0d
Use more idomatic code
Kerry350 Oct 15, 2019
a5d6e45
Use static value for series styles
Kerry350 Oct 15, 2019
abae288
Merge remote-tracking branch 'upstream/master' into 47201-adapt-log-e…
Kerry350 Oct 15, 2019
1d29fb0
Merge branch 'master' into 47201-adapt-log-entry-rate-data-vis
elasticmachine Oct 15, 2019
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -37,22 +37,25 @@ export const logEntryRateAnomaly = rt.type({
typicalLogEntryRate: rt.number,
});

export const logEntryRateDataSetRT = rt.type({
export const logEntryRatePartitionRT = rt.type({
analysisBucketCount: rt.number,
anomalies: rt.array(logEntryRateAnomaly),
averageActualLogEntryRate: rt.number,
dataSetId: rt.string,
maximumAnomalyScore: rt.number,
numberOfLogEntries: rt.number,
partitionId: rt.string,
});

export const logEntryRateHistogramBucket = rt.type({
dataSets: rt.array(logEntryRateDataSetRT),
partitions: rt.array(logEntryRatePartitionRT),
startTime: rt.number,
});

export const getLogEntryRateSuccessReponsePayloadRT = rt.type({
data: rt.type({
bucketDuration: rt.number,
histogramBuckets: rt.array(logEntryRateHistogramBucket),
totalNumberOfLogEntries: rt.number,
}),
});

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -9,19 +9,19 @@ import {
EuiFlexGroup,
EuiFlexItem,
EuiPage,
EuiPageBody,
EuiPageContent,
EuiPageContentBody,
EuiPanel,
EuiSuperDatePicker,
EuiBadge,
EuiText,
} from '@elastic/eui';
import { i18n } from '@kbn/i18n';
import numeral from '@elastic/numeral';
Kerry350 marked this conversation as resolved.
Show resolved Hide resolved
import { FormattedMessage } from '@kbn/i18n/react';
import moment from 'moment';
import React, { useCallback, useMemo, useState } from 'react';

import euiStyled from '../../../../../../common/eui_styled_components';
import { TimeRange } from '../../../../common/http_api/shared/time_range';
import { bucketSpan } from '../../../../common/log_analysis';
import euiStyled from '../../../../../../common/eui_styled_components';
import { LoadingPage } from '../../../components/loading_page';
import {
StringTimeRange,
Expand All @@ -31,6 +31,8 @@ import {
import { useTrackPageview } from '../../../hooks/use_track_metric';
import { FirstUseCallout } from './first_use';
import { LogRateResults } from './sections/log_rate';
import { AnomaliesResults } from './sections/anomalies';
import { useKibanaUiSetting } from '../../../utils/use_kibana_ui_setting';

export const AnalysisResultsContent = ({
sourceId,
Expand All @@ -42,6 +44,8 @@ export const AnalysisResultsContent = ({
useTrackPageview({ app: 'infra_logs', path: 'analysis_results' });
useTrackPageview({ app: 'infra_logs', path: 'analysis_results', delay: 15000 });

const [dateFormat] = useKibanaUiSetting('dateFormat', 'MMMM D, YYYY h:mm A');

const {
timeRange: selectedTimeRange,
setTimeRange: setSelectedTimeRange,
Expand All @@ -56,13 +60,13 @@ export const AnalysisResultsContent = ({
const bucketDuration = useMemo(() => {
// This function takes the current time range in ms,
// works out the bucket interval we'd need to always
// display 200 data points, and then takes that new
// display 100 data points, and then takes that new
// value and works out the nearest multiple of
// 900000 (15 minutes) to it, so that we don't end up with
// jaggy bucket boundaries between the ML buckets and our
// aggregation buckets.
const msRange = moment(queryTimeRange.endTime).diff(moment(queryTimeRange.startTime));
const bucketIntervalInMs = msRange / 200;
const bucketIntervalInMs = msRange / 100;
const result = bucketSpan * Math.round(bucketIntervalInMs / bucketSpan);
const roundedResult = parseInt(Number(result).toFixed(0), 10);
return roundedResult < bucketSpan ? bucketSpan : roundedResult;
Expand Down Expand Up @@ -130,39 +134,71 @@ export const AnalysisResultsContent = ({
/>
) : (
<>
<EuiPage>
<EuiPanel paddingSize="l">
<EuiFlexGroup justifyContent="spaceBetween">
<EuiFlexItem></EuiFlexItem>
<EuiFlexItem grow={false}>
<EuiSuperDatePicker
start={selectedTimeRange.startTime}
end={selectedTimeRange.endTime}
onTimeChange={handleSelectedTimeRangeChange}
isPaused={autoRefresh.isPaused}
refreshInterval={autoRefresh.interval}
onRefreshChange={handleAutoRefreshChange}
onRefresh={handleQueryTimeRangeChange}
/>
</EuiFlexItem>
</EuiFlexGroup>
</EuiPanel>
</EuiPage>
<ExpandingPage>
<EuiPageBody>
<EuiPageContent>
<EuiPageContentBody>
<ResultsContentPage>
<EuiFlexGroup direction="column">
<EuiFlexItem grow={false}>
<EuiPanel paddingSize="l">
<EuiFlexGroup justifyContent="spaceBetween" alignItems="center">
<EuiFlexItem grow={false}>
{!isLoading && logEntryRate ? (
<EuiText size="s">
<FormattedMessage
id="xpack.infra.logs.analysis.logRateResultsToolbarText"
defaultMessage="Analyzed {numberOfLogs} log entries from {startTime} to {endTime}"
values={{
numberOfLogs: (
<EuiBadge color="primary">
<EuiText size="s" color="ghost">
{numeral(logEntryRate.totalNumberOfLogEntries).format('0.00a')}
</EuiText>
</EuiBadge>
),
startTime: (
<b>{moment(queryTimeRange.startTime).format(dateFormat)}</b>
),
endTime: <b>{moment(queryTimeRange.endTime).format(dateFormat)}</b>,
}}
/>
</EuiText>
) : null}
</EuiFlexItem>
<EuiFlexItem grow={false}>
<EuiSuperDatePicker
start={selectedTimeRange.startTime}
end={selectedTimeRange.endTime}
onTimeChange={handleSelectedTimeRangeChange}
isPaused={autoRefresh.isPaused}
refreshInterval={autoRefresh.interval}
onRefreshChange={handleAutoRefreshChange}
onRefresh={handleQueryTimeRangeChange}
/>
</EuiFlexItem>
</EuiFlexGroup>
</EuiPanel>
</EuiFlexItem>
<EuiFlexItem grow={false}>
<EuiPanel paddingSize="l">
{isFirstUse && !hasResults ? <FirstUseCallout /> : null}
<LogRateResults
isLoading={isLoading}
results={logEntryRate}
setTimeRange={handleChartTimeRangeChange}
timeRange={queryTimeRange}
/>
</EuiPageContentBody>
</EuiPageContent>
</EuiPageBody>
</ExpandingPage>
</EuiPanel>
</EuiFlexItem>
<EuiFlexItem grow={false}>
<EuiPanel paddingSize="l">
<AnomaliesResults
isLoading={isLoading}
results={logEntryRate}
setTimeRange={handleChartTimeRangeChange}
timeRange={queryTimeRange}
/>
</EuiPanel>
</EuiFlexItem>
</EuiFlexGroup>
</ResultsContentPage>
</>
)}
</>
Expand All @@ -183,6 +219,10 @@ const stringToNumericTimeRange = (timeRange: StringTimeRange): TimeRange => ({
).valueOf(),
});

const ExpandingPage = euiStyled(EuiPage)`
flex: 1 0 0%;
// This is needed due to the flex-basis: 100% !important; rule that
// kicks in on small screens via media queries breaking when using direction="column"
export const ResultsContentPage = euiStyled(EuiPage)`
.euiFlexGroup--responsive > .euiFlexItem {
flex-basis: auto !important;
}
`;
Original file line number Diff line number Diff line change
@@ -0,0 +1,148 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License;
* you may not use this file except in compliance with the Elastic License.
*/

import { RectAnnotationDatum, AnnotationId } from '@elastic/charts';
import {
Axis,
BarSeries,
Chart,
getAxisId,
getSpecId,
niceTimeFormatter,
Settings,
TooltipValue,
LIGHT_THEME,
DARK_THEME,
getAnnotationId,
RectAnnotation,
} from '@elastic/charts';
import numeral from '@elastic/numeral';
import { i18n } from '@kbn/i18n';
import moment from 'moment';
import React, { useCallback, useMemo } from 'react';

import { TimeRange } from '../../../../../../common/http_api/shared/time_range';
import { useKibanaUiSetting } from '../../../../../utils/use_kibana_ui_setting';
import { MLSeverityScoreCategories } from '../helpers/data_formatters';

export const AnomaliesChart: React.FunctionComponent<{
chartId: string;
setTimeRange: (timeRange: TimeRange) => void;
timeRange: TimeRange;
series: Array<{ time: number; value: number }>;
annotations: Record<MLSeverityScoreCategories, RectAnnotationDatum[]>;
renderAnnotationTooltip?: (details?: string) => JSX.Element;
}> = ({ chartId, series, annotations, setTimeRange, timeRange, renderAnnotationTooltip }) => {
const [dateFormat] = useKibanaUiSetting('dateFormat', 'Y-MM-DD HH:mm:ss.SSS');
const [isDarkMode] = useKibanaUiSetting('theme:darkMode');

const chartDateFormatter = useMemo(
() => niceTimeFormatter([timeRange.startTime, timeRange.endTime]),
[timeRange]
);

const logEntryRateSpecId = getSpecId('averageValues');

const tooltipProps = useMemo(
() => ({
headerFormatter: (tooltipData: TooltipValue) => moment(tooltipData.value).format(dateFormat),
}),
[dateFormat]
);

const handleBrushEnd = useCallback(
(startTime: number, endTime: number) => {
setTimeRange({
endTime,
startTime,
});
},
[setTimeRange]
);

return (
<div style={{ height: 160, width: '100%' }}>
<Chart className="log-entry-rate-chart">
<Axis
id={getAxisId('timestamp')}
position="bottom"
showOverlappingTicks
tickFormat={chartDateFormatter}
/>
<Axis
id={getAxisId('values')}
position="left"
tickFormat={value => numeral(value.toPrecision(3)).format('0[.][00]a')} // https://github.com/adamwdraper/Numeral-js/issues/194
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could memoizing that callback with useCallback be important for performance given that elastic-chart might call this many times otherwise?

/>
<BarSeries
id={logEntryRateSpecId}
name={i18n.translate('xpack.infra.logs.analysis.anomaliesSectionLineSeriesName', {
defaultMessage: 'Log entries per 15 minutes (avg)',
})}
xScaleType="time"
yScaleType="linear"
xAccessor={'time'}
yAccessors={['value']}
data={series}
barSeriesStyle={barSeriesStyle}
/>
{renderAnnotations(annotations, chartId, renderAnnotationTooltip)}
<Settings
onBrushEnd={handleBrushEnd}
tooltip={tooltipProps}
baseTheme={isDarkMode ? DARK_THEME : LIGHT_THEME}
/>
</Chart>
</div>
);
};

interface SeverityConfig {
annotationId: AnnotationId;
style: {
fill: string;
opacity: number;
};
}

const severityConfigs: Record<string, SeverityConfig> = {
warning: {
annotationId: getAnnotationId(`anomalies-warning`),
style: { fill: 'rgb(125, 180, 226)', opacity: 0.7 },
},
minor: {
annotationId: getAnnotationId(`anomalies-minor`),
style: { fill: 'rgb(255, 221, 0)', opacity: 0.7 },
},
major: {
annotationId: getAnnotationId(`anomalies-major`),
style: { fill: 'rgb(229, 113, 0)', opacity: 0.7 },
},
critical: {
annotationId: getAnnotationId(`anomalies-critical`),
style: { fill: 'rgb(228, 72, 72)', opacity: 0.7 },
},
};

const renderAnnotations = (
annotations: Record<MLSeverityScoreCategories, RectAnnotationDatum[]>,
chartId: string,
renderAnnotationTooltip?: (details?: string) => JSX.Element
) => {
return Object.entries(annotations).map((entry, index) => {
return (
<RectAnnotation
key={`${chartId}:${entry[0]}`}
dataValues={entry[1]}
annotationId={severityConfigs[entry[0]].annotationId}
style={severityConfigs[entry[0]].style}
renderTooltip={renderAnnotationTooltip}
/>
);
});
};

const barSeriesStyle = { rect: { fill: '#D3DAE6', opacity: 0.6 } }; // TODO: Acquire this from "theme" as euiColorLightShade
Loading