Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ingest UI Pattern Review Screen #5892

Merged
merged 25 commits into from
Feb 4, 2016
Merged
Show file tree
Hide file tree
Changes from 15 commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
7dbef2f
Starting work on pattern review page
Bargs Jan 12, 2016
d4aff77
Type selection dropdown
Bargs Jan 12, 2016
27e402b
add index pattern input
Bargs Jan 12, 2016
e5ff5ce
Merge branch 'filebeatWizard' into ingest/patternReview
Bargs Jan 12, 2016
027e807
avoid overwriting pattern properties if they were previously set
Bargs Jan 12, 2016
da6183c
add timeFieldName form input and time based checkbox
Bargs Jan 13, 2016
5fcb4cb
Merge branch 'filebeatWizard' into ingest/patternReview
Bargs Jan 13, 2016
a47fc05
Detect geo_points and dates by looking at the pipeline
Bargs Jan 13, 2016
3bad81d
detect the type of array elements
Bargs Jan 13, 2016
9e4362b
Pick default date field and update date field list when user changes …
Bargs Jan 13, 2016
4df54be
lock type if we detect a geo_point
Bargs Jan 13, 2016
b94631b
Merge branch 'filebeatWizard' into ingest/patternReview
Bargs Jan 25, 2016
d13a6ca
Merge branch 'filebeatWizard' into ingest/patternReview
Bargs Jan 25, 2016
7ae9d3a
Some styling and layout
Bargs Jan 25, 2016
3f02157
Only freeze the field type on geo_point if we detected it based on th…
Bargs Jan 25, 2016
41e7303
Merge branch 'feature/ingest' into ingest/patternReview
Bargs Feb 2, 2016
3d94e1a
Access scope context via controller instead of directly accessing
Bargs Feb 2, 2016
d769ebf
Use const instead of var
Bargs Feb 2, 2016
11fe9aa
Minor updates based on feedback
Bargs Feb 2, 2016
caa7a0f
escape table content
Bargs Feb 4, 2016
6dd73b3
Fix sort by type
Bargs Feb 4, 2016
00ff3d7
Label and instructions for index pattern name field
Bargs Feb 4, 2016
0111071
prevent bouncing of field table when checking and unchecking 'time ba…
Bargs Feb 4, 2016
bd45e77
Require the user to fill in index pattern name before enabling save b…
Bargs Feb 4, 2016
ac0974b
Merge branch 'feature/ingest' into ingest/patternReview
Bargs Feb 4, 2016
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -1,7 +1,28 @@
<h2>Pattern review step</h2>
<div class="wizard-step-title">
<h3>Review the index pattern</h3>
Here we'll define how and where to store your parsed events. We've made some intellient guesses for you, but most
fields can be changed iw we got it wrong!

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

iw we got it wrong --> if we got it wrong?

</div>

<div>
Docs: {{sampleDocs}}
<div class="pattern-review">
<input ng-model="indexPattern.id" class="pattern-input"/>
<label>
<input ng-model="isTimeBased" type="checkbox"/>
time based
</label>
<label ng-if="isTimeBased" class="time-field-input">
Time Field
<select ng-model="indexPattern.timeFieldName" name="time_field_name">
<option ng-repeat="field in dateFields" value="{{field}}">
{{field}}
</option>
</select>
</label>
</div>

<button ng-click="indexPattern = {id: 'logstash-*', title: 'myFirstIndexPattern'}">Create an index pattern</button>
<div>
<paginated-table
columns="columns"
rows="rows">
</paginated-table>
</div>
Original file line number Diff line number Diff line change
@@ -1,5 +1,70 @@
var modules = require('ui/modules');
var template = require('plugins/kibana/settings/sections/indices/directives/pattern_review_step.html');
var _ = require('lodash');
var editFieldTypeHTML = require('plugins/kibana/settings/sections/indices/partials/_edit_field_type.html');

var testData = {
message: '11/24/2015 ip=1.1.1.1 bytes=1234',
clientip: '1.1.1.1',
bytes: 1234,
geoip: {
lat: 37.3894,
lon: 122.0819
},
location: {
lat: 37.3894,
lon: 122.0819
},
'@timestamp': '2015-11-24T00:00:00.000Z',
otherdate: '2015-11-24T00:00:00.000Z',
codes: [1, 2, 3, 4]
};

var testPipeline = [
{
grok: {
match_field: 'message',
match_pattern: 'foo'
}
},
{
geoip: {
source_field: 'ip'
}
},
{
geoip: {
source_field: 'ip',
target_field: 'location'
}
},
{
date: {
match_field: 'initialDate',
match_formats: ['dd/MM/yyyy hh:mm:ss']
}
},
{
date: {
match_field: 'initialDate',
match_formats: ['dd/MM/yyyy hh:mm:ss'],
target_field: 'otherdate'
}
}
];

function pickDefaultTimeFieldName(dateFields) {
if (_.isEmpty(dateFields)) {
return undefined;
}

let fieldName = dateFields[0];

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Might be cleaner to do

const fieldname = _.includes(dateFields, '@timestamp') ? '@timestamp' : dateFields[0];

?

if (_.includes(dateFields, '@timestamp')) {
fieldName = '@timestamp';
}

return fieldName;
}

modules.get('apps/settings')
.directive('patternReviewStep', function () {
Expand All @@ -9,6 +74,76 @@ modules.get('apps/settings')
sampleDocs: '=',
indexPattern: '=',
pipeline: '='
},
controller: function ($scope, Private) {
$scope.sampleDocs = testData;
$scope.pipeline = testPipeline;

if (_.isUndefined($scope.indexPattern)) {
$scope.indexPattern = {};
}

const knownFieldTypes = {};
$scope.dateFields = [];
$scope.pipeline.forEach(function (processor) {
if (processor.geoip) {
let field = processor.geoip.target_field || 'geoip';

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of declaring the variable field for every item, maybe cache it above?

let field;
$scope.pipeline.forEach(function (processor) {
  if (processor.geoip) {
    field = ...
  }
  if (processor.date) {
    field = ...
  }
});

knownFieldTypes[field] = 'geo_point';
}
if (processor.date) {
let field = processor.date.target_field || '@timestamp';
knownFieldTypes[field] = 'date';
$scope.dateFields.push(field);
}
});

_.defaults($scope.indexPattern, {
id: 'filebeat-*',
title: 'filebeat-*',
timeFieldName: pickDefaultTimeFieldName($scope.dateFields),
fields: _.map($scope.sampleDocs, (value, key) => {
let type = knownFieldTypes[key] || typeof value;
if (type === 'object' && _.isArray(value) && !_.isEmpty(value)) {
type = typeof value[0];
}
return {name: key, type: type};

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does the team use ES6 shorthand?

return {type, name: key};

})
});

$scope.isTimeBased = !!$scope.indexPattern.timeFieldName;

$scope.$watch('indexPattern.id', function (value) {
$scope.indexPattern.title = value;
});
$scope.$watch('isTimeBased', function (value) {
if (value) {
$scope.indexPattern.timeFieldName = pickDefaultTimeFieldName($scope.dateFields);
}
else {
delete $scope.indexPattern.timeFieldName;
}
});
$scope.$watch('indexPattern.fields', (fields) => {
$scope.dateFields = _.map(_.filter(fields, {type: 'date'}), 'name');
}, true);

$scope.columns = [
{title: 'Field'},
{title: 'Type'},
{title: 'Example', sortable: false}
];

$scope.rows = _.map($scope.indexPattern.fields, (field) => {
const sampleValue = $scope.sampleDocs[field.name];
return [
field.name,
{
markup: editFieldTypeHTML,
scope: _.assign($scope.$new(), {field: field, knownFieldTypes: knownFieldTypes})
},
typeof sampleValue === 'object' ? JSON.stringify(sampleValue) : sampleValue
];
});
}
};
});
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
<select ng-if="knownFieldTypes[field.name] !== 'geo_point'" name="field_type" ng-model="field.type">
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could geo_point be a string?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not 100% sure I understand the question, are you asking if a geo_point field could be of type string in the elasticsearch mappings?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, or on the flip side I see I can convert bytes to geo_point for example. How are the rules defined for which options are available?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So the geoip processor is a special case, because its target field must be classified as a geo_point type in the index template and index pattern. That's why I'm doing a special check here for any geoip processors and then preventing any changes to that target field's type.

Other than that, the user can change anything they want. For instance, they might have lat/lon coordinates in their data which they could create their own geo_point field out of without using the geoip processor. In that case they'd need to manually update the type of that field to geo_point.

<option value="string">string</option>
<option value="number">number</option>
<option value="boolean">boolean</option>
<option value="date">date</option>
<option value="geo_point">geo_point</option>
<option value="geo_shape">geo_shape</option>
<option value="ip">ip</option>
</select>

<span ng-if="knownFieldTypes[field.name] === 'geo_point'">
geo_point
</span>
15 changes: 15 additions & 0 deletions src/plugins/kibana/public/settings/styles/main.less
Original file line number Diff line number Diff line change
Expand Up @@ -204,3 +204,18 @@ kbn-settings-indices {
}
}
}

.wizard-step-title {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not worried about this yet, but eventually it may make these styles easier to edit if they're split into their own file

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, these will probably get moved out when I try to do some refactoring when creating additional wizards.

padding-bottom: 1em;
}

.pattern-review {
.time-field-input {
padding-left: 1em;
}

.pattern-input {
width: 300px;
}
}