Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ingest UI Pattern Review Screen #5892

Merged
merged 25 commits into from
Feb 4, 2016
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
7dbef2f
Starting work on pattern review page
Bargs Jan 12, 2016
d4aff77
Type selection dropdown
Bargs Jan 12, 2016
27e402b
add index pattern input
Bargs Jan 12, 2016
e5ff5ce
Merge branch 'filebeatWizard' into ingest/patternReview
Bargs Jan 12, 2016
027e807
avoid overwriting pattern properties if they were previously set
Bargs Jan 12, 2016
da6183c
add timeFieldName form input and time based checkbox
Bargs Jan 13, 2016
5fcb4cb
Merge branch 'filebeatWizard' into ingest/patternReview
Bargs Jan 13, 2016
a47fc05
Detect geo_points and dates by looking at the pipeline
Bargs Jan 13, 2016
3bad81d
detect the type of array elements
Bargs Jan 13, 2016
9e4362b
Pick default date field and update date field list when user changes …
Bargs Jan 13, 2016
4df54be
lock type if we detect a geo_point
Bargs Jan 13, 2016
b94631b
Merge branch 'filebeatWizard' into ingest/patternReview
Bargs Jan 25, 2016
d13a6ca
Merge branch 'filebeatWizard' into ingest/patternReview
Bargs Jan 25, 2016
7ae9d3a
Some styling and layout
Bargs Jan 25, 2016
3f02157
Only freeze the field type on geo_point if we detected it based on th…
Bargs Jan 25, 2016
41e7303
Merge branch 'feature/ingest' into ingest/patternReview
Bargs Feb 2, 2016
3d94e1a
Access scope context via controller instead of directly accessing
Bargs Feb 2, 2016
d769ebf
Use const instead of var
Bargs Feb 2, 2016
11fe9aa
Minor updates based on feedback
Bargs Feb 2, 2016
caa7a0f
escape table content
Bargs Feb 4, 2016
6dd73b3
Fix sort by type
Bargs Feb 4, 2016
00ff3d7
Label and instructions for index pattern name field
Bargs Feb 4, 2016
0111071
prevent bouncing of field table when checking and unchecking 'time ba…
Bargs Feb 4, 2016
bd45e77
Require the user to fill in index pattern name before enabling save b…
Bargs Feb 4, 2016
ac0974b
Merge branch 'feature/ingest' into ingest/patternReview
Bargs Feb 4, 2016
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -1,7 +1,32 @@
<h2>Pattern review step</h2>
<div class="wizard-step-title">
<h3>Review the index pattern</h3>
Here we'll define how and where to store your parsed events. We've made some intellient guesses for you, but most
fields can be changed if we got it wrong!
</div>

<div>
Docs: {{sampleDocs}}
<div class="pattern-review">
<div>
<label>Index name or pattern</label>
<kbn-info info="Patterns allow you to define dynamic index names using * as a wildcard. Example: filebeat-*"></kbn-info>
</div>
<input ng-model="reviewStep.indexPattern.id" class="pattern-input"/>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. thoughts on a label for this saying it's the index pattern name?
  2. should this be required before the save button is enabled?
  3. if 2, thoughts on making this a form and using angular's validation?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added a label, and disabled the save button if the pattern is empty. Spencer brought up the idea of using Angular's form validation as well, when reviewing the wizard shell PR. I think it's a good idea but I'm going to tackle it in another pull so that I can convert the entire wizard into a form, with each step being a sub-form, which will be a part of a larger effort to make the wizard itself more reusable.

<label>
<input ng-model="reviewStep.isTimeBased" type="checkbox"/>
time based
</label>
<label ng-if="reviewStep.isTimeBased" class="time-field-input">
Time Field
<select ng-model="reviewStep.indexPattern.timeFieldName" name="time_field_name">
<option ng-repeat="field in reviewStep.dateFields" value="{{field}}">
{{field}}
</option>
</select>
</label>
</div>

<button ng-click="indexPattern = {id: 'logstash-*', title: 'myFirstIndexPattern'}">Create an index pattern</button>
<div>
<paginated-table
columns="reviewStep.columns"
rows="reviewStep.rows">
</paginated-table>
</div>
Original file line number Diff line number Diff line change
@@ -1,5 +1,65 @@
var modules = require('ui/modules');
var template = require('plugins/kibana/settings/sections/indices/add_data_steps/pattern_review_step.html');
const modules = require('ui/modules');
const template = require('plugins/kibana/settings/sections/indices/add_data_steps/pattern_review_step.html');
const _ = require('lodash');
const editFieldTypeHTML = require('plugins/kibana/settings/sections/indices/partials/_edit_field_type.html');

const testData = {
message: '11/24/2015 ip=1.1.1.1 bytes=1234',
clientip: '1.1.1.1',
bytes: 1234,
geoip: {
lat: 37.3894,
lon: 122.0819
},
location: {
lat: 37.3894,
lon: 122.0819
},
'@timestamp': '2015-11-24T00:00:00.000Z',
otherdate: '2015-11-24T00:00:00.000Z',
codes: [1, 2, 3, 4]
};

const testPipeline = [
{
grok: {
match_field: 'message',
match_pattern: 'foo'
}
},
{
geoip: {
source_field: 'ip'
}
},
{
geoip: {
source_field: 'ip',
target_field: 'location'
}
},
{
date: {
match_field: 'initialDate',
match_formats: ['dd/MM/yyyy hh:mm:ss']
}
},
{
date: {
match_field: 'initialDate',
match_formats: ['dd/MM/yyyy hh:mm:ss'],
target_field: 'otherdate'
}
}
];

function pickDefaultTimeFieldName(dateFields) {
if (_.isEmpty(dateFields)) {
return undefined;
}

return _.includes(dateFields, '@timestamp') ? '@timestamp' : dateFields[0];
}

modules.get('apps/settings')
.directive('patternReviewStep', function () {
Expand All @@ -9,6 +69,83 @@ modules.get('apps/settings')
sampleDocs: '=',
indexPattern: '=',
pipeline: '='
},
controllerAs: 'reviewStep',
bindToController: true,
controller: function ($scope, Private) {
this.sampleDocs = testData;
this.pipeline = testPipeline;

if (_.isUndefined(this.indexPattern)) {
this.indexPattern = {};
}

const knownFieldTypes = {};
this.dateFields = [];
this.pipeline.forEach((processor) => {
if (processor.geoip) {
const field = processor.geoip.target_field || 'geoip';
knownFieldTypes[field] = 'geo_point';
}
else if (processor.date) {
const field = processor.date.target_field || '@timestamp';
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should there be any target_field conflict checking on the front end? do you know if this is handled on the server?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The ES ingest API simply overwrites the target_field if it already exists, so I don't think we have to worry about conflicts.

knownFieldTypes[field] = 'date';
this.dateFields.push(field);
}
});

_.defaults(this.indexPattern, {
id: 'filebeat-*',
title: 'filebeat-*',
timeFieldName: pickDefaultTimeFieldName(this.dateFields),
fields: _.map(this.sampleDocs, (value, key) => {
let type = knownFieldTypes[key] || typeof value;
if (type === 'object' && _.isArray(value) && !_.isEmpty(value)) {
type = typeof value[0];
}
return {name: key, type: type};
})
});

this.isTimeBased = !!this.indexPattern.timeFieldName;

$scope.$watch('reviewStep.indexPattern.id', (value) => {
this.indexPattern.title = value;
});
$scope.$watch('reviewStep.isTimeBased', (value) => {
if (value) {
this.indexPattern.timeFieldName = pickDefaultTimeFieldName(this.dateFields);
}
else {
delete this.indexPattern.timeFieldName;
}
});
$scope.$watch('reviewStep.indexPattern.fields', (fields) => {
this.dateFields = _.map(_.filter(fields, {type: 'date'}), 'name');
}, true);

const buildRows = () => {
this.rows = _.map(this.indexPattern.fields, (field) => {
const sampleValue = this.sampleDocs[field.name];
return [
_.escape(field.name),
{
markup: editFieldTypeHTML,
scope: _.assign($scope.$new(), {field: field, knownFieldTypes: knownFieldTypes, buildRows: buildRows}),
value: field.type
},
typeof sampleValue === 'object' ? _.escape(JSON.stringify(sampleValue)) : _.escape(sampleValue)
];
});
};

this.columns = [
{title: 'Field'},
{title: 'Type'},
{title: 'Example', sortable: false}
];

buildRows();
}
};
});
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,10 @@ <h1>Tail a File</h1>

<div class="nav-buttons">
<button ng-click="wizard.prevStep()">Prev</button>
<button ng-disabled="!wizard.stepResults.indexPattern" ng-click="wizard.nextStep()">Save</button>
<button ng-disabled="!wizard.stepResults.indexPattern || !wizard.stepResults.indexPattern.id"
ng-click="wizard.nextStep()">
Save
</button>
</div>
</div>

Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
<select ng-if="knownFieldTypes[field.name] !== 'geo_point'" name="field_type" ng-model="field.type" ng-change="buildRows()">
<option value="string">string</option>
<option value="number">number</option>
<option value="boolean">boolean</option>
<option value="date">date</option>
<option value="geo_point">geo_point</option>
<option value="geo_shape">geo_shape</option>
<option value="ip">ip</option>
</select>

<span ng-if="knownFieldTypes[field.name] === 'geo_point'">
geo_point
</span>
12 changes: 12 additions & 0 deletions src/plugins/kibana/public/settings/styles/main.less
Original file line number Diff line number Diff line change
Expand Up @@ -216,9 +216,21 @@ kbn-settings-indices {
padding-bottom: 1em;
}

.pattern-review {
.time-field-input {
padding-left: 1em;
margin-bottom: 0;
}

.pattern-input {
width: 300px;
}
}

.paste-samples {
textarea {
width: 100%;
height: 250px;
}
}