Exclusion of source fields (elastic#8920)

Backports PR elastic#7402 **Commit 1:** added source filtering * Original sha: e7204be * Authored by Stéphane Campinas <stephane.campinas@gmail.com> on 2016-01-12T10:08:35Z **Commit 2:** ditched the new 'retrieved fields' tab and added checkbox to exclude a field in the field control * Original sha: b228c66 * Authored by Stéphane Campinas <stephane.campinas@gmail.com> on 2016-01-13T14:44:38Z **Commit 3:** disable field exclusion checkbox if field is a metafield * Original sha: 0d06220 * Authored by Stéphane Campinas <stephane.campinas@gmail.com> on 2016-01-15T15:39:47Z **Commit 4:** [indexPattern] copy excluded field property when refreshing fields * Original sha: c3c1d48 * Authored by spalger <spalger@users.noreply.github.com> on 2016-01-21T16:01:48Z * Committed by Stéphane Campinas <stephane.campinas@gmail.com> on 2016-06-02T20:40:15Z **Commit 5:** [indexPattern/field] move isMetaField consideration into Field * Original sha: 5b0b00a * Authored by spalger <spalger@users.noreply.github.com> on 2016-01-21T16:22:52Z * Committed by Stéphane Campinas <stephane.campinas@gmail.com> on 2016-06-02T20:44:37Z **Commit 6:** [indexPattern/edit] invert the "retreived" column, for accuracy * Original sha: 1dee6f1 * Authored by spalger <spalger@users.noreply.github.com> on 2016-01-21T16:24:44Z * Committed by Stéphane Campinas <stephane.campinas@gmail.com> on 2016-06-02T20:47:21Z **Commit 7:** [indexPattern/field] touchup the field.exclude message * Original sha: ec0c334 * Authored by spalger <spalger@users.noreply.github.com> on 2016-01-21T16:25:24Z * Committed by Stéphane Campinas <stephane.campinas@gmail.com> on 2016-06-02T20:47:23Z **Commit 8:** Fix typo * Original sha: 062931a * Authored by Spencer <spalger@users.noreply.github.com> on 2016-01-21T19:33:14Z * Committed by Stéphane Campinas <stephane.campinas@gmail.com> on 2016-06-02T20:48:53Z **Commit 9:** [indexPattern] handle index patterns without fields * Original sha: 9d70208 * Authored by spalger <spalger@users.noreply.github.com> on 2016-01-22T18:27:45Z * Committed by Stéphane Campinas <stephane.campinas@gmail.com> on 2016-06-02T20:51:11Z **Commit 10:** [courier/searchSource] auto add source filter for index pattern * Original sha: 5a97b6b * Authored by spalger <spalger@users.noreply.github.com> on 2016-01-23T01:08:52Z * Committed by Stéphane Campinas <stephane.campinas@gmail.com> on 2016-06-02T21:04:09Z **Commit 11:** [docTable] remove irrelevant test about source filtering * Original sha: 8211990 * Authored by spalger <spalger@users.noreply.github.com> on 2016-02-12T08:17:56Z * Committed by Stéphane Campinas <stephane.campinas@gmail.com> on 2016-06-02T21:04:11Z **Commit 12:** [settings/indices] cleanup imports * Original sha: fa5b22d * Authored by spalger <spalger@users.noreply.github.com> on 2016-04-05T00:29:36Z * Committed by Stéphane Campinas <stephane.campinas@gmail.com> on 2016-06-02T21:06:11Z **Commit 13:** [settings/indexPattern/fields] add "field filters" tab * Original sha: 7819fa9 * Authored by spalger <spalger@users.noreply.github.com> on 2016-04-05T06:07:31Z * Committed by Stéphane Campinas <stephane.campinas@gmail.com> on 2016-06-02T21:11:20Z **Commit 14:** [imports] fix old testUtils import * Original sha: d4b9c11 * Authored by spalger <spalger@users.noreply.github.com> on 2016-04-05T06:08:46Z * Committed by Stéphane Campinas <stephane.campinas@gmail.com> on 2016-06-02T21:11:24Z **Commit 15:** [ui/fieldWildcard] add lib to match names based on field-style wildcards * Original sha: d75a63f * Authored by spalger <spalger@users.noreply.github.com> on 2016-04-05T08:02:01Z * Committed by Stéphane Campinas <stephane.campinas@gmail.com> on 2016-06-02T21:11:24Z **Commit 16:** [settings/fieldFilters] list filter matches, remove excluded fields from fieldata_fields * Original sha: 08f8fd7 * Authored by spalger <spalger@users.noreply.github.com> on 2016-04-05T08:59:39Z * Committed by Stéphane Campinas <stephane.campinas@gmail.com> on 2016-06-02T21:11:24Z **Commit 17:** [fieldWildcard] properly escape regexp control chars * Original sha: 385a062 * Authored by spalger <spalger@users.noreply.github.com> on 2016-04-05T09:13:21Z * Committed by Stéphane Campinas <stephane.campinas@gmail.com> on 2016-06-02T21:11:25Z **Commit 18:** [settings/indexPatterns] mark fields excluded if they match an exclude pattern * Original sha: da62b2a * Authored by spalger <spalger@users.noreply.github.com> on 2016-04-05T09:29:03Z * Committed by Stéphane Campinas <stephane.campinas@gmail.com> on 2016-06-02T21:11:25Z **Commit 19:** [fieldWildcard] properly bind the regexp to the ends * Original sha: e89ceea * Authored by spalger <spalger@users.noreply.github.com> on 2016-04-05T23:59:06Z * Committed by Stéphane Campinas <stephane.campinas@gmail.com> on 2016-06-02T21:11:25Z **Commit 20:** [indexPattern] remove unneeded lodash chain * Original sha: 14d5458 * Authored by spalger <spalger@users.noreply.github.com> on 2016-04-06T00:00:28Z * Committed by Stéphane Campinas <stephane.campinas@gmail.com> on 2016-06-02T21:11:25Z **Commit 21:** [settings/indices] use settings-indices- prefix for tab direcives * Original sha: dc53a10 * Authored by spalger <spalger@users.noreply.github.com> on 2016-04-06T00:03:45Z * Committed by Stéphane Campinas <stephane.campinas@gmail.com> on 2016-06-02T21:12:19Z **Commit 22:** corrected rebase on master * Original sha: 4be31b2 * Authored by Stéphane Campinas <stephane.campinas@gmail.com> on 2016-06-02T21:29:19Z **Commit 23:** Do not match exclusion on meta/scripted fields. Disable filter bar when on 'Filter fields' tab. Removed exclusion checkbox in the field controls page. Corrected typos. Improved documentation phrasing. * Original sha: 79e87ca * Authored by Stéphane Campinas <stephane.campinas@gmail.com> on 2016-06-09T08:14:12Z **Commit 24:** Merge branch 'master' into issue4366 * Original sha: 984d334 * Authored by Stéphane Campinas <stephane.campinas@gmail.com> on 2016-06-09T08:29:38Z **Commit 25:** corrected error in merge with _index_pattern * Original sha: 3375fd1 * Authored by Stéphane Campinas <stephane.campinas@gmail.com> on 2016-06-09T08:35:25Z **Commit 26:** removed unused code * Original sha: 6a0b331 * Authored by Stéphane Campinas <stephane.campinas@gmail.com> on 2016-06-09T08:56:56Z **Commit 27:** Merge branch 'master' into issue4366 * Original sha: 81a56f9 * Authored by Stéphane Campinas <stephane.campinas@gmail.com> on 2016-06-09T21:24:12Z **Commit 28:** added missing fieldFilters to test dumps * Original sha: c209ce3 * Authored by Stéphane Campinas <stephane.campinas@gmail.com> on 2016-06-11T09:14:10Z **Commit 29:** Merge branch 'master' into issue4366 * Original sha: 9511486 * Authored by Stéphane Campinas <stephane.campinas@gmail.com> on 2016-06-13T08:57:50Z **Commit 30:** added the management section refactoring from master * Original sha: 9e32bbc * Authored by Stéphane Campinas <stephane.campinas@gmail.com> on 2016-06-30T09:14:19Z
airow · Nov 1, 2016 · b3e0138 · b3e0138
1 parent 9eb4f5b
commit b3e0138
Show file tree

Hide file tree

Showing 32 changed files with 639 additions and 54 deletions.
diff --git a/src/core_plugins/kibana/public/doc/controllers/doc.js b/src/core_plugins/kibana/public/doc/controllers/doc.js
@@ -51,7 +51,7 @@ app.controller('doc', function ($scope, $route, es, timefilter) {
         }
       },
       stored_fields: computedFields.storedFields,
-      _source: computedFields._source,
+      _source: true,
       script_fields: computedFields.scriptFields,
       docvalue_fields: computedFields.docvalueFields
     }

diff --git a/src/core_plugins/kibana/public/management/index.js b/src/core_plugins/kibana/public/management/index.js
@@ -6,6 +6,7 @@ import 'ui/filters/start_from';
 import 'ui/field_editor';
 import 'plugins/kibana/management/sections/indices/_indexed_fields';
 import 'plugins/kibana/management/sections/indices/_scripted_fields';
+import 'plugins/kibana/management/sections/indices/source_filters/source_filters';
 import 'ui/directives/bread_crumbs';
 import uiRoutes from 'ui/routes';
 import uiModules from 'ui/modules';

diff --git a/src/core_plugins/kibana/public/management/sections/indices/_edit.html b/src/core_plugins/kibana/public/management/sections/indices/_edit.html
@@ -37,17 +37,30 @@
 
       <br />
 
+      <!-- tab list -->
       <ul class="nav nav-tabs">
-        <li class="kbn-management-tab" ng-class="{ active: state.tab === fieldType.index }" ng-repeat="fieldType in fieldTypes">
-          <a ng-click="changeTab(fieldType)">
-            {{ fieldType.title }}
-            <small>({{ fieldType.count }})</small>
+        <li class="kbn-management-tab" ng-class="{ active: state.tab === editSection.index }" ng-repeat="editSection in editSections">
+          <a ng-click="changeTab(editSection)">
+            {{ editSection.title }}
+            <small>({{ editSection.count }})</small>
           </a>
         </li>
       </ul>
 
-      <indexed-fields ng-show="state.tab == 'indexedFields'" class="fields indexed-fields"></indexed-fields>
-      <scripted-fields ng-show="state.tab == 'scriptedFields'" class="fields scripted-fields"></scripted-fields>
+      <!-- tabs -->
+      <indexed-fields
+        ng-show="state.tab == 'indexedFields'"
+        class="fields indexed-fields"
+      ></indexed-fields>
+      <scripted-fields
+        ng-show="state.tab == 'scriptedFields'"
+        class="fields scripted-fields"
+      ></scripted-fields>
+      <source-filters
+        ng-show="state.tab == 'sourceFilters'"
+        index-pattern="indexPattern"
+        class="fields source-filters"
+      ></source-filters>
 
     </div>
   </kbn-management-indices>

diff --git a/src/core_plugins/kibana/public/management/sections/indices/_edit.js b/src/core_plugins/kibana/public/management/sections/indices/_edit.js
@@ -1,10 +1,11 @@
 import _ from 'lodash';
 import 'plugins/kibana/management/sections/indices/_indexed_fields';
 import 'plugins/kibana/management/sections/indices/_scripted_fields';
+import 'plugins/kibana/management/sections/indices/source_filters/source_filters';
 import 'plugins/kibana/management/sections/indices/_index_header';
 import RefreshKibanaIndex from 'plugins/kibana/management/sections/indices/_refresh_kibana_index';
 import UrlProvider from 'ui/url';
-import IndicesFieldTypesProvider from 'plugins/kibana/management/sections/indices/_field_types';
+import IndicesEditSectionsProvider from 'plugins/kibana/management/sections/indices/_edit_sections';
 import uiRoutes from 'ui/routes';
 import uiModules from 'ui/modules';
 import editTemplate from 'plugins/kibana/management/sections/indices/_edit.html';
@@ -51,9 +52,8 @@ uiModules.get('apps/management')
   docTitle.change($scope.indexPattern.id);
   const otherIds = _.without($route.current.locals.indexPatternIds, $scope.indexPattern.id);
 
-  const fieldTypes = Private(IndicesFieldTypesProvider);
   $scope.$watch('indexPattern.fields', function () {
-    $scope.fieldTypes = fieldTypes($scope.indexPattern);
+    $scope.editSections = Private(IndicesEditSectionsProvider)($scope.indexPattern);
   });
 
   $scope.changeTab = function (obj) {
@@ -62,7 +62,7 @@ uiModules.get('apps/management')
   };
 
   $scope.$watch('state.tab', function (tab) {
-    if (!tab) $scope.changeTab($scope.fieldTypes[0]);
+    if (!tab) $scope.changeTab($scope.editSections[0]);
   });
 
   $scope.$watchCollection('indexPattern.fields', function () {

diff --git a/src/core_plugins/kibana/public/management/sections/indices/_edit_sections.js b/src/core_plugins/kibana/public/management/sections/indices/_edit_sections.js
@@ -0,0 +1,33 @@
+import _ from 'lodash';
+export default function GetFieldTypes() {
+
+  return function (indexPattern) {
+    const fieldCount = _.countBy(indexPattern.fields, function (field) {
+      return (field.scripted) ? 'scripted' : 'indexed';
+    });
+
+    _.defaults(fieldCount, {
+      indexed: 0,
+      scripted: 0,
+      sourceFilters: 0
+    });
+
+    return [
+      {
+        title: 'fields',
+        index: 'indexedFields',
+        count: fieldCount.indexed
+      },
+      {
+        title: 'scripted fields',
+        index: 'scriptedFields',
+        count: fieldCount.scripted
+      },
+      {
+        title: 'source filters',
+        index: 'sourceFilters',
+        count: fieldCount.sourceFilters
+      }
+    ];
+  };
+};
diff --git a/src/core_plugins/kibana/public/management/sections/indices/_field_types.js b/src/core_plugins/kibana/public/management/sections/indices/_field_types.js
diff --git a/src/core_plugins/kibana/public/management/sections/indices/_indexed_fields.js b/src/core_plugins/kibana/public/management/sections/indices/_indexed_fields.js
@@ -4,13 +4,15 @@ import nameHtml from 'plugins/kibana/management/sections/indices/_field_name.htm
 import typeHtml from 'plugins/kibana/management/sections/indices/_field_type.html';
 import controlsHtml from 'plugins/kibana/management/sections/indices/_field_controls.html';
 import uiModules from 'ui/modules';
+import FieldWildcardProvider from 'ui/field_wildcard';
 import indexedFieldsTemplate from 'plugins/kibana/management/sections/indices/_indexed_fields.html';
 
 uiModules.get('apps/management')
-.directive('indexedFields', function ($filter) {
+.directive('indexedFields', function (Private, $filter) {
   const yesTemplate = '<i class="fa fa-check" aria-label="yes"></i>';
   const noTemplate = '';
   const filter = $filter('filter');
+  const { fieldWildcardMatcher } = Private(FieldWildcardProvider);
 
   return {
     restrict: 'E',
@@ -26,6 +28,7 @@ uiModules.get('apps/management')
         { title: 'searchable', info: 'These fields can be used in the filter bar' },
         { title: 'aggregatable' , info: 'These fields can be used in visualization aggregations' },
         { title: 'analyzed', info: 'Analyzed fields may require extra memory to visualize' },
+        { title: 'excluded', info: 'Fields that are excluded from _source when it is fetched' },
         { title: 'controls', sortable: false }
       ];
 
@@ -34,14 +37,17 @@ uiModules.get('apps/management')
       function refreshRows() {
         // clear and destroy row scopes
         _.invoke(rowScopes.splice(0), '$destroy');
-
         const fields = filter($scope.indexPattern.getNonScriptedFields(), $scope.fieldFilter);
-        _.find($scope.fieldTypes, {index: 'indexedFields'}).count = fields.length; // Update the tab count
+        const sourceFilters = $scope.indexPattern.sourceFilters && $scope.indexPattern.sourceFilters.map(f => f.value) || [];
+        const fieldWildcardMatch = fieldWildcardMatcher(sourceFilters);
+        _.find($scope.editSections, {index: 'indexedFields'}).count = fields.length; // Update the tab count
 
         $scope.rows = fields.map(function (field) {
           const childScope = _.assign($scope.$new(), { field: field });
           rowScopes.push(childScope);
 
+          const excluded = fieldWildcardMatch(field.name);
+
           return [
             {
               markup: nameHtml,
@@ -66,6 +72,10 @@ uiModules.get('apps/management')
               markup: field.analyzed ? yesTemplate : noTemplate,
               value: field.analyzed
             },
+            {
+              markup: excluded ? yesTemplate : noTemplate,
+              value: excluded
+            },
             {
               markup: controlsHtml,
               scope: childScope

diff --git a/src/core_plugins/kibana/public/management/sections/indices/_scripted_fields.js b/src/core_plugins/kibana/public/management/sections/indices/_scripted_fields.js
@@ -38,7 +38,7 @@ uiModules.get('apps/management')
         rowScopes.length = 0;
 
         const fields = filter($scope.indexPattern.getScriptedFields(), $scope.fieldFilter);
-        _.find($scope.fieldTypes, {index: 'scriptedFields'}).count = fields.length; // Update the tab count
+        _.find($scope.editSections, {index: 'scriptedFields'}).count = fields.length; // Update the tab count
 
         $scope.rows = fields.map(function (field) {
           const rowScope = $scope.$new();

diff --git a/src/core_plugins/kibana/public/management/sections/indices/source_filters/controls.html b/src/core_plugins/kibana/public/management/sections/indices/source_filters/controls.html
@@ -0,0 +1,26 @@
+<div class="actions">
+  <button
+    aria-label="Edit source filter"
+    ng-if="sourceFilters.editing !== filter"
+    ng-click="sourceFilters.editing = filter"
+    type="button"
+    class="btn btn-xs btn-default">
+    <i aria-hidden="true" class="fa fa-pencil"></i>
+  </button>
+  <button
+    aria-label="Save source filter"
+    ng-if="sourceFilters.editing === filter"
+    ng-click="sourceFilters.save()"
+    ng-disabled="!filter.value"
+    type="button"
+    class="btn btn-xs btn-primary">
+    <i aria-hidden="true" class="fa fa-save"></i>
+  </button>
+  <button
+    aria-label="Delete source filter"
+    ng-click="sourceFilters.delete(filter)"
+    type="button"
+    class="btn btn-xs btn-danger">
+    <i aria-hidden="true" class="fa fa-trash"></i>
+  </button>
+</div>
diff --git a/src/core_plugins/kibana/public/management/sections/indices/source_filters/filter.html b/src/core_plugins/kibana/public/management/sections/indices/source_filters/filter.html
@@ -0,0 +1,12 @@
+<div class="value">
+  <span ng-if="sourceFilters.editing !== filter">{{ filter.value }}</span>
+
+  <input
+    ng-model="filter.value"
+    input-focus
+    ng-if="sourceFilters.editing === filter"
+    placeholder="{{ sourceFilters.placeHolder }}"
+    type="text"
+    required
+    class="form-control">
+</div>
diff --git a/...core_plugins/kibana/public/management/sections/indices/source_filters/source_filters.html b/...core_plugins/kibana/public/management/sections/indices/source_filters/source_filters.html
@@ -0,0 +1,37 @@
+<h3>Source Filters</h3>
+
+<p>
+  Source filters can be used to exclude one or more fields when fetching the document source. This happens when viewing a document in the Discover app, or with a table displaying results from a saved search in the Dashboard app. Each row is built using the source of a single document, and if you have documents with large or unimportant fields you may benefit from filtering those out at this lower level.
+</p>
+
+<p>
+  Note that multi-fields will incorrectly appear as matches in the table below. These filters only actually apply to fields in the original source document, so matching multi-fields are not actually being filtered.
+</p>
+
+<div ng-class="{ saving: sourceFilters.saving }" class="source-filters-container">
+
+  <form name="form" ng-submit="sourceFilters.create()">
+    <div class="input-group">
+      <input
+        ng-model="sourceFilters.newValue"
+        placeholder="{{ sourceFilters.placeHolder }}"
+        type="text"
+        class="form-control">
+
+      <div class="input-group-btn" role="group" aria-label="Source Filter Editor Controls">
+        <button
+          type="submit"
+          class="btn btn-primary"
+          ng-disabled="!sourceFilters.newValue">
+          Add
+        </button>
+      </div>
+    </div>
+  </form>
+
+  <paginated-table
+    columns="columns"
+    rows="rows"
+    per-page="perPage">
+  </paginated-table>
+</div>