Update mappings for kibana index #9280

ycombinator · 2016-11-30T17:27:11Z

In elastic/elasticsearch#21670 backwards compatibility of mappings was removed. This included support for the string/not_analyzed mapping; the keyword mapping must be used instead. This PR makes this change in the mapping of the Kibana index (default: .kibana).

Note that this PR fixes the easy part of the problem :) The harder problem is to automatically, at runtime, upgrade the mapping for existing (i.e. pre-6.0) users of Kibana. This upgrade issue will be dealt with in #6409.

ycombinator · 2016-11-30T19:20:39Z

@lukasolson @LeeDr @tylersmalley this is ready for review now.

tylersmalley · 2016-11-30T19:39:29Z

@ycombinator, looks like there is a failing test: expected 'keyword' to equal 'string'

tylersmalley · 2016-11-30T20:20:03Z

src/ui/public/index_patterns/_index_pattern.js

@@ -38,14 +38,14 @@ export default function IndexPatternFactory(Private, Notifier, config, kbnIndex,
  });

  const mapping = mappingSetup.expandShorthand({
-    title: 'string',
-    timeFieldName: 'string',
+    title: 'keyword',


Are we sure we want these to be not-analyzed? We might want to use text in these cases.

From what I could tell the title here is the index pattern, for example, logstash-*. So I'm fairly certain this ought to be not analyzed.

I ended up changing this to text anyway. I explained the reason in f654def.

ycombinator · 2016-11-30T20:36:12Z

BTW, I'm getting this error now when I load up the index patterns page:

Error: [illegal_argument_exception] mapper [title] cannot be changed from type [keyword] to [text]

So there's more work to be done on this PR. Removing review label for now...

ycombinator · 2016-11-30T21:50:10Z

Strange, I don't see the error from my previous comment any more. Perhaps I was switching between branches or something. Marking this as ready for review again.

lukasolson · 2016-11-30T22:46:31Z

So, this seems to fix the problems with opening Kibana. But as soon as I choose an index pattern and try to go to discover, I get this error:

Error: [mapper_parsing_exception] No handler for type [string] declared on field [title]
    at respond (elasticsearch.angular.js:23653)
    at checkRespForFailure (elasticsearch.angular.js:23612)
    at elasticsearch.angular.js:299
    at processQueue (angular.js:14745)
    at angular.js:14761
    at Scope.$eval (angular.js:15989)
    at Scope.$digest (angular.js:15800)
    at Scope.$apply (angular.js:16097)
    at done (angular.js:10546)
    at completeRequest (angular.js:10744)

Here's the request causing the error:

PUT https://localhost:5601/mkn/elasticsearch/.kibana/_mapping/search
{
  "search": {
    "properties": {
      "title": {
        "type": "string"
      },
      "description": {
        "type": "string"
      },
      "hits": {
        "type": "integer"
      },
      "columns": {
        "type": "string"
      },
      "sort": {
        "type": "string"
      },
      "version": {
        "type": "integer"
      },
      "kibanaSavedObjectMeta": {
        "properties": {
          "searchSourceJSON": {
            "type": "string"
          }
        }
      }
    }
  }
}

Here's the response:

{
  "error": {
    "root_cause": [
      {
        "type": "mapper_parsing_exception",
        "reason": "No handler for type [string] declared on field [title]"
      }
    ],
    "type": "mapper_parsing_exception",
    "reason": "No handler for type [string] declared on field [title]"
  },
  "status": 400
}

lukasolson · 2016-11-30T22:51:23Z

Seems we might need to change it in these places as well:

src/core_plugins/kibana/public/dashboard/services/_saved_dashboard.js
src/core_plugins/kibana/public/discover/saved_searches/_saved_search.js
src/core_plugins/kibana/public/visualize/saved_visualizations/_saved_vis.js
src/core_plugins/timelion/public/services/_saved_sheet.js

lukasolson · 2016-11-30T22:56:40Z

Also possibly related:

src/core_plugins/kibana/server/lib/init_default_field_props.js
src/core_plugins/kibana/server/lib/mapping_overrides.js
src/core_plugins/kibana/server/lib/processors/convert/kibana_to_es_converter.js

ycombinator · 2016-12-01T09:19:16Z

@Bargs I could you use your help with the first two files listed in #9280 (comment):

The first file, init_default_field_props.js, doesn't seem to be referenced anywhere in the code. So I can't tell how it is being used and what impact any changes to it might have. Do you have any idea if it safe to just delete this file as it doesn't seem to be referenced from anywhere in the codebase?

The second file mapping_overrides.js only seems to be referenced from init_default_field_props.js (the first file). So would it be safe to delete this one as well as it is unreachable from any other part of the codebase?

ycombinator · 2016-12-01T09:23:58Z

@BigFunger Do you know if the string ingest type on this line needs to be changed to text or keyword? My guess is that it should be changed to text but I don't know enough about ingest types to be sure.

This needs to be text because other types in the Kibana index also have a field named "title", whose mapping needs to be text. We can't have the same field have different mappings in different types within the same index.

Bargs · 2016-12-01T18:39:22Z

@ycombinator both of those files can be deleted, they're leftovers from the add data stuff.

ycombinator · 2016-12-01T18:43:18Z

I ❤️ you @Bargs

ycombinator · 2016-12-02T00:36:00Z

Checked with @BigFunger about #9280 (comment). That file (src/core_plugins/kibana/server/lib/processors/convert/kibana_to_es_converter.js) isn't needed any more, along with a bunch of other files related to pipelines. He's removing them in #9321. Once that is done (and I've rebased this PR), it will be ready for review again.

ycombinator · 2016-12-02T15:31:14Z

Chatted with @BigFunger about the code that's being deleted in #9321. That code is currently not reachable so we don't need to wait for it to be deleted in order to make progress on this PR.

@lukasolson @tylersmalley Mind giving this another looksie?

tylersmalley · 2016-12-02T17:00:58Z

Looks like it's referenced from the tests.

Mocha Failure: Warning: Cannot find module '../init_default_field_props'� Use --force to continue.

lukasolson

Things seem to be functional and the changes LGTM.

tylersmalley

LGTM - also went through some smoke testing, especially around saved objects.

harobed · 2017-04-04T08:06:10Z

It is possible to backport this patch to Kibana 5.x ?

Update buildNum mapping from string/not_analyzed to keyword

f3ecd5d

ycombinator added the review label Nov 30, 2016

ycombinator assigned tylersmalley and LeeDr Nov 30, 2016

Change index-pattern type mappings from string -> keyword

197b841

ycombinator mentioned this pull request Nov 30, 2016

must switch kibana index from string to keyword #9276

Closed

ycombinator assigned lukasolson Nov 30, 2016

Updating unit test

513c734

tylersmalley reviewed Nov 30, 2016

View reviewed changes

ycombinator removed the review label Nov 30, 2016

ycombinator added the review label Nov 30, 2016

lukasolson added the v6.0.0 label Nov 30, 2016

ycombinator added 2 commits December 1, 2016 01:30

Updating string mappings for various saved object types

fcb70a4

Fixing mapping for index pattern title field

f654def

This needs to be text because other types in the Kibana index also have a field named "title", whose mapping needs to be text. We can't have the same field have different mappings in different types within the same index.

Removing unused files

cdcf47d

ycombinator removed the review label Dec 2, 2016

Removing test file because corresponding source file is gone now

008d7e5

lukasolson approved these changes Dec 2, 2016

View reviewed changes

tylersmalley approved these changes Dec 2, 2016

View reviewed changes

ycombinator merged commit d635c11 into elastic:master Dec 2, 2016

ycombinator deleted the bugfix/kibana-index-mapping branch December 2, 2016 18:54

ycombinator mentioned this pull request Dec 3, 2016

Change mapping of index-pattern fields to text #9353

Merged

jbudz mentioned this pull request Apr 3, 2017

ES 5.3.0 + Kibana 5.3.0 => The [string] field is deprecated, please use [text] or [keyword] instead on [title] #10993

Closed

jimgoodwin added release_note:roadmap Team:Operations Team label for Operations Team labels Jul 25, 2017

jimgoodwin added the v6.0.0-rc1 label Sep 28, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update mappings for kibana index #9280

Update mappings for kibana index #9280

ycombinator commented Nov 30, 2016

ycombinator commented Nov 30, 2016

tylersmalley commented Nov 30, 2016 •

edited

tylersmalley Nov 30, 2016

ycombinator Nov 30, 2016

ycombinator Dec 1, 2016

ycombinator commented Nov 30, 2016

ycombinator commented Nov 30, 2016

lukasolson commented Nov 30, 2016

lukasolson commented Nov 30, 2016

lukasolson commented Nov 30, 2016

ycombinator commented Dec 1, 2016

ycombinator commented Dec 1, 2016

Bargs commented Dec 1, 2016

ycombinator commented Dec 1, 2016

ycombinator commented Dec 2, 2016

ycombinator commented Dec 2, 2016

tylersmalley commented Dec 2, 2016

lukasolson left a comment

tylersmalley left a comment

harobed commented Apr 4, 2017

Update mappings for kibana index #9280

Update mappings for kibana index #9280

Conversation

ycombinator commented Nov 30, 2016

ycombinator commented Nov 30, 2016

tylersmalley commented Nov 30, 2016 • edited

tylersmalley Nov 30, 2016

Choose a reason for hiding this comment

ycombinator Nov 30, 2016

Choose a reason for hiding this comment

ycombinator Dec 1, 2016

Choose a reason for hiding this comment

ycombinator commented Nov 30, 2016

ycombinator commented Nov 30, 2016

lukasolson commented Nov 30, 2016

lukasolson commented Nov 30, 2016

lukasolson commented Nov 30, 2016

ycombinator commented Dec 1, 2016

ycombinator commented Dec 1, 2016

Bargs commented Dec 1, 2016

ycombinator commented Dec 1, 2016

ycombinator commented Dec 2, 2016

ycombinator commented Dec 2, 2016

tylersmalley commented Dec 2, 2016

lukasolson left a comment

Choose a reason for hiding this comment

tylersmalley left a comment

Choose a reason for hiding this comment

harobed commented Apr 4, 2017

tylersmalley commented Nov 30, 2016 •

edited