[utils] add util for converting between es and kibana types #11967

spalger · 2017-05-23T00:24:14Z

In order to share the same es<->kibana type conversion rules on the server and client this:

moves the _cast_mapping_type module from the ui/index_patterns into src/utils
exposes functions for things like getEsTypes(), rather than port IndexedArray to the server
merges logic from src/ui/public/index_patterns/_field_types.js

Bargs · 2017-05-23T17:41:10Z

src/utils/kbn_field_types.js

+    this.sortable = !!sortable;
+    this.filterable = !!filterable;
+    this.esTypes = Object.freeze(esTypes || []);
+    Object.freeze(this);


Calling Object.freeze in a constructor probably isn't a great pattern to get into the habit of using. If another class wanted to extend this one the freeze would prevent it from doing so. I suppose it's not particularly important in this instance though, it's unlikely anyone will want to extend this class.

Freezing the field type instances is basically the whole reason this constructor exists, and it's only exported so it can be used in the tests. If subclassing was something we wanted to do then extensions would have to comply with the limitation (attach getters/setters/methods to the prototype) or update the KbnFieldType class. I'm fine with that.

I was just playing around with Object.freeze() in another context and, unfortunately, I don't think it's a good idea to use it since it prevents angular from rendering it in an ng-repeat, which requires adding a unique $$hashKey to the object... I'll do something else.

Bargs · 2017-05-23T17:58:45Z

src/utils/__tests__/kbn_field_types.js

+    });
+
+    it('returns undefined for invalid name', () => {
+      expect(getKbnFieldType(Math.random())).to.be(undefined);


Adding randomness to a test like this doesn't seem like a good idea to me. What if it behaves differently if you pass in 0 vs 42? Also what makes the name invalid? Is it the fact that you're passing a non-string? If that's the case I would say 'returns undefined for non-string arguments'.

I generally hesitate to use static strings for "invalid" values, so I wanted to use a little randomness and got lazy.

Bargs · 2017-05-23T20:33:54Z

src/utils/__tests__/kbn_field_types.js

+    });
+
+    it('returns unknown for unknown es types', () => {
+      expect(castEsToKbnFieldTypeName(Math.random())).to.be('unknown');


Wouldn't it make more sense to test a string parameter here?

Bargs · 2017-05-23T20:35:43Z

src/utils/__tests__/kbn_field_types.js

+
+  describe('castEsToKbnFieldType()', () => {
+    it('returns the kbnFieldType instance that matches the esType', () => {
+      expect(castEsToKbnFieldType('keyword')).to.be.a(KbnFieldType);


This is testing that the returned value is a KbnFieldType but not whether it is the correct KbnFieldType.

Bargs · 2017-05-23T20:36:21Z

src/utils/__tests__/kbn_field_types.js

+    });
+
+    it('returns the unknown field type for unknown es types', () => {
+      expect(castEsToKbnFieldType(Math.random())).to.be(getKbnFieldType('unknown'));


Same question about using a string param as above

Bargs · 2017-05-23T20:44:05Z

src/ui/public/field_editor/field_editor.js

@@ -21,7 +21,7 @@ uiModules
  const fieldTypesByLang = {
    painless: ['number', 'string', 'date', 'boolean'],
    expression: ['number'],
-    default: _.keys(Private(IndexPatternsCastMappingTypeProvider).types.byType)
+    default: getEsTypes()


This should be a list of Kibana types, not ES types.

Bargs · 2017-05-23T21:12:27Z

src/fixtures/stubbed_logstash_index_pattern.js

-    if (!_.has(field, 'sortable')) field.sortable = type.sortable;
-    if (!_.has(field, 'filterable')) field.filterable = type.filterable;
-    return field;
+    const kbnType = castEsToKbnFieldType(field.type);


This is always returning the unknown field type because field.type already has the kibana type name.

…ect instance

spalger · 2017-05-23T23:19:58Z

Thanks for the review @Bargs, everything should be resolved.

Object.freeze() keeps properties of an object from being redefined or removed, which we want for the kbnFieldTypes, but it also prevents them from being iterated by angular since angular needs to add a unique `$hashKey` property to each object. To keep the properties read only but allow extension KbnFieldType uses Object.defineProperties() instead.

kimjoar

Just some minor nitpicks, but just feel free to skip them. LGTM

kimjoar · 2017-05-24T20:06:10Z

src/utils/kbn_field_types.js

+      esTypes = []
+    } = options;
+
+    Object.defineProperties(this, {


A bit meh, but I understand why, so I'm good with it. If this is what we have to do to have immutable objects, this is what we do.

kimjoar · 2017-05-24T20:09:07Z

src/utils/kbn_field_types.js

+ *  @return {Array<string>}
+ */
+export function getKbnTypeNames() {
+  return KBN_FIELD_TYPES.map(type => type.name);


Almost feels like we should make KBN_FIELD_TYPES an object keyed on the name. That would make this and getKbnFieldType simpler. Not a big deal, though.

I have a thing about storing lists of objects in anything other than an array 😷 . But I would be fine creating a secondary object like KBN_FIELD_TYPES_BY_NAME or something.

No worries. I'm fine with keeping this as-is.

I agree it's fine as-is, but just to be philosophical about things, this is a scenario where a Map would be useful no?

An extra map, for optimizing access by name, sure. But there are other types of access we might want to optimize for. And these optimizations are definitely on the nano-scale

kimjoar · 2017-05-24T20:13:54Z

src/fixtures/logstash_fields.js

@@ -51,6 +53,10 @@ function stubbedLogstashFields() {
      scripted = !!script,
    } = metadata;

+    const type = (esType === 'conflict' || esType === 'unknown')


esType can never be unknown here.

Also, what is conflict and why is it special-cased here? (instead of being handled within castEsToKbnFieldTypeName)

(maybe just add a comment above this line that describes why we're special-casing these?)

Correct, unknown is not possible here. The conflict type isn't from es, but is a special kbnFieldType, but we don't have any other way of representing here.

kimjoar · 2017-05-24T20:18:29Z

src/ui/public/field_editor/__tests__/field_editor.js

-    .when('GET', '/api/kibana/scripts/languages')
-    .respond(['expression', 'painless']);
+    getScriptedLangsResponse = $httpBackend.when('GET', '/api/kibana/scripts/languages');
+    getScriptedLangsResponse.respond(['expression', 'painless']);


My preference in beforeEach: if it gets overridden in any test, it should always be handled within each test (so I only beforeEach something if it concerns all the tests it covers, otherwise I find it just becomes more difficult to track the state of things)

I tried to do something like this at first but it just spiraled out of control and changed way too much stuff.

No worries, just skip it if it adds noise to this PR

kimjoar · 2017-05-24T20:20:45Z

src/utils/__tests__/kbn_field_types.js

+
+  describe('getKbnTypeNames()', () => {
+    it('returns a list of all kbnFieldType names', () => {
+      const esTypes = getKbnTypeNames();


Maybe just .sort() this and do an equals check?

My intention was more about not duplicating the whole list of kbnType names, but I'm fine with it. The list will basically never change

This is perfect for snapshot testing 🎉 (you don't want to maintain the list yourself really, you just want to make sure you have something that fails if the list changes)

kimjoar · 2017-05-24T21:55:26Z

Bargs · 2017-05-24T22:30:59Z

LGTM

…11967) * [utils] add util for converting between es and kibana types * [utils/kbnFieldTypes] use random strings instead of numbers * [utils/kbnFieldTypes] ensure that castEsToKbnFieldType() returns correct instance * [utils/kbnFieldTypes] change getEsTypes() -> getKbnTypeNames() * [fieldEditor] update test to validate limited type support * [fieldEditor] unexpected scripted field langs should list all kbn types * [test/stubbedLogstashIndexPattern] fix kbnFieldType use * [utils/kbnFieldTypes] remove unused castEsToKbnFieldType() fn * [utils/kbnFieldTypes] don't use Object.freeze() Object.freeze() keeps properties of an object from being redefined or removed, which we want for the kbnFieldTypes, but it also prevents them from being iterated by angular since angular needs to add a unique `$hashKey` property to each object. To keep the properties read only but allow extension KbnFieldType uses Object.defineProperties() instead. * [fixtures/logstashFields] fix use of "unknown" and "conflict" types * [stubs/logstashFields] mention why "conflict" is special * [utils/kbnFieldTypes] check complete output of getKbnTypeNames() (cherry picked from commit c9afc8b)

…1967) (#11993) * [utils] add util for converting between es and kibana types (#11967) * [utils] add util for converting between es and kibana types * [utils/kbnFieldTypes] use random strings instead of numbers * [utils/kbnFieldTypes] ensure that castEsToKbnFieldType() returns correct instance * [utils/kbnFieldTypes] change getEsTypes() -> getKbnTypeNames() * [fieldEditor] update test to validate limited type support * [fieldEditor] unexpected scripted field langs should list all kbn types * [test/stubbedLogstashIndexPattern] fix kbnFieldType use * [utils/kbnFieldTypes] remove unused castEsToKbnFieldType() fn * [utils/kbnFieldTypes] don't use Object.freeze() Object.freeze() keeps properties of an object from being redefined or removed, which we want for the kbnFieldTypes, but it also prevents them from being iterated by angular since angular needs to add a unique `$hashKey` property to each object. To keep the properties read only but allow extension KbnFieldType uses Object.defineProperties() instead. * [fixtures/logstashFields] fix use of "unknown" and "conflict" types * [stubs/logstashFields] mention why "conflict" is special * [utils/kbnFieldTypes] check complete output of getKbnTypeNames() (cherry picked from commit c9afc8b) * [fieldEditor] remove test that only applies to 6.0+

…11967) * [utils] add util for converting between es and kibana types * [utils/kbnFieldTypes] use random strings instead of numbers * [utils/kbnFieldTypes] ensure that castEsToKbnFieldType() returns correct instance * [utils/kbnFieldTypes] change getEsTypes() -> getKbnTypeNames() * [fieldEditor] update test to validate limited type support * [fieldEditor] unexpected scripted field langs should list all kbn types * [test/stubbedLogstashIndexPattern] fix kbnFieldType use * [utils/kbnFieldTypes] remove unused castEsToKbnFieldType() fn * [utils/kbnFieldTypes] don't use Object.freeze() Object.freeze() keeps properties of an object from being redefined or removed, which we want for the kbnFieldTypes, but it also prevents them from being iterated by angular since angular needs to add a unique `$hashKey` property to each object. To keep the properties read only but allow extension KbnFieldType uses Object.defineProperties() instead. * [fixtures/logstashFields] fix use of "unknown" and "conflict" types * [stubs/logstashFields] mention why "conflict" is special * [utils/kbnFieldTypes] check complete output of getKbnTypeNames()

spalger added :Management v5.5.0 v6.0.0 labels May 23, 2017

spalger requested review from Bargs and BigFunger May 23, 2017 00:24

spalger force-pushed the share/es-kbn-type-conversion branch from a2e43b8 to 9ced8d5 Compare May 23, 2017 00:49

[utils] add util for converting between es and kibana types

23b5e47

spalger force-pushed the share/es-kbn-type-conversion branch from 9ced8d5 to 23b5e47 Compare May 23, 2017 00:50

This was referenced May 23, 2017

[indexPatterns/field] remove indexed/analyzed/doc_values awareness #11969

Merged

[IndexPatterns] Support cross cluster search #11114

Merged

Bargs suggested changes May 23, 2017

View reviewed changes

spalger added 7 commits May 23, 2017 15:33

[utils/kbnFieldTypes] use random strings instead of numbers

89b9300

[utils/kbnFieldTypes] ensure that castEsToKbnFieldType() returns corr…

11c45b2

…ect instance

[utils/kbnFieldTypes] change getEsTypes() -> getKbnTypeNames()

c4c6422

[fieldEditor] update test to validate limited type support

87013de

[fieldEditor] unexpected scripted field langs should list all kbn types

4237f12

[test/stubbedLogstashIndexPattern] fix kbnFieldType use

a6e90e4

[utils/kbnFieldTypes] remove unused castEsToKbnFieldType() fn

ec00377

spalger added 2 commits May 23, 2017 18:00

[fixtures/logstashFields] fix use of "unknown" and "conflict" types

2b65f6c

spalger force-pushed the share/es-kbn-type-conversion branch from a33c795 to 2b65f6c Compare May 24, 2017 01:10

spalger requested review from kimjoar and removed request for BigFunger May 24, 2017 16:57

spalger added a commit to spalger/kibana that referenced this pull request May 24, 2017

squashed commit of all changes, awaiting elastic#11967 and elastic#11969

5e1cbdf

spalger added the review label May 24, 2017

kimjoar approved these changes May 24, 2017

View reviewed changes

spalger added 2 commits May 24, 2017 14:21

[stubs/logstashFields] mention why "conflict" is special

6c081ac

[utils/kbnFieldTypes] check complete output of getKbnTypeNames()

a4754aa

Bargs approved these changes May 24, 2017

View reviewed changes

spalger merged commit c9afc8b into elastic:master May 24, 2017

jsoref referenced this pull request in jsoref/kibana Jun 21, 2018

spelling: attachment

d565720

spalger deleted the share/es-kbn-type-conversion branch October 18, 2019 17:43

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[utils] add util for converting between es and kibana types #11967

[utils] add util for converting between es and kibana types #11967

spalger commented May 23, 2017 •

edited

Bargs May 23, 2017

spalger May 23, 2017 •

edited

spalger May 24, 2017

Bargs May 23, 2017

spalger May 23, 2017

Bargs May 23, 2017

Bargs May 23, 2017

Bargs May 23, 2017

Bargs May 23, 2017

Bargs May 23, 2017

spalger commented May 23, 2017

kimjoar left a comment

kimjoar May 24, 2017

kimjoar May 24, 2017

spalger May 24, 2017 •

edited

kimjoar May 24, 2017

Bargs May 24, 2017

spalger May 24, 2017 •

edited

kimjoar May 24, 2017

spalger May 24, 2017

kimjoar May 24, 2017

spalger May 24, 2017

kimjoar May 24, 2017

kimjoar May 24, 2017

spalger May 24, 2017

kimjoar May 24, 2017

kimjoar commented May 24, 2017

Bargs commented May 24, 2017

[utils] add util for converting between es and kibana types #11967

[utils] add util for converting between es and kibana types #11967

Conversation

spalger commented May 23, 2017 • edited

Choose a reason for hiding this comment

spalger May 23, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

spalger commented May 23, 2017

kimjoar left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

spalger May 24, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

spalger May 24, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kimjoar commented May 24, 2017

Bargs commented May 24, 2017

spalger commented May 23, 2017 •

edited

spalger May 23, 2017 •

edited

spalger May 24, 2017 •

edited

spalger May 24, 2017 •

edited