TSVB Make mustache template field accessors consistent

[fpompermaier]

Kibana version:
7.5.0
Elasticsearch version:
7.5.0
Server OS version:
Centos 7
Browser version:
Chrome 80.0.3987.122 (Official Build) (64-bit) or Firefox 72.0.2 (64-bit)
Browser OS version:
Windows 10
Original install method (e.g. download page, yum, from source, etc.):
yum
Describe the bug:
On TSVB Annotations I can't use fields with dots (e.g. user.name)
Steps to reproduce:

1. Create a TSVB visualization
2. In the Time Series / Annotations tab use an index pattern having an object field (e.g. user)
3. In the Fields input write user.name and in the Row Template use {{user.name}}
4. If you go over the icon the label is rendered as {{user.name}} an error is displayed (Error processing your markdown: {{name}} is an unknown variable)

Expected behavior:
See the value of user.name

Screenshots (if relevant):

Any additional context:
If I do the same but using fields that does not contain a dot it works

[fpompermaier]

I found that the problem arise when user.name is indexed as a plain field instead of an object.

"user.name" : "Test" make this error appears, while { "user" : { "name" : "Test" }} does not.

[elasticmachine]

Pinging @elastic/kibana-app (Team:KibanaApp)

[fpompermaier]

Anyone here?

[flash1293]

@fpompermaier could you check whether this syntax works for you:

{{this.[user.name]}}

[fpompermaier]

Right now I can't..I have to set up an empty local Kibana and test it..I'll come back to you in some days..

[fpompermaier]

{{this.[user.name]}} works if I index documents using the flat version (i.e. "user.name" : "xxx").
But then it doesn't work if I index using the object version..(i.e. "user" : {"name": "xxx" }).
But what if I index using a mix of the 2 versions (flat and object)?
Can we just define a single way of accessing a field?

[flash1293]

When compiling these templates TSVB takes the _source of the individual document and then executes handlebars. If you are ingesting two documents with the object and the flat version respectively, the source will look different:

POST test/_doc
{ "foo.bar": 1 }

POST test/_doc
{ "foo": { "bar": 1 } }

GET test/_search

The search will return the following:

[
      {
        "_index" : "test",
        "_id" : "N5aoEHEB54e1Wipy1VC4",
        "_score" : 1.0,
        "_source" : {
          "foo.bar" : 1
        }
      },
      {
        "_index" : "test",
        "_id" : "S5apEHEB54e1WipyE1Ap",
        "_score" : 1.0,
        "_source" : {
          "foo" : {
            "bar" : 1
          }
        }
      }
]

I don't think TSVB should preprocess _source somehow to hide this difference, it seems like this layer of magic introduces more problems it creates, as the current behavior is not necessarily "wrong" and existing visualizations could depend on it.

If you actually have data coming in both ways, you can normalize it in Elasticsearch by defining an ingest pipeline using the dot_expander processor: https://www.elastic.co/guide/en/elasticsearch/reference/current/dot-expand-processor.html

[fpompermaier]

Yes they are 2 different structures but this is how elastic works..you don't see any difference if you index in one way or the other. Moreover the dot expand processor does not have a Logstash equivalent at the moment (the fact that the ingestion pipelines and Logstash are diverging is VERY BAD IMHO..I also complained about this on the ES forum [1] but it looks like I'm the only one concerned about that)

[1] https://discuss.elastic.co/t/logstash-and-painless-is-logstash-going-to-be-replaced-by-ingest-nodes/216393

[flash1293]

It's not a difference for the indexed fields, but TSVB annotations are not operating on the indexed values, but on the raw _source. I see your point that this difference is pretty confusing (and it should definitely be mentioned in the docs). Maybe a nice solution would be to use docvalues instead of _source for the template to make things consistent.

I will change the title to reflect the problem we are seeing here.

[richcollier]

@flash1293 - is this bug/ER related to the issue seen in this discuss post: https://discuss.elastic.co/t/fields-and-row-template-in-annotations-tsvb/272433

[flash1293]

@richcollier Answered on the discuss post, this is related but your case is actually solvable.

[stratoula]

Thank you for contributing to this issue, however, we are closing this issue due to inactivity as part of a backlog grooming effort. If you believe this feature/bug should still be considered, please reopen with a comment.