TSDB support for Lens, TSVB and Timelion

[flash1293]

Fixes #131517
Fixes #131518
Fixes #136403
Fixes #136404

This PR is implementing the first parts of supporting TSDB in Lens, TSVB and Timelion.

How to test

• Check out https://github.com/elastic/elasticsearch/tree/kibana-downsampling-branch next to your kibana working copy
• Start ES with yarn es source instead of snapshot (this will build the given branch)
• Clone https://github.com/csoulios/downsampling-demo
• Run the commands in downsample-index.es (indexing the actual data is too much for dev console, so you have to do this via curl curl -XPOST http://elastic:changeme@localhost:9200/sample-01/_bulk\?pretty -H 'Content-Type: application/json' --data-binary @sample-01-raw.json)
• After running all commands, there should be sample-01-rollup which has the rolled up data
• Create a data view sample-01 for testing TSDB without rollups and a data view sample-01,sample-01-rollup for testing TSDB with rollups

Lens

• Implement new field type picker
• Add counter and gauge field types to the list as separate types
• When dragging a field that is also defined on rollup indices, do not default to median, instead default to average
• When rendering the function list in the dimension editor, split the function list into "safe" functions that work with rolled up data too and "unsafe" functions that won't work
• When a field is defined both

TSVB

If the current data view is hitting rolled up data, date histogram aggregations are always using fixed_interval instead of calendar_interval and UTC instead of the configured timezone.

This can be checked via the inspector.

It should not influence TSVB charts which are executed on index patterns not hitting rolled up timeseries indices.

Timelion

Like TSVB, if the current data view is hitting rolled up data, date histogram aggregations are always using fixed_interval instead of calendar_interval and UTC instead of the configured timezone.

This can't be checked via inspector as Timelion isn't reporting executed requests - the easiest way to test is to add a console.log for the sent search in

kibana/src/plugins/vis_types/timelion/server/series_functions/es/index.js

Line 131 in 1681ebe

body,

Split out intentionally

• Functional tests (this is not merged yet on Elasticsearch side, gonna follow up in another PR)
• Shard failure handling
• Splitting function list in the dropdown of moving average and differences
• Some kind of indication of safe and unsafe functions in Lens formula
• UTC and fixed interval enforcement in Lens and agg based visualizations: UTC and fixed interval not enforced for rolled up time series indices #139081

[elasticmachine]

Pinging @elastic/kibana-vis-editors @elastic/kibana-vis-editors-external (Team:VisEditors)

[KOTungseth]

Since the results are partial instead of the application, would it make sense to rename Partially applicable functions to Partial results functions?

How about this for the tooltip:

The full time range of your data is unsupported by the Partial results functions (if this is what we want to go with). For example, when you use rolled up historical data. You can still use these functions, but the results are partial and the visualization displays a warning.

[ppisljar]

LGTM

[MichaelMarcialis]

When was it decided that we would be splitting the quick functions list in this manner? I'm personally not a fan of this direction. I would much rather we take a similar approach to how we handle incompatible quick functions, where we keep a single unified function list but visually offset the incompatible functions (with the yellow dot) and provide a tooltip there. In this particular case, it doesn't have to be the same yellow dot, but it could be similar in concept. Thoughts?

[flash1293]

This is different than the “incompatible” dot, as it’s possible to use this function with the current field, but it might produce partial data. Requiring the user to hover over the dot to learn about this seems too subtle to me, but if you and @ghudgins think it would be the way to go I’m happy with that too. What about a different kind of indicator for this?

[flash1293]

I wrote my comment before you added the last sentence - agreed with that 😁 no concerns with an indicator per function in a unified list but it should be different than the existing one.

[drewdaemon]

I get this error when I run yarn es source

These are the contents of my .idea folder

I tried installing Eclipse and using it to open Elasticsearch. I also tried adding an empty XML file where it's looking for it, but it just keeps looking for more and more XML files.

Do you know how to solve this @flash1293 ?

[flash1293]

Unfortunately I don't, I didn't run into this error. @spalger do you know what's going on here? #139020 (comment)

Edit: Checking the code of install_source - it seems like this is related to some dirty-state checking of the repo:

kibana/packages/kbn-es/src/install/install_source.ts

Line 97 in 662adf8

status.files.forEach((file) => {

This is weird, the comment says it is only checking modified files but then it tries to iterate over status.files which is

   /**
    * All files represented as an array of objects containing the `path` and status in `index` and
    * in the `working_dir`.
    */
   files: FileStatusResult[];

. Is this working as intended?

@andrewctate To make it work temporarily I have two ideas what to try:

• You could try commenting out this loop so it doesn't try to load these files. However the root cause might fail other things too
• You could try running es source from a directory called kibana next to the elasticsearch directory (instead of kibana-red). This is what I did, not sure whether it changes anything, but worth a try.

[spalger]

I have very limited experience with yarn es source and I think very few people use it. Are there modifications in the ES repo that might not be fully applied? I wouldn't be surprised if the "install_source" stuff should be reworked.

[MichaelMarcialis]

I wrote my comment before you added the last sentence - agreed with that 😁 no concerns with an indicator per function in a unified list but it should be different than the existing one.

Cool. EUI happens to have a partial icon. We could append that icon to the functions in question, similar to how we append the yellow "incompatible" dots. @ghudgins, did you want to weigh in on this direction versus splitting the quick function list?

[drewdaemon]

I tested all three apps and went through the code. LGTM—just had a few questions.

1.

"unsafe" functions that won't work

Why do we allow these to be selected at all?

2.

I verified that both TSVB and Timelion switch their requests to use UTC as soon as they're looking at rolled-up data. However, in my case, they both attached fixed_interval regardless of whether or not they were dealing with rolled up data. Could have been my visualization configuration, but thought it was worth mentioning since I didn't figure out how to make them use calendar interval for non-rolled-up data.

3.

Am I correct in understanding that we are ultimately relying on the field caps API to report a list of possible fixed intervals for a rolled-up field?

All my other comments will be addressed by your follow-up list 👍

[drewdaemon]

Suggested change

	!referencedField ||
	!referencedField.softRestrictions ||
	!referencedField.softRestrictions[navItem.id!]
	!referencedField?.softRestrictions?.[navItem.id!]

[ghudgins]

Why do we allow these to be selected at all?

Because the time range can change and they can suddenly work. The time range dictates if the indexes that are returned are downsampled or not

[drewdaemon]

does a unit test make sense here?

[flash1293]

Am I correct in understanding that we are ultimately relying on the field caps API to report a list of possible fixed intervals for a rolled-up field?

That's correct, but we don't leverage the specific value somewhere, it's just about whether there is an index which is rolled up in a certain way - if it is, we can still use other fixed intervals without error (even if smaller than the rollup interval), but we can't safely apply calendar intervals anymore.

[flash1293]

Thanks for the review, addressed all comments.

Why do we allow these to be selected at all?

As Graham noted, this is a "soft" restriction because in 99% of the cases there is also recent non-rolled up data for which it makes sense to apply these functions. So we allow users to pick it and live with the warnings. Note that in case of the sample data as described above the rolled-up and regular data is in the same time range but that's rather unusual. In most cases the rolled up data will be in another time range (e.g. last week won't be rolled up, but the month before that will), so the user won't actually get a warning as long as looking at the last week.

However, in my case, they both attached fixed_interval regardless of whether or not they were dealing with rolled up data

As long as the selected data view includes at least one index that's rolled up that's the expected behavior - it's difficult but possible to detect this situation on a per-query basis and adjust the interval based on this information, but we decided to not go with this as it will be extra confusing when suddenly crossing the boundary (for example looking at the last 5 days will give you calendar intervals in the current time zone, changing the time range to 8 days will suddenly give you fixed intervals in UTC). Data views which don't include rolled up data at all should not be affected (not sure which case you ran into - the described behavior is what I'm seeing locally)

This is how the function list looks now (I like it, what do you think @MichaelMarcialis ?):

Note that the "incompatible" dot will never show up together with the partial icon - if a field is incompatible with a function the current field is not transferred over so we do not know whether it will be partially applicable or not (as this depends on the choice the user didn't make yet)

[MichaelMarcialis]

Thanks for making that change to the quick function list, @flash1293! That looks much better to my eyes. I'm leaving a few comments below for your review, but nothing earth-shattering. Assuming those get addressed, I'm approving now so I don't hold you up further.

• I noticed that the "Search field names" input above the field list has a double border (CSS box-shadow in reality) effect happening in this PR that doesn't seem to appear in the main branch. Any chance to fix it so this artificial thickening of the input border doesn't happen?

• Since you mentioned that the quick functions' incompatible dot icons can't appear simultaneously with the newly implemented partial icon, would you be willing to try coloring the partial icon in the same yellow color we use for the incompatible dot icons? Assuming it doesn't look awful, that might help to both visually offset it from the text color and be consistent with the notion that it's a warning.

[flash1293]

It looks good IMHO

[kibana-ci]

💚 Build Succeeded

• Buildkite Build
• Commit: 730b7b7

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
dataVisualizer	565.7KB	565.8KB	+76.0B
discover	462.4KB	462.5KB	+76.0B
graph	412.5KB	412.6KB	+76.0B
lens	1.2MB	1.2MB	+2.0KB
maps	2.6MB	2.6MB	+76.0B
presentationUtil	130.5KB	130.5KB	+76.0B
securitySolution	5.7MB	5.7MB	+76.0B
stackAlerts	99.5KB	99.6KB	+76.0B
total			+2.5KB

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
osquery	16.3KB	16.3KB	+76.0B

History

• 💚 Build #67955 succeeded 61ce65e
• 💔 Build #67947 failed 662e1d5
• 💚 Build #66231 succeeded 3b5fdd2
• 💚 Build #65977 succeeded 3bc324a
• 💚 Build #65957 succeeded ee154e2
• 💔 Build #65788 failed 1681ebe

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

[ghudgins]

dots look good. sorry for the late 👀