[MD] [Research] Onboard Vis timeseries (TSVB) to use datasource

[kristenTian]

Onboard Vis timeseries (TSVB) to use datasource

Code Ref:

Vis timeseries is also using getFieldsForWildcard so data source Id need to be added: /packages/Kibana/blobs/7.10.2/--/src/plugins/vis_type_timeseries/server/lib/search_strategies/strategies/abstract_search_strategy.ts

/packages/Kibana/blobs/b03c8bc565d8e58365b9229e2b0bbc41e29731[…]6/--/src/plugins/vis_type_timeseries/server/lib/get_fields.ts

Update to pass datasource in following place:

/packages/Kibana/blobs/b03c8bc565d8e58365b9229e2b0bbc41e2973186/--/src/plugins/vis_type_timeseries/server/lib/get_fields.ts#L48,L76

Front end call: /packages/Kibana/blobs/7.10.2/--/src/plugins/vis_type_timeseries/public/application/lib/fetch_fields.js

->
/packages/Kibana/blobs/f429e08e31b94428cb2efe7b15c1c7dd7ba3ab77/--/src/plugins/vis_type_timeseries/public/application/components/vis_editor.js#L107

[zhongnansu]

@kristenTian Please don't expose internal code amazon link on github

[seraphjiang]

@kristenTian Please don't expose internal code amazon link on github

Thanks Zhongnan. edit the content accordingly

[zhongnansu]

TSVB

• API:
  • api/metrics/vis/data to query data
  • api/metircs/fields to get fields capability, see https://www.elastic.co/guide/en/elasticsearch/reference/7.17/search-field-caps.html
• Research:
  1. TVSB register to expression plugin and add viz configs here (https://github.com/opensearch-project/OpenSearch-Dashboards/blob/1.2.0/src/plugins/vis_type_timeseries/public/plugin.ts#L82-L85)
  2. Unlike other viz types, TVSB plugin has it’s own backend
     1. In the backend of TVSB, it builds the query request from index pattern (

        OpenSearch-Dashboards/src/plugins/vis_type_timeseries/server/lib/vis_data/get_table_data.js

        Line 47 in e3b34df

        const { indexPatternObject } = await getIndexPatternObject(req, panelIndexPattern);

        ) and add default search strategy (https://github.com/opensearch-project/opensearch-dashboards/blob/e3b34df1dea59a253884f6da4e49c3e717d362c9/src/plugins/vis_type_timeseries/server/lib/search_strategies/search_strategy_registry.ts#L46) “opensearch” to it. which failed for external datasource
  3. Generated Expression:

  opensearchDashboards | opensearch_dashboards_context | tsvb params='{"axis_formatter":"number","axis_position":"left","axis_scale":"normal","default_index_pattern":"opensearch_dashboards_sample_data_ecommerce","default_timefield":"order_date","filter":"","gauge_color_rules":[{"gauge":"rgba(104,188,0,1)","id":"6da070c0-b891-11e8-b645-195edeb9de84","operator":"gte","value":150},{"gauge":"rgba(244,78,59,1)","id":"9b0cdbc0-b891-11e8-b645-195edeb9de84","operator":"lt","value":150}],"gauge_inner_width":10,"gauge_max":"300","gauge_style":"half","gauge_width":"15","id":"61ca57f0-469d-11e7-af02-69e470af7417","index_pattern":"opensearch_dashboards_sample_data_ecommerce","interval":"1d","isModelInvalid":false,"series":[{"axis_position":"right","chart_type":"line","color":"#68BC00","fill":0.5,"formatter":"number","id":"61ca57f1-469d-11e7-af02-69e470af7417","label":"Trxns / day","line_width":1,"metrics":[{"id":"61ca57f2-469d-11e7-af02-69e470af7417","type":"count"}],"point_size":1,"separate_axis":0,"split_color_mode":"gradient","split_mode":"everything","stacked":"none"}],"show_grid":1,"show_legend":1,"time_field":null,"tooltip_mode":"show_all","type":"gauge"}'  uiState='{}'
  

• Solution: (this solution might be abandoned, can ignore)

  1. No need to make change to the generated expression

  2. Unlike other vis types, TSVB has its own backend. From the expressions TSVB generated, it doesn't go through opensearchaggs to create the searchSource obj, and call high level search API. Instead, it retrieves the index pattern, build the OpenSearch query from their plugin and directly query OpenSearch in 2 ways.

     1. by calling the data plugin low level search API
     2. directly use legacyOpenSearchClient to query OS to get some fields.

  3. We should add logic in TVSB backend to identify if datasource field is used in index-pattern, then adding the correct data source search params.

---

Some actions items for v2.5.0

• UX gap, reach out to UX to discuss the change.
• Engineering side: explore if we can use data plugin high level API, mostly likely that we can't, but still worth taking a look.

[zhongnansu]

UX refactor discussion , need input from UX team

cc @KrooshalUX @zengyan-amazon

Background

• TSVB is different with other index-pattern based visualizations, such as pie chart, metric, etc. MD feature first launch in 2.4, non index-pattern based visualizations are supported out of the box. In 2.5, we want to onboard TSVB
• TSVB currently doesn't only work with saved index patterns. Specifically, it allows the user to enter a string (which can be the index pattern title, an index name, multiple indices names, multiple index-patterns, wildcards).

Proposal

• As metioned here. In the future version of MD feature, we'll change to allow duplicate index-pattern names, with unique identifier that takes in data source info + index pattern info. For TSVB use case, we need a new UX that users can select data source first, then continue with specifying string(index-pattern/index/wildcard, etc). Because cross-datasource query, aggregation is not supported, and the underlying relation between visualization and data source is still 1 to 1.
• I'd propose adding an extra step for user in TSVB to first select data source(can only select one), then select index, index-pattern, or multiple indices within that data source. Noticed that this is still backward compatible if user just select default data source(local cluster).
• Having data source info from client side, will also make server side much easier to integrate with data source.

[seraphjiang]

Hi team @huyaboo @zhongnansu would you review the previous research for tsvb to see if there is any idea we could leverage.

I'm closing it as @huyaboo is actively working it.

cc: @bandinib-amzn @BionIT

[zhongnansu]

Hi team @huyaboo @zhongnansu would you review the previous research for tsvb to see if there is any idea we could leverage.

I'm closing it as @huyaboo is actively working it.

cc: @bandinib-amzn @BionIT

@seraphjiang Sure, I'll sync with @huyaboo.
Linking the active issue here #6290