Using pipelines to manipulate data: Spacetime

[wylieconlon]

This PR contains two separate but related pieces of work to answer my own questions about how we should approach data manipulation in Lens and Kibana as a whole. There are two pieces:

• An app called "pipeline builder" which lets users make POST requests to Elasticsearch, transform them into tables, join tables, and visualize the results.

• An integration into the Lens editor which lets you create "hidden queries"

After doing this work, I have concluded that:

• Lens should have a feature like the one I've implemented, where users can edit the entire query in one screen. I've written this up as architectural requirements for calculations in Lens
• Letting advanced users type their queries as JSON is doable. We can convert JSON -> Table by converting every Key into a column, and arrays into rows. This is [Lens] New datasource for Elasticsearch queries as JSON #59439

What are some outcomes of this work?

In the pipeline builder app, I built a "stacked as percentage" visualization from individual documents:

In Lens, I was able to build a cumulative sum + initial starting value:

Other conclusions, in no particular order:

• Lens needs to use a unified table
• It is very important to support the basic relational algebra operations, including joining tables.
• Anything that requires users to type is going to be harder to get right- let's do as much as possible without typing.

[kibanamachine]

💔 Build Failed

• continuous-integration/kibana-ci/pull-request
• Commit: 6a6790e
• Pipeline Steps (look for red circles / failed steps)
• Interpreting CI Failures

Failed CI Steps

• Execute xpack-kibana-ciGroup10
• Execute xpack-kibana-ciGroup8
• Execute xpack-kibana-ciGroup7
• Execute xpack-kibana-ciGroup6
• Execute xpack-kibana-ciGroup4
• Execute xpack-kibana-ciGroup2
• Execute xpack-kibana-ciGroup1
• Execute kibana-ciGroup1
• Execute kibana-ciGroup1
• Execute x-pack-intake
• Execute kibana-intake

---

Test Failures

Kibana Pipeline / x-pack-intake-agent / X-Pack Jest Tests.x-pack/plugins/lens/public/editor_frame_service/editor_frame.editor_frame state update should re-render config panel with updated datasource api after datasource state update

Link to Jenkins

Standard Out

Failed Tests Reporter:
  - Test has not failed recently on tracked branches

Stack Trace

TypeError: Cannot read property 'map' of undefined
    at LayerPanel (/dev/shm/workspace/kibana/x-pack/plugins/lens/public/editor_frame_service/editor_frame/config_panel/layer_panel.tsx:91:29)
    at renderWithHooks (/dev/shm/workspace/kibana/node_modules/react-dom/cjs/react-dom.development.js:16260:18)
    at updateFunctionComponent (/dev/shm/workspace/kibana/node_modules/react-dom/cjs/react-dom.development.js:18347:20)
    at beginWork$1 (/dev/shm/workspace/kibana/node_modules/react-dom/cjs/react-dom.development.js:20176:16)
    at HTMLUnknownElement.callCallback (/dev/shm/workspace/kibana/node_modules/react-dom/cjs/react-dom.development.js:336:14)
    at invokeEventListeners (/dev/shm/workspace/kibana/node_modules/jest-environment-jsdom/node_modules/jsdom/lib/jsdom/living/events/EventTarget-impl.js:193:27)
    at HTMLUnknownElementImpl._dispatch (/dev/shm/workspace/kibana/node_modules/jest-environment-jsdom/node_modules/jsdom/lib/jsdom/living/events/EventTarget-impl.js:119:9)
    at HTMLUnknownElementImpl.dispatchEvent (/dev/shm/workspace/kibana/node_modules/jest-environment-jsdom/node_modules/jsdom/lib/jsdom/living/events/EventTarget-impl.js:82:17)
    at HTMLUnknownElementImpl.dispatchEvent (/dev/shm/workspace/kibana/node_modules/jest-environment-jsdom/node_modules/jsdom/lib/jsdom/living/nodes/HTMLElement-impl.js:30:27)
    at HTMLUnknownElement.dispatchEvent (/dev/shm/workspace/kibana/node_modules/jest-environment-jsdom/node_modules/jsdom/lib/jsdom/living/generated/EventTarget.js:143:21)
    at Object.invokeGuardedCallbackDev (/dev/shm/workspace/kibana/node_modules/react-dom/cjs/react-dom.development.js:385:16)
    at invokeGuardedCallback (/dev/shm/workspace/kibana/node_modules/react-dom/cjs/react-dom.development.js:440:31)
    at beginWork$$1 (/dev/shm/workspace/kibana/node_modules/react-dom/cjs/react-dom.development.js:25780:7)
    at performUnitOfWork (/dev/shm/workspace/kibana/node_modules/react-dom/cjs/react-dom.development.js:24698:12)
    at workLoopSync (/dev/shm/workspace/kibana/node_modules/react-dom/cjs/react-dom.development.js:24671:22)
    at performSyncWorkOnRoot (/dev/shm/workspace/kibana/node_modules/react-dom/cjs/react-dom.development.js:24270:11)
    at /dev/shm/workspace/kibana/node_modules/react-dom/cjs/react-dom.development.js:12199:24
    at unstable_runWithPriority (/dev/shm/workspace/kibana/node_modules/scheduler/cjs/scheduler.development.js:697:12)
    at runWithPriority$2 (/dev/shm/workspace/kibana/node_modules/react-dom/cjs/react-dom.development.js:12149:10)
    at flushSyncCallbackQueueImpl (/dev/shm/workspace/kibana/node_modules/react-dom/cjs/react-dom.development.js:12194:7)
    at flushSyncCallbackQueue (/dev/shm/workspace/kibana/node_modules/react-dom/cjs/react-dom.development.js:12182:3)
    at batchedUpdates$1 (/dev/shm/workspace/kibana/node_modules/react-dom/cjs/react-dom.development.js:24392:7)
    at act (/dev/shm/workspace/kibana/node_modules/react-dom/cjs/react-dom-test-utils.development.js:1092:14)
    at Object.it (/dev/shm/workspace/kibana/x-pack/plugins/lens/public/editor_frame_service/editor_frame/editor_frame.test.tsx:709:7)

---

Kibana Pipeline / x-pack-intake-agent / X-Pack Jest Tests.x-pack/plugins/lens/public/editor_frame_service/editor_frame.editor_frame passing state back to the caller should call onChange only when the active datasource is finished loading

Link to Jenkins

Standard Out

Failed Tests Reporter:
  - Test has not failed recently on tracked branches

Stack Trace

Error: expect(jest.fn()).toHaveBeenNthCalledWith(n, ...expected)

n: 1
Expected: {"doc": {"expression": "", "id": undefined, "state": {"datasourceMetaData": {"filterableIndexPatterns": [{"id": "1", "title": "resolved"}]}, "datasourceStates": {"testDatasource": undefined}, "filters": [], "query": {"language": "lucene", "query": ""}, "visualization": null}, "title": "", "type": "lens", "visualizationType": "testVis"}, "filterableIndexPatterns": [{"id": "1", "title": "resolved"}]}
Received
->     1
        @@ -13,10 +13,15 @@
                },
                "datasourceStates": Object {
                  "testDatasource": undefined,
                },
                "filters": Array [],
        +       "pipeline": Object {
        +         "postjoin": Array [],
        +         "prejoin": Object {},
        +         "timeRangeOverrides": Object {},
        +       },
                "query": Object {
                  "language": "lucene",
                  "query": "",
                },
                "visualization": null,,
       2: {"doc": {"expression": "", "id": undefined, "state": {"datasourceMetaData": {"filterableIndexPatterns": [{"id": "1", "title": "resolved"}]}, "datasourceStates": {"testDatasource": undefined}, "filters": [], "pipeline": {"postjoin": [], "prejoin": {}, "timeRangeOverrides": {}}, "query": {"language": "lucene", "query": ""}, "visualization": {"initialState": true}}, "title": "", "type": "lens", "visualizationType": "testVis"}, "filterableIndexPatterns": [{"id": "1", "title": "resolved"}]}

Number of calls: 2
    at Object.it (/dev/shm/workspace/kibana/x-pack/plugins/lens/public/editor_frame_service/editor_frame/editor_frame.test.tsx:1487:24)

---

Kibana Pipeline / x-pack-intake-agent / X-Pack Jest Tests.x-pack/plugins/lens/public/editor_frame_service/editor_frame.editor_frame passing state back to the caller should send back a persistable document when the state changes

Link to Jenkins

Standard Out

Failed Tests Reporter:
  - Test has not failed recently on tracked branches

Stack Trace

Error: expect(jest.fn()).toHaveBeenNthCalledWith(n, ...expected)

n: 3
Expected: {"doc": {"expression": StringContaining "vis \"expression\"", "id": undefined, "state": {"datasourceMetaData": {"filterableIndexPatterns": []}, "datasourceStates": {"testDatasource": undefined}, "filters": [], "query": {"language": "lucene", "query": "new query"}, "visualization": {"initialState": true}}, "title": "", "type": "lens", "visualizationType": "testVis"}, "filterableIndexPatterns": []}
Received
       2: {"doc": {"expression": "", "id": undefined, "state": {"datasourceMetaData": {"filterableIndexPatterns": []}, "datasourceStates": {"testDatasource": undefined}, "filters": [], "pipeline": {"postjoin": [], "prejoin": {}, "timeRangeOverrides": {}}, "query": {"language": "lucene", "query": ""}, "visualization": {"initialState": true}}, "title": "", "type": "lens", "visualizationType": "testVis"}, "filterableIndexPatterns": []}
->     3
        @@ -1,17 +1,25 @@
          Object {
            "doc": Object {
        -     "expression": StringContaining "vis \"expression\"",
        +     "expression": "kibana
        + | kibana_context  query=\"{\\\"query\\\":\\\"new query\\\",\\\"language\\\":\\\"lucene\\\"}\" filters=\"[]\"
        + | lens_merge_tables layerIds=\"first\" tables={data \"expression\"}
        + | vis \"expression\"",
              "id": undefined,
              "state": Object {
                "datasourceMetaData": Object {
                  "filterableIndexPatterns": Array [],
                },
                "datasourceStates": Object {
                  "testDatasource": undefined,
                },
                "filters": Array [],
        +       "pipeline": Object {
        +         "postjoin": Array [],
        +         "prejoin": Object {},
        +         "timeRangeOverrides": Object {},
        +       },
                "query": Object {
                  "language": "lucene",
                  "query": "new query",
                },
                "visualization": Object {,

Number of calls: 3
    at Object.it (/dev/shm/workspace/kibana/x-pack/plugins/lens/public/editor_frame_service/editor_frame/editor_frame.test.tsx:1560:24)

---

and 46 more failures, only showing the first 3.

History

• 💔 Build #51305 failed f881ee3
• 💔 Build #49359 failed ca17d19
• 💔 Build #49181 failed 356e473

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream