[Lens] Architectural requirements for calculations

[wylieconlon]

The most important requirement is the ability to have hidden data that is not being visualized. This data is used to calculate the final visualization. For example, if I have columns a and b, and I want to visualize a + b only, I need a way to hide the inputs. By supporting hidden data, we let the user build the visualization in steps. This makes Lens learnable.

The second most important requirement is the ability to join two tables. Without this ability, there is a huge limit on what users can express with their calculations. For example, joining will let us do month over month calculations.

The third most important requirement is the ability to apply lazily evaluated functions to the table. By building a set of generic table manipulation functions, as well as a set of generic functions to handle specific data types like numbers or strings or dates, we can build the standard library that users expect a visualization tool to have. This set of functions should resemble the relational algebra: selection, projection, filtering, and renaming are core concepts.

Finally, the Lens editor needs to take a more active role in the data flow and editing. This will let us keep the same functionality regardless of the datasource: for example, this will let us combine aggregated data with SQL data in the client.

[elasticmachine]

Pinging @elastic/kibana-app (Team:KibanaApp)

[wylieconlon]

Closing this because it doesn't give enough detail to the point where we can't implement it. Many of the topics have separate issues, for example the more detailed issue for "hidden data" is about changing the operations API in the datasource.