Big Data Support - Scaling Vuu - How to abstract Vuu interfaces to allow different implementations of key classes by adding "features" on construction.

[chrisjstevo]

Discussed in #971

^{Originally posted by chrisjstevo November 15, 2023}

Introduction

Vuu is currently a largely in memory, near caching solution for UIs. In order to allow Vuu to scale to large data sets (10bn rows) we cannot cache all the data in memory, we need to be able to use big data stores, and move a remote cursor around in that data set as users scroll.

In order to facilitate this requirement we have started to think about Vuu rather than as a deeply entwined 'core' library, but more as a series of internal interfaces. Some examples of these interfaces are:

DataTable
ViewPort
ViewPortKeys() //work in progress
Provider
JoinManager
LoginValidator
Authenticator

If we take the first 5 interfaces, these relate to how we store and join data in memory. When we move to a paradigm where all the data is not in memory, but simply a window of the data is in memory, we will need to provide different implementations of these interfaces to facilitate this.

Features & Plugins

Features, as described here, are a way of grouping alternative implementations to some or all of the above interfaces with non standard implementations. For example, we may want to have the situation where we want to back Tables and Viewports by a Big Data store. In this case we would want to have some construct, conceptually like below, where a user could "plugin" a different implementation to use.

Example:

  //in a normal config object
  val config = VuuServerConfig(
    VuuHttp2ServerOptions()
      //only specify webroot if we want to load the source locally, we'll load it from the jar
     //</snip>
    VuuSecurityOptions()
      .withAuthenticator(authenticator)
      .withLoginValidator(new AlwaysHappyLoginValidator),
    VuuThreadingOptions()
      .withViewPortThreads(4)
      .withTreeThreads(4)
  ).withModule(PriceModule())
    .withModule(BasketModule())
    //add big data plugin 
    withPlugin(BigDataTableAndViewportPlugin())

We would then have a plugin type, that would allow the provision of some or all of these interfaces:

class BigDataTableAndViewportPlugin{
     
         registerFeature[DataTable](BigDataTableFeature())
         registerFeature[JoinManager](BigDataJoinManagerFeature())
}