High-level API for defining schemas

[ddimtirov]

This is a proposal for a new API to create Vuu TableDef objects, as well as ensure that any data rows added do not refer missing / misspelled columns.

We would also like to reduce the number of concepts one has to deal with (especially to declare joins), eliminate the need of using weekly typed DSL's (the "name:Type" column notation, or using random types for columns which later fail during serialization). Finally, we need the API to be easy to use from Java, as well as from Scala.

The description below is based on the prototype in vuu-ubs-internal, which we may open at a later stage.
The pseudocode below has been simplified for the sake of illustration (showing only public and protected methods).

class SchemaBuilder<T extends TableDef> {
  // the table name is specified up-front, also helps with debugging and makes the code self documenting
  // the implementation is TableDef or SessionTableDef -- we shall consider whether it is a good idea to have distinct classes for these
  SchemaBuilder(String tableName, Class<T> implementationClass)

  // builds the tabledef used in the Vuu module, and configured using the fluent interface below
  // links fields and indices can be further customized by overriding protected methods  
  T build(boolean autoSubscribe)

  // creates a row from a map, validating the names of the keys correspond to configured columns
  // we do not require all columns to be populated, and we don't validate the data types (vuu tables do some conversion)
  RowWithData createRow(Map<String, Object> data)

  // allows clients of the code to assert that a column exists - returns `columnName` or throws exception
  String check(String columnName)

  /////  Builder interface used to define the actual data schema ///////////////////////////////////

  // all these methods delegate to `simpleColumn(name, type)`
  SchemaBuilder<T> rowKey(String name) // adds a `String` column and marks it as a primary key (must be called exactly once before `build()`)
  SchemaBuilder<T> dataText(String name) // adds a `String` column 
  SchemaBuilder<T> dataChar(String name) // adds a `char` column 
  SchemaBuilder<T> dataLong(String name) // adds a `long` column 
  SchemaBuilder<T> dataInt(String name) // adds a `int` column 
  SchemaBuilder<T> dataDouble(String name) // adds a `double` column 
  SchemaBuilder<T> simpleColumn(String name, Class<?> type) // adds whatever you specify, but all supported types are above

  // the following methods add metadata to the last added column
  SchemaBuilder<T> indexed()    // creates index for the last added column
  SchemaBuilder<T> pk() // marks a column as primary key in a join 
  SchemaBuilder<T> fkTo(String tableName, String pkColumn) // marks a column as a foreign key to a table with the specified name, having primary key as specified
  SchemaBuilder<T> fkTo(String tableName) // marks a column as a foreign key to a table with the specified name, having primary key named same as the current column 


  SchemaBuilder<T> join(SchemaBuilder<T> schema, String pkColumn) // marks a column as a foreign key to the specified schema, having primary key as specified
  SchemaBuilder<T> join(SchemaBuilder<T> schema) // marks a column as a foreign key to the specified schema, having primary key named same as the current column 
}

This API allows for concise, yet expressive table definitions as follows:

    static final SchemaBuilder<TableDef> APPLICATION_SCHEMA = vuuSchemaBuilder("Applications")
            .rowKey("Application")        // name and version of the startup params record
            .dataText("AppName")          // name of the application
            .dataText("JarFile")          // relative path to the application classpath
            .dataChar("MainClass")        // the entrypoint of the application (must have standard main() method)
            ;

    static final SchemaBuilder<TableDef> INSTANCE_SCHEMA = vuuSchemaBuilder("AppInstances")
            .rowKey("id")                 // the qualified name of the process 
            .dataText("state")            // ACTIVE, PASSIVE or UNKNOWN
            .dataText("timestamp")        // the instance-provided timestamp from the hearbeat message
            .dataText("received")         // the time the current state was received
            .dataText("lastSeen")         // the time of the last message from the instance 
            .dataText("lastLog")          // last log message
            .dataText("Notes")            // user-note freeeform (backed by varchar, so longer than the other fields)

            .dataText("Application").join(APPLICATION_SCHEMA)
            ;

Then later we can use these public schema builders to create rows (and if we mistype any of the columns we will get an easy to diagnose error):

    RowWithData applicationsSnapshot(JsonObject json) {
        return APPLICATION_SCHEMA.createRow(Map.of(
                        "Application", ReferenceData.applicationKey(json),
                        "AppName", json.getString("AppName"),
                        "VersionNumber", json.getString("VersionNo"),
                        "AppDesc", json.getString("AppDesc"),
                        "Disabled", json.getBoolean("Disabled"),
                        "JarFile", requireNonNullElse(json.getString("JarFile"), "[unmanaged]"),
                        "MainClass", requireNonNullElse(json.getString("MainClassPath"), "[unmanaged]"),
                        "LastUpdateDt", json.getString("LastUpdDt"),
                        "ApplicationType", requireNonNullElse(json.getString("ApplicationType"), ""),
                        "LastUseDt", requireNonNullElse(json.getString("LastUsedDt"), "")
                ));
    }

Possible improvements:

• Separate the schema configuration and factory interfaces - add a done() to transition from configuration to creation lifecycle phase
• Consider a garbage free way for specifying rows - right now the scala code is map-based and copies quite a bit, so will require deeper refactoring
• Reconsider whether we really want to distinguish between TableDef and SessionTableDef - right now they do the same and are handled differently based on pattern matching. Can we make it a field instead of adding a new type? The generics make it uglier than needed.
• We should look at adding calculated columns, and see if we need to extend it to specify column state (grouping/treeing, sorting, rendering hints)
• We need better tests

[heswell]

I can imagine a future state of Vuu where we could drive code like the above from a purely declarative representation of the table definitions - a json schema for example. I see numerous benefits on the UI side for such a feature.

A source of frustration for UI devs on early projects has been periods of server downtime / missing data on Vuu instances that block UI development. To help mitigate this a contribution has been initiated (#849) to provide a client side datasource that implements a working subset of Vuu functionality - sorting, filtering, grouping, aggregations. This also has a facility for quickly generating realistic test data (using Faker.js). As a drop-in replacement for the Vuu-connected RemoteDataSource, this allows UI devs to switch to local data for development when remote data is unavailable.

In addition to this, new work on the Vuu UI includes a comprehensive table configuration tool - each column of a table can be configured with data type specifics, formatting suggestions etc.

If we used a json document to describe a table definition, this could be shared on the client to guarantee that when client side tables are constructed, they are generated to exactly the same spec as the server tables. This could be taken even further - using the new UI tools to configure a table, we could create an admin tool that allows a table definition to be created with a nice UI, that could be immediately generated as a client table and tested . The json definition created this way could then be exported and used to generate the tables on the server.

Would anything in the implementation proposal above prevent us from this kind of approach it we wanted to pursue this.

[ddimtirov]

The data-declarative approach can be combined with the builder by adding extra configuration method that accepts a file or a stream.

There is still value of keeping the builder API in addition to the config file, as we already have cases where the data schema is not fully known at build time and we need to dynamically tweak it at startup time (command params session tables).

A richer column model can be accommodated, but I'd be concerned of coupling the evolution of the GUI configuration with a backend API. Perhaps it makes more sense for the server side API to provide a tag ("class" in CSS terms) and the UI framework to define the visual aspects entirely in the front end.

If we want to customize what would be the display attributes for each column type, I suggest that should be a separate server API that would configure this with application-wide scope. Also I expect that the server configuration will be treated as defaults, and any config specified in the UI layer would override it, in turn overridden by client-specific config.

[chrisjstevo]

I like this approach. I especially like the ability to create rows from the schema.

One addition I'd like to do is to hide the immutable map inside the createRow impl. As this would give us the flexibility to change the underlying data structure as we want to.

I wonder if there is a way to address the fields using the schema rather than using the strings?

    RowWithData applicationsSnapshot(JsonObject json) {
        return APPLICATION_SCHEMA.createRow(Map.of(
                        "Application", ReferenceData.applicationKey(json),
                        "AppName", json.getString("AppName"),
                        "VersionNumber", json.getString("VersionNo"),
                        "AppDesc", json.getString("AppDesc"),
                        "Disabled", json.getBoolean("Disabled"),
                        "JarFile", requireNonNullElse(json.getString("JarFile"), "[unmanaged]"),
                        "MainClass", requireNonNullElse(json.getString("MainClassPath"), "[unmanaged]"),
                        "LastUpdateDt", json.getString("LastUpdDt"),
                        "ApplicationType", requireNonNullElse(json.getString("ApplicationType"), ""),
                        "LastUseDt", requireNonNullElse(json.getString("LastUsedDt"), "")
                ));
    }

    RowWithData applicationsSnapshot(JsonObject json) {
        return APPLICATION_SCHEMA.newRow()
                         .Application("foo")
                         .AppName("bar")
                         .VersionNumber(1234)
                        //etc..
                ));
    }

[ddimtirov]

We can do it by alternative API - instead of a schema builder you can define a row interface, with setter for each field and annotations for metadata such as PK and indexes.

interface RowFactory {
   RowWithData create();
}
interface ApplicationRow extends RowFactory {
   @RowKey
   ApplicationRow application(String value);
   @Indexed
   ApplicationRow appName(String value);
   ApplicationRow versionNumber(String value);

}

While it has the advantage of eliminating strings altogether, and providing IDE autocompletion, it also has a few disadvantages:

• Does not allow for columns discovered at runtime (think session tables of admin commands)
• Significantly more machinery - we need reflection or (byte)code generator to implement the interface
• I like being able to see directly the name mappings
• Joins are not natural to express (and any other relationships that cannot be encoded as constants)
• Multi-field indexes and constraints are not natural to express

This is quite far from the current design and if this is the direction we want, there is not much to reuse - we would need to do another PoC (perhaps a spike in the same application).

---

On a separate note, if we would like zero-garbage interface for row creation, we can still use the string constants (which are interned and pose no overhead), and a row builder as follows:

    RowWithData applicationsSnapshot(JsonObject json) {
        return APPLICATION_SCHEMA.startRow()
                         .value("Application", "foo")
                         .value("AppName", "bar")
                         .value("VersionNumber", 1234)
                          ...
                         .buildRow();
    }

The name to index mapping can be done by linear probing hash table (there are many available). Furthermore, we can have separate internal arrays for object and primitive value types, to avoid boxing (though I believe the current scala implementation would box).

The row builder object can be pooled internally in a thread-local (released after buildRow()).

We may add a different mehtod to create a detached row builder (i.e. to build multiple rows at the same time, or to avoid restating values if one needs to update many rows with small differences), but I'd rather wait for such a requirement to materialize before I put it in code.

[naleeha]

Summarising requirements for Table definition

• Need concise api for declaring a table schema
• Any update/addition to table row need to enforce or validate that value type matches the column type - i.e. cannot add String to column defined as Int. Need to handle if we only want to update few of the columns if it is update to existing row. API need to concise and self enforcing as possible to prevent any possible human error
  • When data comes from data source - i.e. in the Providers, can be sourced from csv, rest endpoint etc etc
  • When data comes from user input - i.e. rcp calls such as cell/row updates etc
• Can serialise/deserialise to right type when sent between UI and Server e.g. handle the fact javascript type has Number but Scala type has Int / Long / Double etc. Currently we send data as json
• De-couple the internal construct (immutable map used for row data) from the table api
• Be able to generate schema for UI usage - easy to share the table definition for ui usage and ensure they are in sync with server side
• Can look up/reference the column names and types when we need to map the to external data source schema e.g. data schema in ignite. See Handle when ignite data schema is different from VUU table schema #1167

[chrisjstevo]

I'd personally like to look at (for viability) whether we could code gen to get something which looks like this:

    RowWithData applicationsSnapshot(JsonObject json) {
        return APPLICATION_SCHEMA.newRow()
                         .Application("foo")
                         .AppName("bar")
                         .VersionNumber(1234)
                        //etc..
                ));
    }