Handle when ignite data schema is different from VUU table schema #1167

naleeha · 2024-01-29T11:33:08Z

Currently done using a simple map for sorting - but the name of the columns on ignite and vuu table is hard coded. This needs to referenced from the ignite data schema and VUU table schema to avoid them getting out of sync with whats specified in the map
Need to support when ignite and vuu column names or the column types differ for filtering. Currently only works if they are the same

Places where we currently need to know the Ignite field names/types are

Ignite configuration for ignite server side to specify the schema - can be specified when server start up or at run time.
When we read the ignite data from the sql field query (see toChildOrder() https://github.com/finos/vuu/blob/main/example/apache-ignite/src/main/scala/org/finos/vuu/example/ignite/IgniteOrderStore.scala)
When we update the table rows with data fetched from ignite (https://github.com/finos/vuu/blob/main/example/apache-ignite/src/main/scala/org/finos/vuu/example/ignite/provider/IgniteOrderDataProvider.scala)
When we need to translate sort + filter specified on the table to ignite data sort and filter (see ColumnMap in https://github.com/finos/vuu/blob/main/example/apache-ignite/src/main/scala/org/finos/vuu/example/ignite/provider/IgniteOrderDataProvider.scala)

naleeha · 2024-01-31T15:34:00Z

Currently we are experimenting with different ways of specifying the data (entity) schema on Ignite server side.

Using QueryEntity to define the fields + indexes https://github.com/finos/vuu/blob/main/plugin/ignite-plugin/src/test/scala/org/finos/vuu/feature/ignite/IgniteTestStore.scala
Using Dto object

Looks like the fields defined using QueryEntity has api for getting the field name and values
https://ignite.apache.org/releases/latest/dotnetdoc/api/Apache.Ignite.Core.Cache.Configuration.QueryEntity.html#Apache_Ignite_Core_Cache_Configuration_QueryEntity_ValueFieldName

naleeha · 2024-01-31T15:34:38Z

It would be good if the table schema we define for this can be used for all other usecases #853 (comment)

Started some spikes for ideas here https://github.com/naleeha/vuu/tree/table_ignite_schema_spike

- uses SchemaMapper to build the schema mappings b/w external and internal. - Sort + Filters to use the mapper to build SQL queries instead of internal table def.

- uses SchemaMapper to build the schema mappings b/w external and internal. - Sort + Filters to use the mapper to build SQL queries instead of internal table def. - refactors and improves the flow of IgniteOrderDataProvider.

- uses SchemaMapper to build the schema mappings b/w external and internal. - potential design/performance tradeoff: inside IgniteOrderStore.findChilOrder method toChildOrder can be a performance hit, can keep as raw list as we convert back to a list anyways when mapping to row data. - Sort + Filters to use the mapper to build SQL queries instead of internal table def. - refactors and improves the flow of IgniteOrderDataProvider.

- convert map->filter pipeline to simply a flatMap in SchemaMapper - SchemaMapper to take in separate tableColumns & fields map (external->internal) to allow better separation of concerns around DataModule & DataProvider. - clean-up validation logic for SchemaMapper & IgniteEntitySchemaBuilder - Small refactoring in DataProvider to define vals for HOFs - ExternalStoreEntitySchema to use val instead of def since we're assuming that schemaFields will stay the same once schema's instantiated.

- both does essentially the same lazy eval, the difference is that the Iterator is only for single traversal whereas the other memoises the results. Using Iterator as its simpler and works for our use-cases.

) * #1167 handle when external data schema is different from internal - uses SchemaMapper to build the schema mappings b/w external and internal. - potential design/performance tradeoff: inside IgniteOrderStore.findChilOrder method toChildOrder can be a performance hit, can keep as raw list as we convert back to a list anyways when mapping to row data. - Sort + Filters to use the mapper to build SQL queries instead of internal table def. - refactors and improves the flow of IgniteOrderDataProvider. * #1167 suggestions from PR review + clean-ups - convert map->filter pipeline to simply a flatMap in SchemaMapper - SchemaMapper to take in separate tableColumns & fields map (external->internal) to allow better separation of concerns around DataModule & DataProvider. - clean-up validation logic for SchemaMapper & IgniteEntitySchemaBuilder - Small refactoring in DataProvider to define vals for HOFs - ExternalStoreEntitySchema to use val instead of def since we're assuming that schemaFields will stay the same once schema's instantiated. * #1167 use Iterator instead of LazyList for lazy eval - both does essentially the same lazy eval, the difference is that the Iterator is only for single traversal whereas the other memoises the results. Using Iterator as its simpler and works for our use-cases.

- allows for cleaner flow of code

…invalid indexes

…pper validation

- IgniteEntitySchema's invalid index exception to check for and include all invalid indexes - SchemaMapper validation to not check for missing columns in fields map since there's a use-case (basket constituents) where it makes sense to not specify all internal columns in the fields map.

- allows for cleaner flow of code

- IgniteEntitySchema's invalid index exception to check for and include all invalid indexes - SchemaMapper validation to not check for missing columns in fields map since there's a use-case (basket constituents) where it makes sense to not specify all internal columns in the fields map.

- uses SchemaMapper to build the schema mappings b/w external and internal. - potential design/performance tradeoff: inside IgniteOrderStore.findChilOrder method toChildOrder can be a performance hit, can keep as raw list as we convert back to a list anyways when mapping to row data. - Sort + Filters to use the mapper to build SQL queries instead of internal table def. - refactors and improves the flow of IgniteOrderDataProvider. * finos#1167 suggestions from PR review + clean-ups - convert map->filter pipeline to simply a flatMap in SchemaMapper - SchemaMapper to take in separate tableColumns & fields map (external->internal) to allow better separation of concerns around DataModule & DataProvider. - clean-up validation logic for SchemaMapper & IgniteEntitySchemaBuilder - Small refactoring in DataProvider to define vals for HOFs - ExternalStoreEntitySchema to use val instead of def since we're assuming that schemaFields will stay the same once schema's instantiated. * finos#1167 use Iterator instead of LazyList for lazy eval - both does essentially the same lazy eval, the difference is that the Iterator is only for single traversal whereas the other memoises the results. Using Iterator as its simpler and works for our use-cases.

- allows for cleaner flow of code

- IgniteEntitySchema's invalid index exception to check for and include all invalid indexes - SchemaMapper validation to not check for missing columns in fields map since there's a use-case (basket constituents) where it makes sense to not specify all internal columns in the fields map.