Luthier - external configuration loading for fili

[michael-mclawhorn]

This prototype feature will be to build up a file-based configuration for Fili.

Proposed feature roll out stages:

Phase One:

1 - Create a 'Luthier' submodule that is a sample application based on the Wikipedia autowiring
2 - Create Definition classes for Metrics, Makers, Dimensions, Tables (Logical and Physical)
3 - Create a definition reader to read from JSON to create definitions
4 - Create a loader to produce dictionary entries from the definition sources
5 - Enhance luthier example to autowire code to output definition files as a CLI rather than launching directly from autowire

Phase Two:

1 - Enhance definition support to extended maker types, metric types, complex metrics
2 - Create Lua library to produce calculated metrics and extended makers

Phase Three:

1 - Enhance definition support to include extended dimension types (registered lookup support)
2- Enhance definition support to include extensible Table types

[michael-mclawhorn]

Sample of a possible json representation of dimension configuration.

Field sets can be optional if a fieldset named default is defined in the loader. If a field set is specified in the dimension json, it overrides the code default. If a dimension doesn't specify a 'fields' attribute it loads the default fields. If a dimension gives a string value for fields, the code attempts to resolve a field set with that name, and errors if it cannot find one. If the fields attribute is a json list, it must parse with the same structure as a value in the field set object.

{
  "fieldSets": {
    "default": [
      {
        "name": "id",
        "description": "Dimension ID",
        "tags": [
          "primaryKey"
        ]
      },
      {
        "name": "description",
        "description": "Dimension description"
      }
    ],
    "carFields": [
      {
        "name": "id",
        "description": "Dimension ID",
        "tags": [
          "primaryKey"
        ]
      },
      {
        "name": "make",
        "description": "Manufacturer"
      },
      {
        "name": "model",
        "description": "Model of a car"
      }
    ]
  },
  "dimensions": [
    {
      "name": "dimension1",
      "description": "a dimension implicitly using default fields by leaving the fields parameter out"
    },
    {
      "name": "cars",
      "description": "a dimension explicitly using the car dimension field set by name",
      "fields": "carfields"
    },
    {
      "name": "customLocationDimension",
      "description": "a dimension explicitly using a custom field definition",
      "fields": [
        {
          "name": "id",
          "description": "Geolocation Code",
          "tags": [
            "primaryKey",
            "geoType"
          ]
        },
        {
          "name": "county",
          "description": "County of location"
        },
        {
          "name": "state",
          "description": "State of location"
        },
        {
          "name": "country",
          "description": "Country of location"
        }
      ]
    }
  ]
}

[QubitPi]

@michael-mclawhorn What is the planed mechanism of deserialization via Jackson? Are fieldSet and a dimension, etc, gonna be deserialized to their corresponding objects and load them into *Dictionary

[ForestCold]

Sample of a possible json representation of metric configuration.

Makers can specify parameters (no need to set MetricMakerDictionary and DimensionDictionary), Luthier would find appropriate constructor depending on these declared parameters. If more than one constructor meets requirement, it would choose the one that use most declared parameters.

Dependency metrics can be either columns or other defined metrics, the number and type of dependency metrics should meet the requirement of maker.

{
  "makers": [
    {
      "name" : "makerNameOne",
      "class" : "full/path/to/class",
      "params" : {
        "paramName1" : "paramValue1",
        "paramName2" : "paramValue2"
      }
    },
    {
      "name" : "makerNameTwo",
      "class" : "full/path/to/class",
      "params" : {
        "paramName1" : "paramValue1",
        "paramName2" : "paramValue2"
      }
    },
    {
      "name" : "makerHasNoParams",
      "class" : "full/path/to/class"
    }
   ],
  "metrics": [
    {
      "apiName": "metric's api name",
      "longName": "metric's long name",
      "description": "Description for metric",
      "maker": "metric's maker",
      "dependencyMetricNames": [
        "dependency metric one",
        "dependency metric two"
      ]
    },
    {
      "apiName": "metric's api name",
      "longName": "metric's long name",
      "description": "Description for metric",
      "maker": "metric's maker",
      "dependencyMetricNames": [
        "dependency metric one"
      ]
    },
    {
      "apiName": "metric's api name",
      "longName": "metric's long name",
      "description": "Description for metric",
      "maker": "metric's maker"
    }
 ]
}

[ForestCold]

Sample of a possible json representation of table configuration.

A group of physical tables and logical tables, each logical table can depends on more than one physical tables (table group), api metrics defined in logical table should only depends on physical metrics that exist in corresponding physical table group.

{
  "physicalTables": [
    {
      "name": "physical table one",
      "description": "description for table",
      "metrics": [
        "metric one",
        "metric two"
      ],
      "dimensions": [
        "dimension one",
        "dimension two"
      ],
      "granularity": "DAY/HOUR/ALL"
    },
    {
      "name": "physical table two",
      "description": "description for two",
      "metrics": [
        "metric one",
        "metric two"
      ],
      "dimensions": [
        "dimension one",
        "dimension two",
        "dimension three"
      ],
      "granularity": "DAY/HOUR/ALL"
    }
  ],
  "logicalTables": [
    {
      "name": "logical table one",
      "description": "description for logical table one",
      "apiMetricNames": [
        "api metric one",
        "api metric two"
      ],
      "physicalTables": [
        "physical table one",
        "physical table two"
      ],
      "granularity": ["ALL", "HOUR", "DAY"]
    },
      {
      "name": "logical table two",
      "description": "description for logical table two",
      "apiMetricNames": [
        "api metric one",
        "api metric two"
      ],
      "physicalTables": [
        "physical table one"
      ],
      "granularity": ["HOUR", "DAY"]
      }
    ]
}