[Enh]: Support for `@file()`

[JerryNixon]

What?

In the same way @env() and @akv() allow for string replacements in the DAB configuration, introduce @file() which will read from a designated JSON file with a simple name/value dictionary <string, string>.

Configuration

{
  "metadata-file": {
    "path": "./metadata.json"
  }
}

Rules

These rules govern the implementation.

1. Any property name or value can be set using @file().
2. The @file() function is a string replacement in the same way @env() and @akv() are.
3. The metadata file cannot have duplicate keys (this is a core feature of a dictionary already).
4. The metadata file is read before configuration parsing and is not hot-reloaded.

Order of operation

For string replacement in the configuration:

1. @env() is first and optional.
2. @akv() is second and optional.
3. @file() is third and optional.

sequenceDiagram
  actor Engine as Engine
  participant ConfigInMem as ConfigInMem
  participant Environment as Environment
  participant AKV as AKV
  participant File as MetadataFile
  participant Config as ConfigFile

  Engine ->> Engine: Pre-Start
  Engine ->> Config: Load Config
  Config -->> Engine: Config Data
  Engine ->> ConfigInMem: Create In-Memory Config

  Note over Engine: Perform Config Replacements

  activate Engine
  ConfigInMem -->> Engine: Parse @env Values
  Engine ->> Environment: Get
  Environment -->> Engine: Values
  Engine ->> ConfigInMem: Replace @env Values
  deactivate Engine

  Note over Engine: Use Config With ENV Replacements

  activate Engine
  ConfigInMem -->> Engine: Parse @akv Values
  Engine ->> AKV: Request
  AKV -->> Engine: Secrets
  Engine ->> ConfigInMem: Replace @akv Values
  deactivate Engine

  Note over Engine: Use Config With AKV Replacements

  activate Engine
  ConfigInMem -->> Engine: Parse @file Values
  Engine ->> File: Read
  File -->> Engine: Key/Value Pairs
  Engine ->> ConfigInMem: Replace @file Values
  deactivate Engine

  Note over Engine: Use Config With FILE Replacements

  Engine ->> Engine: Start

Loading

Now: this means metadata-file.path could be set using @env() or @akv if the developer wants to.

JSON schema

1. metadata-file is an optional property of type object.
2. metadata-file.path is an optional property of type string.
3. There are no schema-level constraints for these properties.

Command line

Configuration

1. Add dab configure --metadata-file.path './file.json'.

Validation & errors

These rules are implemented in dab validate and dab start:

1. If @file() is present in the configuration, then metadata-file.path is required.
2. If metadata-file.path is set using @file(), then throw an exception.
3. If metadata-file.path does not point to an existing file, then throw an exception.
4. If metadata-file.path does not contain valid JSON with a <string, string> dictionary, then throw an exception.
5. If a @file(key) reference is not found in the metadata file, then throw an exception.
6. If metadata-file.path is defined but no @file(key) references are present, then do not throw an exception.

Metadata file format

This format is intentionally as simple as possible.

{
  "key1": "value",
  "key2": "value"
}

Configuration sample

This sample shows how both names and values can be replaced using @file().

{
  "metadata-file": {
    "path": "./metadata.json"
  },
  "entities": {
    "@file('dbo.Users.Alias')": {
      "description": "@file('dbo.Users.MS_Description')",
      "source": {
        "object": "dbo.User",
        "type": "table"
      }
    }
  }
}

Consideration

1. Include an OTEL activity wrapping the replacement.
2. (Would be nice) Include an OTEL activity wrapping the replacement for @env() and @akv().