Unable to use Delta Kusto reliably to maintain ADX tables and mappings

[delarooster]

I'm attempting to utilize this project to help put my ADX tables under source control and am failing miserably. I'm starting from the presupposition of initializing a brand-new database from scatch without any preexisting schema or data.

There is a script file that defines two tables. Full repository here -> Io.Adx.Kusto

1. Currently my pipeline is set up to look at the existing database and retrieve current state (as a potential rollback later, if required).

2. I then compare my schema script against the existing database to create a delta.

3. As the documentation suggests, I publish these files as an artifact to be retrieved by the next step.

4. In the next stage I retrieve the published artifact and attempt an update of the ADX database, using the published artifact as the source.

A few observations:

1. Is there a documented best-practice for passing in the desired state of a database via a script file? I had originally included multiple script files (one for each table, I've since reduced the tables to two, removed mappings, policies, to help alleviate questions while debugging) and in my latest iteration put both table .create functions into a single file.
2. During the push-state job it appears the current and target are flip-flopped parameters:

    push-state:
        current:
            adx:
                clusterUri: to-be-overriden
                database: to-be-overriden
        target:
            scripts:
                - filePath: ../schema/script.kql

Throughout the documentation I read that current is the source and target the desired instance to update from current, but here whenever I set it up as current as a file (either my schema or the created delta file), I continually get this error:

Error:  Issue with the following parameter override:  'System.String[]'
  Exception encountered:  System.Collections.Generic.KeyNotFoundException ; The given key 'DeltaKustoIntegration.Parameterization.AdxSourceParameterization' was not present in the dictionary.

3. Similar to point 2 above when running delta-dev I use my scripts as the current and target as the adx database I want to initialize, but it appears the program is attempting to compare what's in the ADX db as the source of truth instead of my scripts.

  delta-dev:
    priority: 1
    current:
        scripts:
            - filePath: ../schema/script.kql
    target:
        adx:
            clusterUri:  to-be-overridden
            database:  to-be-overridden

Delta output is as follows:

//  Drop Tables

.drop table Connectivity

.drop table Telemetry

//  Alter Caching Policies

.alter database Dvusahoth policy caching hot = 31d

//  Alter Merge Policies

.alter database Dvusahoth policy merge
{
  "RowCountUpperBoundForMerge": 16000000,
  "OriginalSizeMBUpperBoundForMerge": 30000,
  "MaxExtentsToMerge": 100,
  "LoopPeriod": "01:00:00",
  "MaxRangeInHours": 24,
  "AllowRebuild": true,
  "AllowMerge": true,
  "Lookback": {
    "Kind": 0,
    "CustomPeriod": null
  },
  "Origin": 2
}

//  Alter Retention Policies

.alter database Dvusahoth policy retention
{
  "SoftDeletePeriod": "365.00:00:00",
  "Recoverability": "Enabled"
}

Appears to be the exact opposite of what I'm looking to create. And if I attempt to flip (like in push-state-to-db-parameters.yaml) I get the same error as point 2.

4. The only thing that does currently work is the first time I run a setup on a bare database using my desired schema:

sendErrorOptIn: true
jobs:
    push-state:
        current:
            adx:
                clusterUri: to-be-overriden
                database: to-be-overriden
        target:
            scripts:
                - filePath: ../schema/script.kql
        action:
            pushToCurrent: true
            pushToConsole: true

Adding to a fresh table works just fine, updating when there is a table mapping that previously exists, fails miserably.
See:

Compute Delta...
Exception encountered:  System.Text.Json.JsonException ; Expected end of string, but instead reached end of data. Path: $[1].Properties.Path | LineNumber: 0 | BytePositionInLine: 116.
Stack trace:     at System.Text.Json.ThrowHelper.ReThrowWithPath(ReadStack& , JsonReaderException)
   at System.Text.Json.Serialization.JsonConverter`1.ReadCore(Utf8JsonReader&, JsonSerializerOptions, ReadStack& )
   at System.Text.Json.JsonSerializer.ReadFromSpan[TValue](ReadOnlySpan`1, JsonTypeInfo, Nullable`1 )
   at System.Text.Json.JsonSerializer.ReadFromSpan[TValue](ReadOnlySpan`1, JsonTypeInfo)
   at System.Text.Json.JsonSerializer.Deserialize[TValue](String , JsonSerializerOptions )
   at DeltaKustoLib.KustoModel.MappingModel.MappingAsJsonEquals(QuotedText otherMappingAsJson)
   at DeltaKustoLib.DeltaHelper.GetUpdated[M,K](IEnumerable`1 currents, IEnumerable`1 targets, Func`2 keyExtractor)+MoveNext()
   at System.Linq.Enumerable.SelectEnumerableIterator`2.MoveNext()
   at System.Linq.Enumerable.ConcatIterator`1.MoveNext()
   at System.Linq.Enumerable.SelectEnumerableIterator`2.MoveNext()
   at System.Linq.Enumerable.ConcatIterator`1.MoveNext()
   at DeltaKustoLib.KustoModel.TableModel.ComputeDelta(TableModel targetModel)+MoveNext()
   at System.Linq.Enumerable.SelectManySingleSelectorIterator`2.MoveNext()
   at System.Linq.Enumerable.ConcatIterator`1.MoveNext()
   at System.Collections.Generic.LargeArrayBuilder`1.AddRange(IEnumerable`1)
   at System.Collections.Generic.SparseArrayBuilder`1.ReserveOrAdd(IEnumerable`1)
   at System.Linq.Enumerable.ConcatNIterator`1.LazyToArray()
   at System.Collections.Immutable.ImmutableArray.CreateRange[T](IEnumerable`1)
   at DeltaKustoLib.KustoModel.DatabaseModel.ComputeDelta(DatabaseModel targetModel)
   at delta_kusto.DeltaOrchestration.ProcessJobAsync(MainParameterization parameters, IKustoManagementGatewayFactory kustoGatewayFactory, IFileGateway localFileGateway, String jobName, JobParameterization job)
   at delta_kusto.DeltaOrchestration.ComputeDeltaAsync(String parameterFilePath, IEnumerable`1 pathOverrides, String sessionId)
   at delta_kusto.Program.RunOptionsAsync(CommandLineOptions options, String sessionId)
   at delta_kusto.Program.<>c__DisplayClass2_0.<<Main>b__2>d.MoveNext()
--- End of stack trace from previous location ---
   at CommandLine.ParserResultExtensions.WithParsedAsync[T](ParserResult`1 result, Func`2 action)
   at delta_kusto.Program.Main(String[] args)
  Exception encountered:  System.Text.Json.JsonReaderException ; Expected end of string, but instead reached end of data. LineNumber: 0 | BytePositionInLine: 116.
  Stack trace:     at System.Text.Json.ThrowHelper.ThrowJsonReaderException(Utf8JsonReader&, ExceptionResource, Byte , ReadOnlySpan`1 )
   at System.Text.Json.Utf8JsonReader.ConsumeString()
   at System.Text.Json.Utf8JsonReader.ConsumeValue(Byte)
   at System.Text.Json.Utf8JsonReader.ReadSingleSegment()
   at System.Text.Json.Utf8JsonReader.Read()
   at System.Text.Json.Serialization.Converters.ObjectDefaultConverter`1.OnTryRead(Utf8JsonReader&, Type, JsonSerializerOptions, ReadStack& , T& )
   at System.Text.Json.Serialization.JsonConverter`1.TryRead(Utf8JsonReader&, Type, JsonSerializerOptions, ReadStack& , T& )
   at System.Text.Json.Serialization.Metadata.JsonPropertyInfo`1.ReadJsonAndSetMember(Object, ReadStack& , Utf8JsonReader&)
   at System.Text.Json.Serialization.Converters.ObjectDefaultConverter`1.OnTryRead(Utf8JsonReader&, Type, JsonSerializerOptions, ReadStack& , T& )
   at System.Text.Json.Serialization.JsonConverter`1.TryRead(Utf8JsonReader&, Type, JsonSerializerOptions, ReadStack& , T& )
   at System.Text.Json.Serialization.JsonCollectionConverter`2.OnTryRead(Utf8JsonReader&, Type, JsonSerializerOptions, ReadStack& , TCollection& )
   at System.Text.Json.Serialization.JsonConverter`1.TryRead(Utf8JsonReader&, Type, JsonSerializerOptions, ReadStack& , T& )
   at System.Text.Json.Serialization.JsonConverter`1.ReadCore(Utf8JsonReader&, JsonSerializerOptions, ReadStack& )

Hoping to use this tool, or another, to manage my ADX databases using source control. Big big use case that would be highly beneficial to my team!

Many thanks,
Austin

[vplauzon]

After discussion, the mapping error was due to a regression introduced in .show db schema (Kusto command used by DK) that has since been resolved.