Decoding error when events spill into new dynamo row

[Akash-Mair]

Hi guys!

I'm hoping this is a "me" issue and I'm lacking some config or haven't set something up correctly.

When I am running the eqx dump on my aggregate I am running into:

System.InvalidOperationException: Decoder ran into invalid data.
   at System.IO.Compression.BrotliStream.TryDecompress(Span`1 destination, Int32& bytesWritten)
   at System.IO.Compression.BrotliStream.Read(Span`1 buffer)
   at System.IO.Compression.BrotliStream.Read(Byte[] buffer, Int32 offset, Int32 count)
   at System.IO.Stream.CopyTo(Stream destination, Int32 bufferSize)
   at FsCodec.Impl.brotliDecompressTo(Stream output, ReadOnlyMemory`1 data) in /_//src/FsCodec/Encoding.fs:line 34
   at FsCodec.Impl.decode@42-1.Invoke(MemoryStream output, ReadOnlyMemory`1 data)
   at FsCodec.Impl.unpack[a](FSharpFunc`2 alg, a compressedBytes) in /_//src/FsCodec/Encoding.fs:line 37
   at FsCodec.Impl.decode(ValueTuple`2 _arg1) in /_//src/FsCodec/Encoding.fs:line 42
   at FsCodec.Encoding.ToBlob(ValueTuple`2 x) in /_//src/FsCodec/Encoding.fs:line 77
   at <StartupCode$FsCodec>.$Encoding.Compressed@93-1.Invoke(ValueTuple`2 x)
   at <StartupCode$FsCodec>.$FsCodec.MapBodies@125-3.FsCodec.IEventData<'Mapped>.get_Data() in /_//src/FsCodec/FsCodec.fs:line 131
   at Equinox.Tool.Program.Dump.dumpEvents@829-5.Invoke(ITimelineEvent`1 x) in /_//tools/Equinox.Tool/Program.fs:line 840
   at Microsoft.FSharp.Control.AsyncPrimitives.CallThenInvoke[T,TResult](AsyncActivation`1 ctxt, TResult result1, FSharpFunc`2 part2) in D:\a\_work\1\s\src\FSharp.Core\async.fs:line 508
   at Microsoft.FSharp.Control.Trampoline.Execute(FSharpFunc`2 firstAction) in D:\a\_work\1\s\src\FSharp.Core\async.fs:line 112
--- End of stack trace from previous location ---
   at Microsoft.FSharp.Control.AsyncResult`1.Commit() in D:\a\_work\1\s\src\FSharp.Core\async.fs:line 453
   at Microsoft.FSharp.Control.AsyncPrimitives.QueueAsyncAndWaitForResultSynchronously[a](CancellationToken token, FSharpAsync`1 computation, FSharpOption`1 timeout) in D:\a\_work\1\s\src\FSharp.Core\async.fs:line 1133
   at Microsoft.FSharp.Control.AsyncPrimitives.RunSynchronously[T](CancellationToken cancellationToken, FSharpAsync`1 computation, FSharpOption`1 timeout) in D:\a\_work\1\s\src\FSharp.Core\async.fs:line 1160
   at Microsoft.FSharp.Control.FSharpAsync.RunSynchronously[T](FSharpAsync`1 computation, FSharpOption`1 timeout, FSharpOption`1 cancellationToken) in D:\a\_work\1\s\src\FSharp.Core\async.fs:line 1504
   at Equinox.Tool.Program.main(String[] argv) in /_//tools/Equinox.Tool/Program.fs:line 884

I have a minimal repo setup here: https://github.com/Akash-Mair/eqx-tipping-point

Some notes:

• I used the eqx tool to create my dynamo table
• The issue seems to be when the events spill over into a new row

[bartelink]

Hi Akash, thanks for logging this (and especially for creating a repro).

I've not personally used DynamoStore in recent times so first off, some things to note:

• I'm 99.9% sure the tests cover the normal overflow case (i.e. this is not some new hack/trick or something that rarely comes into play - it's mainline functionality)
• CosmosStore and DynamoStore both use the same scheme of storing a D/M int representing the encoding scheme alongside a d/n blob associated with it (see EventSchema/UnfoldSchema). These values get roundtripped into the stored row and then feed into the decoding call that's failing. Nothing much has changed in recent times in the DynamoStore impl (CosmosStore has had the identical approach rolled out in recent times as it just makes sense).
• FsCodec's Brotli support has had plenty prod usage with multiple stores, but it's always possible there's a problem in there and/or some variance in behavior due to the underlying .NET impl or its configuration.

So figuring out the problem boils down to figuring out whether the encoding value is wrong, or the body is wrong/mangled/mismatched. The fact that it's trying to decode from Brotli means that D or M yielded 2. Next question is whether the d/m value is a valid Brotli blob. In general I'd sanity check that by inspecting the underlying data. (I don't have a test rig to hand and am not sure when I'll get around to looking atm)

Looking at the package dependencies, the fact you're using AWSSDK v4 and the 4.2 RC DynamoStore nuget suggests the recent changes in Fsharp.Aws.DynamoDb may be triggering the issue (though the changes seem innocuous and the underlying library has plenty prod usage for it not to be something basic). Does the problem go away if you downgrade to AWS SDK 3 and DynamoStore 4.1 ? I assume this is from an empty table with no potential interop issues re data produced by older versions (this should not be an issue; but it's one potential factor to exclude). Also I'm not sure the tests on master have been run (the CI does not rig all the simulator and disables the integration tests)

---

Minor nit that doesn't affect the repro: your limit guards are > MaxValue / < MinValue, but should be = and/or account for overflow in the case of a value > 1

[Akash-Mair]

Thanks for the quick reply!
I also should have shared the results of the dynamo data.

Like you said when D is 2 then that indicates to use the Brotli encoding and if you see the second row (very noisy) - here is a snippet:
{""B"":""OiIxOGU0ZGJlZDhlZTU0NGNkYjMyZmJmYzhlZGVlMmY5MyJ9LCJuIjp7Ik4iOiI3MDMifX19I+A=""},""D"":{""N"":""2""}}}

This data is not properly encoded and so runs into the Decoding error. In comparison earlier events look correct Gz0AAATEbame5ScNwqOpCX9VUA0itiSW+OKR5E7Uh0WxnIoy4xr+wuvFOCsmSh87FkZzlz2kjYAP

I have also run into another issue where the first event seems to get overwritten with this base64 encoded error:
eyJDbGllbnRSZXF1ZXN0VG9rZW4iOiJiNTY0Nzg4OC0yYzQ2LTQzYmItOTAzMy1lZWEwODg4YzQwYmQiLCJSZXR1cm5Db25zdW1lZENhcGFjaXR5IjoiSU5ERVhFUyIsIlRyYW5zYWN0SXRlbXMiOlt7IlB1dCI6eyJDb25kaXRpb25FeHByZXNzaW9uIjoiKCBhdHRyaWJ1dGVfbm90X2V4aQ==

{"ClientRequestToken":"b5647888-2c46-43bb-9033-eea0888c40bd","ReturnConsumedCapacity":"INDEXES","TransactItems":[{"Put":{"ConditionExpression":"( attribute_not_exi

Here is a little fsx script I've been using to confirm if the data is properly encoded.

open System
open System.IO
open System.IO.Compression
open System.Text

let decodeBrotliPayload (base64String: string) =
  try
    let compressedBytes = Convert.FromBase64String(base64String)
    
    use inputStream = new MemoryStream(compressedBytes)
    use brotliStream = new BrotliStream(inputStream, CompressionMode.Decompress)
    use outputStream = new MemoryStream()
    
    brotliStream.CopyTo(outputStream)
    
    let decompressedBytes = outputStream.ToArray()
    let json = Encoding.UTF8.GetString(decompressedBytes)

    printfn $"Successfully decoded:\n%s{json}\n"
  with 
  | :? InvalidOperationException as ex ->
    printfn $"Decompression failed (InvalidOperationException): %s{ex.Message}\n"
  | ex ->
      printfn $"Unexpected error: %s{ex.Message}\n"

decodeBrotliPayload "OiIxOGU0ZGJlZDhlZTU0NGNkYjMyZmJmYzhlZGVlMmY5MyJ9LCJuIjp7Ik4iOiI3MDMifX19I+A="

Broken events.csv

Also side question, how is the encoding chosen? Is it to do with how big an event is?
I see that the Subtract events are only base64 encoded eyJkZWNyZWFzZSI6MzM4LCJyZXF1ZXN0ZWRCeSI6InVzZXIifQ==

[bartelink]

The fact that there are encoding errors being saved etc suggests that going back to the previous version is likely to resolve things (on the assumption that the AWS breaking changes in AWSSDK's V3 to V4 update are the root cause). The Equinox-side PR is relatively innocuous, but the underlying one has many more considerations in play). Running the integration test suite against the Docker Compose config in the repo will likely repro the issue too...

---

The encoding scheme is based on the fact that reads and writes are both almost linearly (though its in blocks of xK) tied to the size of the blobs being roundtripped. This is doubly so where the page splits (you pay double the individual cost for a transactionally managed set of requests). For events with bodies of 1K of JSON, compression is a no brainer (which is why its the default in the examples) - for stuff under 100 bytes it'll likely be slow AND make the size bigger.

For simple cases where you don't have large event bodies, this will be way less of a factor (you could switch if off by changing the conditional compression config in your repro to Uncompressed. One clear benefit is not having to interpret/decompress when inspecting the underlying data (as opposed to using eqx dump)

[bartelink]

cc @njlr wondering if you tried the RC and/or ran into similar (I've yet to try it this side)

[njlr]

cc @njlr wondering if you tried the RC and/or ran into similar (I've yet to try it this side)

Some findings:

• Adding Amazon.AWSConfigs.InitializeCollections <- true on program start-up appears to fix the repro project
• Applying these PRs (Adds various null checks required for AWS SDK v4 fsprojects/FSharp.AWS.DynamoDB#88 Fixes bugs in Print with AWS SDK v4 fsprojects/FSharp.AWS.DynamoDB#87) appears to fix the repro project

Afraid I don't know which fix in particular to FSharp.AWS.DynamoDB was the important piece - I just mechanically went through and found every attribute that may be null in v4 and added a check.

[bartelink]

Thanks so much @nljr
@Akash-Mair I now realise that I misspoke above saying

though the changes seem innocuous and the underlying library has plenty prod usage for it not to be something basic
In fact, @samritchie flagged that the FSharp.AWS.DynamoDB changes were causing problems in his context, so some aspects] that happen not to be covered by tests, which is not unreasonable given how many API calls are involved] are impacted by the changes:
fsprojects/FSharp.AWS.DynamoDB#86 (comment)
fsprojects/FSharp.AWS.DynamoDB#86 (comment)

[samritchie]

@bartelink just to clarify,InitializeCollections didn’t help me at all, I had a transitive version incompatibility (that has since been resolved). If it fixes the repro there’s almost certainly a v4 null somewhere though.

[njlr]

Now that FSharp.AWS.DynamoDB version 0.13.1-beta is out, is there a failing test here that we can use?

[bartelink]

I would expect that the normal suite should fail just with the docker-compose file's DDB simulator

If not, @Akash-Mair 's repro should suffice?

If there were cases where the response body is erroneously empty that should hit lots of tests (the issue is that the CI wiring does not set up simulated DBs so only a local test run validates a full release, and I'm 90% sure I skipped that for 4.2.0-rc of Dynamo (there is no 4.2 rc of anything else yet)

[njlr]

I suspect there may be a lack of test coverage here?

Here is what I tried...

1. Checkout master at commit f5859d05.

2. Start the following Docker services:

services:

  dynamodb:
    image: amazon/dynamodb-local:2.4.0
    command: "-jar DynamoDBLocal.jar -sharedDb"
    healthcheck:
      test: [ "CMD-SHELL", "curl -v http://dynamodb:8000" ]
      interval: 3s
      timeout: 3s
      retries: 5
      start_period: 3s
    ports:
      - "8000:8000"
    restart: unless-stopped

  dynamodb-admin:
    image: aaronshaf/dynamodb-admin:4.6.1
    ports:
      - "8001:8001"
    environment:
      DYNAMO_ENDPOINT: "http://dynamodb:8000"
      AWS_REGION: "us-west-1"
      AWS_ACCESS_KEY_ID: abc
      AWS_SECRET_ACCESS_KEY: def
    depends_on:
      dynamodb:
        condition: service_healthy
    restart: unless-stopped

3. Run the Dynamo integration tests:

dotnet test ./tests/Equinox.DynamoStore.Integration                                                                              
Restore complete (1.1s)
  Equinox net6.0 succeeded (0.5s) → src/Equinox/bin/Debug/net6.0/Equinox.dll
  Domain net6.0 succeeded (0.3s) → samples/Store/Domain/bin/Debug/net6.0/Domain.dll
  Equinox.DynamoStore net6.0 succeeded (0.4s) → src/Equinox.DynamoStore/bin/Debug/net6.0/Equinox.DynamoStore.dll
  Equinox.DynamoStore.Integration net8.0 succeeded (0.2s) → tests/Equinox.DynamoStore.Integration/bin/Debug/net8.0/Equinox.DynamoStore.Integration.dll
[xUnit.net 00:00:00.00] xUnit.net VSTest Adapter v2.5.7+8f2703126a (64-bit .NET 8.0.25)
[xUnit.net 00:00:00.09]   Discovering: Equinox.DynamoStore.Integration
[xUnit.net 00:00:00.15]   Discovered:  Equinox.DynamoStore.Integration
[xUnit.net 00:00:00.15]   Starting:    Equinox.DynamoStore.Integration
[xUnit.net 00:00:12.54]   Finished:    Equinox.DynamoStore.Integration
  Equinox.DynamoStore.Integration test net8.0 succeeded (13.3s)

Test summary: total: 13, failed: 0, succeeded: 13, skipped: 0, duration: 13.3s
Build succeeded in 16.5s

4. Confirm versions:

cat ./src/Equinox.DynamoStore/Equinox.DynamoStore.fsproj 

<Project Sdk="Microsoft.NET.Sdk">

  <Sdk Name="Microsoft.DotNet.PackageValidation" Version="1.0.0-preview.7.21379.12" />

  <PropertyGroup>
    <TargetFramework>net6.0</TargetFramework>
    <PackageValidationBaselineVersion>4.0.0</PackageValidationBaselineVersion>
  </PropertyGroup>

  <ItemGroup>
    <Compile Include="..\Equinox\Infrastructure.fs" />
    <Compile Include="..\Equinox.Core\Internal.fs" />
    <Compile Include="DynamoStore.fs" />
  </ItemGroup>

  <ItemGroup>
    <ProjectReference Condition=" '$(Configuration)' == 'Debug' " Include="..\Equinox\Equinox.fsproj" />
    <PackageReference Condition=" '$(Configuration)' == 'Release' " Include="Equinox" Version="[4.0.3, 5.0.0)" />
  </ItemGroup>

  <ItemGroup>
    <PackageReference Include="MinVer" Version="5.0.0" PrivateAssets="All" />

    <PackageReference Include="FSharp.Core" Version="6.0.7" ExcludeAssets="contentfiles" />

    <PackageReference Include="FSharp.AWS.DynamoDB" Version="0.13.0-beta" />
    <PackageReference Include="FSharp.Control.TaskSeq" Version="0.4.0" />
  </ItemGroup>

</Project>

[bartelink]

@njlr Sounds right - perhaps its to do with the specific handling of Events where the Data winds up being null / [||]

IIRC There are various thorny cases on the CosmosDb side which may have specific coverage, but reading them at the time of porting (the initial impl for DDB was based off the Cosmos impl and tests) I might not have figured out that there's a similar case (the fact that Cosmos works off JSON whereas DDB has blobs would not have helped)

It should be possible to define a generic test case in tests/Equinox.CosmosStore.Integration/DocumentStoreIntegration.fs which represents the failure here (its unclear to me what it is about @Akash-Mair's events that make them trigger the problem whereas the standard cases do not -- perhaps it's due to compression coming into play?)

More than happy to have help defining a failing test (I'm doing some work in FsCodec atm and will eventually have time and focus to give to this repo).

I'm also wondering whether FSharp.AWS.DynamoDB v 0.13.1-beta resolves the issue

Either way definitely interested in coverage re the problems with 0.13.0-beta (and the fact that this would also cover more cases on the CosmosStore side)

[njlr]

@njlr Sounds right - perhaps its to do with the specific handling of Events where the Data winds up being null / [||]

Yes this seems likely!

IIRC There are various thorny cases on the CosmosDb side which may have specific coverage, but reading them at the time of porting (the initial impl for DDB was based off the Cosmos impl and tests) I might not have figured out that there's a similar case (the fact that Cosmos works off JSON whereas DDB has blobs would not have helped)

It should be possible to define a generic test case in tests/Equinox.CosmosStore.Integration/DocumentStoreIntegration.fs which represents the failure here (its unclear to me what it is about @Akash-Mair's events that make them trigger the problem whereas the standard cases do not -- perhaps it's due to compression coming into play?)

Is there a simple high volume test?
Write 1000s of events, check that the final state is as expected?

More than happy to have help defining a failing test (I'm doing some work in FsCodec atm and will eventually have time and focus to give to this repo).

I'm also wondering whether FSharp.AWS.DynamoDB v 0.13.1-beta resolves the issue

It does not appear to for the repro above 🤔

Either way definitely interested in coverage re the problems with 0.13.0-beta (and the fact that this would also cover more cases on the CosmosStore side)

👍

[bartelink]

Is there a simple high volume test?
Write 1000s of events, check that the final state is as expected?

No, the integration tests are all real outside in scenarios that were written as failing and then made to pass (especially wrt any of the access strategies specific to each store). Ultimately stores have to be entrusted with the responsibility of roundtripping data. If that's not the case, the first place to consider putting it would be in FSharp.AWS.DynamoDB.

With regard to the specific case here (overflow/calving), that's absolutely a core function of the library so it should have coverage (including validation that it's transactionally correct, i.e. attempting to stage two competing writes that do a split etc with deterministic behavior. Because a split needs to be handled in a worst case where you add a single event to a tip containing only a single (large) event, having a large number of events is orthogonal - if the config says max 1 event in Tip, the add has to do a calve every time you add another)

For stores like MessageDb and EventStoreDb, there is no complex logic of this nature that would warrant it (i.e. the underlying store manages atomic adding of batches as appropriate)

That said, concrete tests that vary the timing and batter it with load a la the Jepsen Test Suite do have a place in the overall picture, especially where there's store-level transactional logic (which there is for Cosmos and Dynamo). (In the Dynamo case, again the FSharp.AWS.DynamoDB library covers a lot of the testing of basic wiring together of transactional requests).

It does not appear to for the repro above 🤔

Hm. As mentioned I'll hopefully get to looking at it relatively soon.

Off the top of my head it would seem that some debug logging exposing the d/D values near where the exception happens may be needed (from memory there is some conditional Debug/Verbose logging that can be triggered by lowering the Serilog min log level).