KV Mirror with filter subject: Creation works but "No keys found in bucket" when listing keys

[mikoskinen]

Observed behavior

Here's the complete C# code I'm using to create two KV buckets, where the second one is the mirror of the first one with a subject filter defined:

using var conn = new ConnectionFactory().CreateConnection();

var kvm = conn.CreateKeyValueManagementContext();

var kvName1 = "stest";
var kvName2 = "stest-invoices";

var k1 = new KeyValueConfiguration.KeyValueConfigurationBuilder()
    .WithName(kvName1)
    .Build();

kvm.Create(k1);

var mirr = new Mirror.MirrorBuilder()
    .WithName(kvName1)
    .WithFilterSubject("$KV.stest.invoice.*")
    .Build();

var k2 = new KeyValueConfiguration.KeyValueConfigurationBuilder()
    .WithName(kvName2)
    .WithMirror(mirr)
    .Build();

kvm.Create(k2);

Now when using the stest bucket through CLI things seem to work OK:

nats kv put stest invoice.500 'value'
nats kv put stest inv.500 'value'

nats kv ls

But the unexpected behavior is when trying to list the keys in the stest-invoices bucket:

No keys found in bucket

Expected behavior

Listing the keys in the mirror bucket should work OK.

Server and client version

nats-server: v2.10.11
nats cli: 0.1.3
"NATS.Client" Version="1.1.4"

Host environment

Windows 11

Steps to reproduce

1. Create the KVs using the code above
2. nats kv put stest invoice.500 'value'
3. nats kv put stest inv.500 'value'
4. nats kv ls
5. nats kv ls stest-invoices

[mikoskinen]

I'm starting to wonder if this is more of an issue with the NATS server (or CLI?) and the .NET client is correctly creating the mirror KV.

Command:
nats s view KV_stest-invoices

Seems to list things correctly.

Though getting the key from the mirror KV doesn't work:

[scottf]

This is an issue with mirroring or filter subjects. I'm not sure how to resolve this yet.

[mikoskinen]

Just a quick addition. Instead of mirroring I tried to use stest as the source for stest-invoices. It is showing identical behaviour:

Here's the code:

using var conn = new ConnectionFactory().CreateConnection();

var kvm = conn.CreateKeyValueManagementContext();

var kvName1 = "stest";
var kvName2 = "stest-invoices";

var k1 = new KeyValueConfiguration.KeyValueConfigurationBuilder()
    .WithName(kvName1)
    .Build();

kvm.Create(k1);

var src = new Source.SourceBuilder()
    .WithName(kvName1)
    .WithFilterSubject("$KV.stest.invoice.*")
    .Build();

var k2 = new KeyValueConfiguration.KeyValueConfigurationBuilder()
    .WithName(kvName2)
    .WithSources(src)
    .Build();

kvm.Create(k2);

Here's the stream report:

And here's the info for the stest-invoices:

C:\>nats s info KV_stest-invoices
Information for Stream KV_stest-invoices created 2024-03-12 12:56:20

              Replicas: 1
               Storage: File

Options:

             Retention: Limits
       Acknowledgments: true
        Discard Policy: New
      Duplicate Window: 2m0s
     Allows Msg Delete: false
          Allows Purge: true
        Allows Rollups: true

Limits:

      Maximum Messages: unlimited
   Maximum Per Subject: 1
         Maximum Bytes: unlimited
           Maximum Age: unlimited
  Maximum Message Size: unlimited
     Maximum Consumers: unlimited

Replication:

               Sources: KV_stest

Source Information:

           Stream Name: KV_stest
        Subject Filter: $KV.stest.invoice.*
                   Lag: 0
             Last Seen: 147ms

State:

              Messages: 1
                 Bytes: 128 B
        First Sequence: 1 @ 2024-03-12 12:56:30 UTC
         Last Sequence: 1 @ 2024-03-12 12:56:30 UTC
      Active Consumers: 0
    Number of Subjects: 1

[mikoskinen]

Sorry for spamming... As the last idea I tried using "subject transform".

Unfortunately the behavior is the same: bucket reports no keys even though they are in the stream.

[scottf]

It's a subject filter issue. I used JetStream directly to read the messages.

In the stream for stest

Stream:KV_stest | Message Subject: $KV.stest.invoice.500 | Data: invoice
Stream:KV_stest | Message Subject: $KV.stest.inv.500 | Data: inv

In the stream for stest-invoices

Stream:KV_stest-invoices | Message Subject: $KV.stest.invoice.500 | Data: invoice

The subject for the message in stest-invoices does not match the subject for a key in that bucket, so any KV operation, either through the cli or the .NET client will never find the key. It needs to be transformed. I'm working on figuring how to do it.

[scottf]

So problem figured out. There are two things here.

1. You have to use subject transform instead of subject filter and make sure the destination subject is correct.

var kvc1 = new KeyValueConfiguration.KeyValueConfigurationBuilder()
    .WithName("stest")
    .Build();

kvm.Create(kvc1);

var transform = SubjectTransform.Builder()
    .WithSource("$KV.stest.invoice.*")
    .WithDestination("$KV.stest-invoices.invoice.{{wildcard(1)}}")
    .Build();

var mirr = Mirror.Builder()
    .WithName("stest")
    .WithSubjectTransforms(transform)
    .Build();

var kvc2 = new KeyValueConfiguration.KeyValueConfigurationBuilder()
    .WithName("stest-invoices")
    .WithMirror(mirr)
    .Build();

2. I needed to fix something in the client, it could not properly lookup the key subject on a transform. it was a 1 line so that was good.

I added a unit test in this PR: https://github.com/nats-io/nats.net/pull/878/files
I copied the code I built to figure this out here: https://gist.github.com/scottf/7bb8e5d7608d43469783f55f8a0c3230