KV get returns expired value entry when TTL is enabled

[burl21]

Observed behavior

When using KeyValue with per-message TTL enabled, the following code snippet returns a non-null value even though the server has already deleted the key due to expiration:

var kve = kv.get(key);
// kve.getValue() is null
return kve != null && kve.getValue() != null
          ? converter.read(kve.getValue(), Object.class)
          : null;

Expected behavior

keyValue.get(key) should return null when the key has expired and been deleted by the server.

Server and client version

Server:

Version:  2.11.2
Git:      [55efd1d]
Name:     NBTPDMICKE5D3UTWIT4X3YFODFCSLTWJDX6HSQO5OUBAWPQ5RXXQ2EXM
Node:     LmBR30Xm
ID:       NBTPDMICKE5D3UTWIT4X3YFODFCSLTWJDX6HSQO5OUBAWPQ5RXXQ2EXM

Client: 2.21.2

Host environment

No response

Steps to reproduce

nats kv info long-cache:

Bucket Name           : long-cache
History Kept          : 1
Values Stored         : 1
Compressed            : false
Per-Key TTL Supported : true
Limit Marker TTL      : 1d0h0m0s
Backing Store Kind    : JetStream
Description           : Used for cache
Bucket Size           : 327 B
Maximum Bucket Size   : unlimited
Maximum Value Size    : unlimited
Maximum Age           : 1d0h0m0s
JetStream Stream      : KV_long-cache
Storage               : Memory

nats kv ls long-cache:

No keys found in bucket

Debug:

kv.create(key, value, MessageTtl.seconds(20));

Sometimes we see:
Error while putting cache entry with key 'val.57c3e7f5bbee56f876e4cece2405b608'. wrong last sequence: 2 [10071]

[scottf]

This is not a client issue, I say this because the client just asks the server for the data and gets it, nothing the client can do. I supposes it's a timing issue with the TTL and the server's handling of it.

Do you have a piece of java code and maybe a server configuration that can reproduce this?

[scottf]

@burl21 Come to think of it, what ttl are you talking about? The limitMarkerTtl or some stream TTL. limitMarkerTtl is designed to clean up delete markers that are left when a key is deleted. Message TTL is not available in the key value yet, did you manually modify the underlying KV stream?

[burl21]

Thanks @scottf for the quick response.
By TTL, I mean setting a per-message expiration using:
kv.create("test-key-1", "test-value-1".getBytes(), seconds(10));

What I’ve noticed is that the message isn’t actually deleted by the server until after limitMarkerTtl. Is that expected?
For example, nats kv info long-cache still shows Values Stored: 1, but nats kv ls long-cache returns No keys found in bucket.

docker run -it -p 4222:4222 nats -js -D

  @Test
  void should_delete_kv_after_ttl_expired() {
    try (var connection = Nats.connect()) {
      var bucketName = "long-cache";
      connection
          .keyValueManagement()
          .create(
              KeyValueConfiguration.builder(bucketName)
                  .description("long cache bucket")
                  .storageType(Memory)
                  .compression(false)
                  // if i don't set limitMarkerTtl it will throw error:
                  // JetStreamApiException: per-message TTL is disabled [10166]
                  .limitMarker(ofSeconds(5))
                  .ttl(ofDays(1))
                  .build());
      var kvOptions = KeyValueOptions.builder().jsRequestTimeout(ofSeconds(1)).build();
      var kv = connection.keyValue(bucketName, kvOptions);

      kv.create("test-key-1", "test-value-1".getBytes(), seconds(1));
      TimeUnit.SECONDS.sleep(2);

      var kve = kv.get("test-key-1");
      assertNotNull(kve);
      assertNull(kve.getValue());

      // wait for limitMarkerTtl
      TimeUnit.SECONDS.sleep(4);
      assertNull(kv.get("test-key-1"));

    } catch (InterruptedException | IOException | JetStreamApiException e) {
      throw new RuntimeException(e);
    }
  }

The NATS CLI, on the other hand, returns an error:

❯ nats kv get long-cache test-key-1
long-cache > test-key-1 revision: 1 created @ 04 Jun 25 18:56 UTC
test-value-1

❯ nats kv get long-cache test-key-1
nats: error: nats: key not found

❯ nats kv info long-cache
Information for Key-Value Store Bucket long-cache created 2025-06-04T20:56:10+02:00

Configuration:
  Bucket Name:          long-cache
  History Kept:         1
  Values Stored:        1
  Compressed:           false
  Per-Key TTL Supported:true
  Limit Marker TTL:     5.00s
  Backing Store Kind:   JetStream
  Description:          long cache bucket
  Bucket Size:          115 B
  Maximum Bucket Size:  unlimited
  Maximum Value Size:   unlimited
  Maximum Age:          1d0h0m0s
  JetStream Stream:     KV_long-cache
  Storage:              Memory

Cluster Information:
  Name:
  Leader:               NBDRWUBJLUGWASHGCLJQOAD6L2ZUCIGNVCJSRVYDCT7QBZIHA5RFJUWO

[scottf]

Fixed. Thanks for the detail, it took me a minute. #1323

[burl21]

@scottf thanks a lot for the quick fix. I need to understand one thing: we’ve noticed that with the current issue there’s an interesting behavior regarding the various watchers. Basically, when the NATS server deletes the key — either due to MaxAge or Per-Key TTL — the watcher gets notified with a PUT operation and a null value. This way, the application understands that the key was deleted (value == null => key deleted).

keyValue.watch(
    "test-key-1",
    new KeyValueWatcher() {
      @Override
      public void watch(KeyValueEntry keyValueEntry) {
        switch (keyValueEntry.getOperation()) {
          case PUT -> {
            var value = keyValueEntry.getValue();
            if (value == null) {
              // When a key has expired, instead of receiving a DELETE operation notification,
              // we receive a PUT operation with null value.
              log.info("Watch operation PUT {} with null value, key expired", keyValueEntry);
            } else {
              log.info("Watch operation PUT {}", keyValueEntry);
            }
          }
          case DELETE, PURGE -> log.info("Watch operation DEL {}", keyValueEntry);
        }
      }

      @Override
      public void endOfData() {}
    });

With this fix, I assume the watcher will no longer be notified, correct?

[MauriceVanVeen]

With this fix the watcher should be notified, but with a PURGE operation instead of PUT.

[scottf]

I added a unit test that has a watcher, it does in fact get the purge