GC deleted live tservers WAL node in zookeeper.

[keith-turner]

Describe the bug

Saw this while testing #6217 3908c7e, but suspect its a more general issue in main as wal/tserver/GC code was not modified.

At startup tservers create a node in ZK to track their active WALs. Later this node must exists when the tserver creates a WAL.

In the tserver logs saw it obtained it lock

2026-04-03T19:44:16,303 Thread[57] [tserver.TabletServer] DEBUG: Obtained tablet server lock /tservers/accumulo/localhost:9800/zlock#3e1f9d39-5a26-4794-95c1-5238103b9ac8#0000000000 localhost:9800[10000ff8e2b001a]

Then a bit later the GC process removed the tservers WAL node in ZK. The GC only does this if the tserver is not in the live tserver set and it has no wals registered in ZK.

2026-04-03T19:44:50,545 Thread[52] [gc.GarbageCollectWriteAheadLogs] INFO : Removing znode for localhost:9800[10000ff8e2b001a]

Much later the tsever failed to create a new WAL because its node was deleted in ZK. This caused all writes on the tserer to hang.

2026-04-03T21:28:19,818 Thread[142] [log.TabletServerLogger] ERROR: Failed to add new WAL marker for hdfs://10.113.13.85:8020/accumulo/wal/localhost+9800/0286a9ab-1735-428f-b091-263b2e42396b
org.apache.accumulo.server.log.WalStateManager$WalMarkerException: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /wals/localhost:9800[10000ff8e2b001a]/0286a9ab-1735-428f-b091-263b2e42396b
	at org.apache.accumulo.server.log.WalStateManager.updateState(WalStateManager.java:138)
	at org.apache.accumulo.server.log.WalStateManager.addNewWalMarker(WalStateManager.java:124)
	at org.apache.accumulo.tserver.TabletServer.addNewLogMarker(TabletServer.java:1032)
	at org.apache.accumulo.tserver.log.TabletServerLogger.lambda$startLogMaker$0(TabletServerLogger.java:294)
	at org.apache.accumulo.core.util.threads.Threads.lambda$createCriticalThread$0(Threads.java:76)
	at org.apache.accumulo.core.trace.TraceWrappedRunnable.run(TraceWrappedRunnable.java:52)
	at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /wals/localhost:9800[10000ff8e2b001a]/0286a9ab-1735-428f-b091-263b2e42396b
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:117)
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:53)
	at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:1347)
	at org.apache.accumulo.core.zookeeper.ZooSession.create(ZooSession.java:272)
	at org.apache.accumulo.core.fate.zookeeper.ZooReaderWriter.lambda$putPersistentData$1(ZooReaderWriter.java:92)
	at org.apache.accumulo.core.fate.zookeeper.ZooReader.retryLoopMutator(ZooReader.java:174)
	at org.apache.accumulo.core.fate.zookeeper.ZooReader.retryLoop(ZooReader.java:153)
	at org.apache.accumulo.core.fate.zookeeper.ZooReaderWriter.putPersistentData(ZooReaderWriter.java:90)
	at org.apache.accumulo.core.fate.zookeeper.ZooReaderWriter.putPersistentData(ZooReaderWriter.java:65)
	at org.apache.accumulo.server.log.WalStateManager.updateState(WalStateManager.java:136)

[keith-turner]

Saw the following odd message a lot in the manager logs. Only saw this single tserver flapping back and forth, others were fine.

2026-04-03T19:54:03,437 Thread[50] [manager.Manager] INFO : New servers: [localhost:9800[10000ff8e2b001a]]
2026-04-03T19:54:03,457 Thread[50] [manager.Manager] WARN : Lost servers [localhost:9800[10000ff8e2b001a]]
2026-04-03T19:54:08,461 Thread[50] [manager.Manager] INFO : New servers: [localhost:9800[10000ff8e2b001a]]
2026-04-03T19:54:08,485 Thread[50] [manager.Manager] WARN : Lost servers [localhost:9800[10000ff8e2b001a]]
2026-04-03T19:54:13,489 Thread[48] [manager.Manager] INFO : New servers: [localhost:9800[10000ff8e2b001a]]
2026-04-03T19:54:13,508 Thread[48] [manager.Manager] WARN : Lost servers [localhost:9800[10000ff8e2b001a]]

[keith-turner]

Found the cause of this problem. There are two paths like the following in ZK where accumulo and default are resource groups. Only one of the paths below actually has a lock under it. The LiveTServerSet code tracks paths in a map using only host port. When it sees the path w/o that lock for that host port it thinks its dead. When it sees the path with the lock, it thinks its alive. That is why the tserver keeps oscillating between alive and dead. Eventually the path w/ no lock is deleted and LiveTServerSet settles down, but by that time GC has left the tserver in bad state.

/tservers/accumulo/localhost:9800
/tservers/default/localhost:9800

There are test that cover multiple tservers in different resource groups like TabletResourceGroupBalanceIT. However these test use default mini config that has processes choose a random port. This problem was seen using default config which puts tservers in a range of ports starting at 9800.

Also seeing some potentially problematic behavior w/ the dead tserver set in ZK as it does not contain the RG. This needs more investigation.

2026-04-07T20:13:51,255 Thread[43] [manager.LiveTServerSet] TRACE: Received event: WatchedEvent state:SyncConnected type:NodeDeleted path:/dead/tservers/UNKNOWN/localhost:9800 zxid: 391
2026-04-07T20:13:51,260 Thread[43] [manager.LiveTServerSet] TRACE: Received event: WatchedEvent state:SyncConnected type:NodeCreated path:/dead/tservers/UNKNOWN/localhost:9800 zxid: 392
2026-04-07T20:13:53,287 Thread[43] [manager.LiveTServerSet] TRACE: Received event: WatchedEvent state:SyncConnected type:NodeDeleted path:/dead/tservers/UNKNOWN/localhost:9800 zxid: 404
2026-04-07T20:13:53,291 Thread[43] [manager.LiveTServerSet] TRACE: Received event: WatchedEvent state:SyncConnected type:NodeCreated path:/dead/tservers/UNKNOWN/localhost:9800 zxid: 405