Enable clients to scan offline tables using ScanServers by dlmarion · Pull Request #6156 · apache/accumulo

dlmarion · 2026-02-25T22:35:59Z

During a normal client scan the TabletLocator resolves tablets (key extent and location) for a given search range. The location is necessary for the client to be able to create a connection with a tablet server to perform the scan, but the location is not needed when the client is using scan servers. The TabletLocator does not resolve tablets for offline tables.

This change introduces the OfflineTabletLocatorImpl that performs this resolution (range -> key extents) and does not provide any location information. This change also modifies the client to allow scans on offline tables when using scan servers and uses the new OfflineTabletLocatorImpl in that code path.

During a normal client scan the TabletLocator resolves tablets (key extent and location) for a given search range. The location is necessary for the client to be able to create a connection with a tablet server to perform the scan, but the location is not needed when the client is using scan servers. The TabletLocator does not resolve tablets for offline tables. This change introduces the OfflineTabletLocatorImpl that performs this resolution (range -> key extents) and does not provide any location information. This change also modifies the client to allow scans on offline tables when using scan servers and uses the new OfflineTabletLocatorImpl in that code path.

dlmarion · 2026-02-25T22:37:16Z

This is marked as draft as the more complex text in the new IT class is having an issue with running out of memory. These changes are functional in the smaller scale test, so I likely have a scaling issue and maybe some bugs to work out.

core/src/main/java/org/apache/accumulo/core/clientImpl/OfflineTabletLocatorImpl.java

keith-turner · 2026-02-26T15:57:14Z

core/src/main/java/org/apache/accumulo/core/clientImpl/ScannerImpl.java

+    if (getConsistencyLevel() == ConsistencyLevel.IMMEDIATE) {
+      try {
+        String tableName = context.getTableName(tableId);
+        context.requireNotOffline(tableId, tableName);


If this can do ZK operation per creation of a scanner iterator it could cause problems. Not sure what the impl of this method does.

It's using ZooCache, not hitting ZK directly.

dlmarion · 2026-02-26T19:18:22Z

This is marked as draft as the more complex text in the new IT class is having an issue with running out of memory. These changes are functional in the smaller scale test, so I likely have a scaling issue and maybe some bugs to work out.

Took this out of draft as I have the IT working ( I fixed the known issues) and I reworked the cache to hopefully provide better memory management at larger scales.

core/src/main/java/org/apache/accumulo/core/clientImpl/OfflineTabletLocatorImpl.java

keith-turner · 2026-02-26T19:49:14Z

core/src/main/java/org/apache/accumulo/core/clientImpl/OfflineTabletLocatorImpl.java

+      }
+    }
+
+    private KeyExtent findOrLoadExtent(KeyExtent start) {


Could simplify the locking by making extents use a ConcurrentSkip list. Then would not need the read lock. Could have a single lock only for the case of doing updates, the code that currently gets a write lock. Also the eviction handler could directly remove from the extents map if it were a concurrently skip list w/o any locking.

I attempted this locally and it doesn't work the way that we want it to. When the cache becomes full Caffeine may start evicting newly inserted entries which ends up causing the IT to fail.

core/src/main/java/org/apache/accumulo/core/clientImpl/OfflineTabletLocatorImpl.java

keith-turner · 2026-02-26T19:54:53Z

core/src/main/java/org/apache/accumulo/core/clientImpl/OfflineTabletLocatorImpl.java

+          Integer.parseInt(ClientProperty.OFFLINE_LOCATOR_CACHE_SIZE.getValue(clientProperties));
+      prefetch = Integer
+          .parseInt(ClientProperty.OFFLINE_LOCATOR_CACHE_PREFETCH.getValue(clientProperties));
+      cache = Caffeine.newBuilder().expireAfterAccess(cacheDuration).initialCapacity(maxCacheSize)


The cache could have a weigher. The could be useful for the case where tablets splits can widely vary in size.

core/src/main/java/org/apache/accumulo/core/clientImpl/OfflineTabletLocatorImpl.java

dlmarion · 2026-03-02T17:37:08Z

In 40cafc0 I made some changes to handle the max cache size manually instead of letting Caffeine handle this case. The Caffeine cache is being used to handle the expiration of KeyExtents from the extents set based on last access time. The issue with Caffeine when nearing the max size is that it will select entries in the Cache for eviction that are new. We don't want this behavior as we are pre-fetching KeyExtents from the metadata table for performance reasons. The new logic asks Caffeine for a set of entries that are the coldest, then removes the ones that before the KeyExtent that we are searching for.

dlmarion · 2026-03-02T19:13:46Z

core/src/main/java/org/apache/accumulo/core/clientImpl/OfflineTabletLocatorImpl.java

+      // cache so that they are not immediately evicted.
+      if (cacheCount.get() + prefetch + 1 >= maxCacheSize) {
+        int evictionSize = prefetch * 2;
+        Set<KeyExtent> candidates = new HashSet<>(evictionPolicy.coldest(evictionSize).keySet());


evictionPolicy.coldest() works when setting the Caffeine maximumSize option. Another approach would be to not set the maximumSize on the Cache at all, and use cache.policy().expireAfterAccess().orElseThrow().youngest(int) instead. I'm not sure, but this may be functionally equivalent but might allow us to remove some code trying to deal with Caffeine removing objects prematurely based on size.

On second thought, with maximumSize option set, then Caffeine will evict based on a modified LRU algorithm, which is likely what we want. We don't want to blindly evict the youngest entries. In this contrived test the youngest entries are the least recently used, but that's not going to be the case in a long-lived process.

keith-turner · 2026-03-02T22:32:08Z

core/src/main/java/org/apache/accumulo/core/clientImpl/OfflineTabletLocatorImpl.java

+      } finally {
+        lock.readLock().unlock();
+      }
+      lock.writeLock().lock();


The try{}finally{ unlock()} should immediately follow the lock to ensure its unlocked even if there are exceptions.

I'm not sure I'm following your comment. The code is doing:

lock.readLock().lock();
try {
..
} finally {
lock.readLock().unlock();
}
lock.writeLock().lock();
...

There are many lines of code, that could throw an exception, between acquiring the write lock and the try/finally that releases the lock.

Ah, I see what you are referring to now.

core/src/main/java/org/apache/accumulo/core/clientImpl/OfflineTabletLocatorImpl.java

core/src/main/java/org/apache/accumulo/core/clientImpl/TabletLocator.java

core/src/main/java/org/apache/accumulo/core/conf/ClientProperty.java

dlmarion · 2026-03-03T17:31:12Z

Build with sunny profile passed locally for me

dlmarion self-assigned this Feb 25, 2026

dlmarion added this to the 2.1.5 milestone Feb 25, 2026

keith-turner reviewed Feb 26, 2026

View reviewed changes

Made Offline tablet cache configurable, fix IT

dd9f6c8

dlmarion marked this pull request as ready for review February 26, 2026 19:17

keith-turner reviewed Feb 26, 2026

View reviewed changes

dlmarion added 5 commits February 27, 2026 18:02

Implemented some PR suggestions

3ed4585

Fix build

8f3ad2d

Enable cache metrics, fix location check

305297e

Add logging for time it takes to load tablets

63b9d17

Fix formatting

5603230

keith-turner reviewed Feb 27, 2026

View reviewed changes

core/src/main/java/org/apache/accumulo/core/clientImpl/OfflineTabletLocatorImpl.java Outdated Show resolved Hide resolved

Changes for correctness, dealing with cache max size behavior

40cafc0

Merge branch '2.1' into sserver-offline-tables

9305cdf

dlmarion commented Mar 2, 2026

View reviewed changes

dlmarion added 2 commits March 2, 2026 19:29

Change ticker option to scheduler, minor doc edits

b0f64bf

Modified cache size property to be optional

06b6331

keith-turner reviewed Mar 2, 2026

View reviewed changes

core/src/main/java/org/apache/accumulo/core/clientImpl/OfflineTabletLocatorImpl.java Outdated Show resolved Hide resolved

core/src/main/java/org/apache/accumulo/core/clientImpl/OfflineTabletLocatorImpl.java Outdated Show resolved Hide resolved

dlmarion added 2 commits March 3, 2026 15:23

Implemented PR suggestions

bf0ba6e

Fix try/finally for write lock

93820ba

keith-turner approved these changes Mar 3, 2026

View reviewed changes

core/src/main/java/org/apache/accumulo/core/clientImpl/TabletLocator.java Show resolved Hide resolved

core/src/main/java/org/apache/accumulo/core/conf/ClientProperty.java Outdated Show resolved Hide resolved

Implemented PR suggestions, updated property descriptions

206b1a8

dlmarion mentioned this pull request Mar 3, 2026

Clear or remove TabletLocator when table deleted / offlined #6164

Open

dlmarion merged commit 34823ac into apache:2.1 Mar 4, 2026
7 of 8 checks passed

dlmarion deleted the sserver-offline-tables branch March 4, 2026 12:18

dlmarion mentioned this pull request Mar 4, 2026

Remove unused extents from tablet locator #6169

Open

Conversation

dlmarion commented Feb 25, 2026

Uh oh!

dlmarion commented Feb 25, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dlmarion commented Feb 26, 2026

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

dlmarion commented Mar 2, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dlmarion commented Mar 3, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants