fix(table): make evict() return a Promise on RocksDB - #890
Conversation
Patch cherry-pick: conflictCherry-pick onto The conflict markers are committed on branch |
There was a problem hiding this comment.
Code Review
This pull request ensures that the transaction commit in the evict function always returns a Promise by wrapping the RocksDB commit in Promise.resolve, allowing the cleanup loop to safely call .catch on the resolution. However, the reviewer identified a critical issue where returning a value from a finally block can swallow errors and override early returns. A refactoring of the evict function was suggested to return the commit promises from the try block and use the finally block solely for transaction cleanup.
| return (lmdbTransaction as any).commit(); | ||
| } | ||
| // RocksDB: eviction writes went directly into the raw transaction via options; | ||
| // commit it directly, as DatabaseTransaction.commit() would abort it (no tracked writes) | ||
| return (transaction as any)?.commit?.(); | ||
| // commit it directly, as DatabaseTransaction.commit() would abort it (no tracked writes). | ||
| // Wrap in Promise.resolve so callers can rely on a thenable return regardless of engine. | ||
| return Promise.resolve((transaction as any)?.commit?.()); |
There was a problem hiding this comment.
Returning a value from a finally block is an unsafe pattern in JavaScript/TypeScript (often flagged by ESLint's no-unsafe-finally rule). It has two major unintended side effects:
- Swallows Errors: Any exception thrown in the
tryblock (e.g., duringremoveEntryorupdateIndices) will be silently discarded, and the value returned by thefinallyblock will be returned instead. This makes debugging eviction failures extremely difficult. - Overrides Early Returns: Any early
return;statement in thetryblock (which is intended to returnundefinedwhen eviction is skipped) is overridden by thefinallyblock's return value. As a result,evict()always returns a Promise even when no eviction was performed, which causes the cleanup loop to unnecessarily track and await these skipped operations.
To fix this, we should return the commit promises directly from the try block, and use the finally block only to abort/cleanup the transaction if it wasn't committed.
Here is the suggested refactoring for the evict function:
static evict(id, existingRecord, existingVersion) {
let entry;
const lmdbTransaction = txnForContext({ transaction: new DatabaseTransaction() });
let transaction = lmdbTransaction.getReadTxn();
let options = { transaction };
let committed = false;
try {
if (hasSourceGet || audit) {
if (!existingRecord) return;
entry = primaryStore.getEntry(id, options);
if (!entry || !existingRecord) return;
if (entry.version !== existingVersion) return;
}
if (hasSourceGet) {
// if there is a resolution in-progress, abandon the eviction
if (primaryStore.hasLock(id, entry.version)) return;
}
// evictions never go in the audit log, so we can not record a deletion entry for the eviction
// as there is no corresponding audit entry and it would never get cleaned up. So we must simply
// removed the entry entirely, but first cleanup indices
if (primaryStore.ifVersion) {
// lmdb
primaryStore.ifVersion?.(id, existingVersion, () => {
updateIndices(id, existingRecord, null);
});
removeEntry(primaryStore, entry ?? primaryStore.getEntry(id), existingVersion);
} else {
updateIndices(id, existingRecord, null, options);
removeEntry(primaryStore, entry ?? primaryStore.getEntry(id), options);
}
committed = true;
if (primaryStore.ifVersion) {
return (lmdbTransaction as any).commit();
}
return Promise.resolve((transaction as any)?.commit?.());
} finally {
if (!committed) {
if (primaryStore.ifVersion) {
lmdbTransaction.abort();
} else {
(transaction as any)?.abort?.();
}
}
}
}This ensures that errors are properly propagated, and early returns correctly return undefined.
There was a problem hiding this comment.
both failure modes are real as far as I can tell, and predate this PR. Probably best for another PR to be opened for it and let this one keep the focus on this specific fix
kriszyp
left a comment
There was a problem hiding this comment.
Table.ts line 4549 throws an error trying to call a function? I don't see any call on line 4549: https://github.com/HarperFast/harper/blob/main/resources/Table.ts#L4549
So you are saying that there is an error thrown by evict? It seems like we should move the commit() call out of the finally (should only execute if removeEntry doesn't throw), and possibly (synchronously) try/catch the evict call. The code generally looks like it is designed to handle a MaybePromise , which should be the expected return type of the evict (except for the missing try/catch if are expecting only rejections, no throws, I guess). Is there a GH issue that documents the error this is fixing?
|
Reviewed; no blockers found. |
|
@kriszyp You're right on the line number, should have been 4551 I pushed changes, commit moves into the try block with a committed flag, finally only aborts when !committed No GH issue, came direct fro diagnosis on the stage cluster, happy to file one if you want it for tracking. Stack: (line 4473 from 5.0.22, claude ran with that, but on main is 4551) cause: Repro: any 5.0.22 rocksDB node with a TTL evicted table. Cleanup scan |
…mise<void> | undefined
kriszyp
left a comment
There was a problem hiding this comment.
Good, I think appropriate fixes are now in place.
Co-authored-by: Kris Zyp <kriszyp@gmail.com>
Summary
Promise.resolve(...)inevict()'s finally so it always returns a thenable, matching the LMDB branch.(resolution as any).catchband-aid from the cleanup scanner now that the source contract is honored.Context
39c3a24 ("storage-engine-aware commit in evict()") changed the RocksDB path to
return transaction?.commit(), which is synchronous (the LMDB wrapper returned a Promise). The cleanup scanner at Table.ts:4549 chains.catchon the return, which now throwsTypeError: resolution.catch is not a functionevery cycle on RocksDB. The scanner crashes, eviction stops draining records, and peer counts hold at the full count instead of shrinking whenexpiresAtis set.Affects v5.0.22 and v5.0.23.
mainmasks the crash withif (resolution && (resolution as any).catch), which silences the TypeError but silently skips the concurrency control and never awaits the eviction's commit, so eviction is still broken onmain, just quieter.This restores the Promise contract at the source so both engines behave the same. The cleanup loop goes back to the simpler
if (resolution) { ... resolution.catch(...) }.closes #1089