(wip | no review) feat(tables): post-commit stats push to optimizer#608
(wip | no review) feat(tables): post-commit stats push to optimizer#608mkuchenbecker wants to merge 5 commits into
Conversation
| public static final String HTS_SEARCH_TABLES_TIME = "hts_search_tables_time"; | ||
|
|
||
| // Optimizer post-commit stats push (Tables → Optimizer) | ||
| public static final String OPTIMIZER_STATS_DURATION = "optimizer_stats_duration"; |
There was a problem hiding this comment.
max duration timeout
There was a problem hiding this comment.
Renamed to POSTCOMMIT_OP_DURATION (tagged op + outcome). The timer is bounded by the dispatcher's outer tables.postcommit.per-op-timeout-ms (default 3000 ms) — comment on the constant calls that out explicitly. Latest commit.
| public static final String OPTIMIZER_STATS_DURATION = "optimizer_stats_duration"; | ||
| public static final String OPTIMIZER_STATS_ATTEMPTS = "optimizer_stats_attempts"; | ||
| public static final String OPTIMIZER_STATS_SKIPPED = "optimizer_stats_skipped"; | ||
| public static final String OPTIMIZER_STATS_FAILED_FINAL = "optimizer_stats_failed_final"; |
There was a problem hiding this comment.
Failed vs failed_final.
There was a problem hiding this comment.
Dropped _FINAL. The dispatcher now emits a single POSTCOMMIT_OP_FAILED counter tagged with the classified outcome. There is no per-attempt failure counter at the dispatcher; per-attempt retry behavior lives inside each op and is observable through that op's own logging if it needs it.
|
|
||
| /** | ||
| * Pushes a stats record to the optimizer's {@code PUT /v1/optimizer/stats/{tableUuid}} endpoint | ||
| * after a successful Iceberg snapshot commit. Fire-and-forget — the {@link #report} call returns |
There was a problem hiding this comment.
we should see if we get an error. this working is ambiguous and non-useful.
There was a problem hiding this comment.
Rewrote the javadoc on OptimizerStatsPostCommitOperation to state the contract: prepare() returns a Mono that completes on HTTP 2xx and signals an error otherwise; the dispatcher owns timeout, subscription, error swallowing, and metric emission. Latest commit.
| public class OptimizerStatsClient { | ||
|
|
||
| /** Per-call URL path. */ | ||
| static final String PATH_TEMPLATE = "/v1/optimizer/stats/{tableUuid}"; |
There was a problem hiding this comment.
is this the canonical way to do this?
There was a problem hiding this comment.
There wasn't one. Added TableStatsController.BASE_PATH and TableStatsController.TABLE_PATH_TEMPLATE as public constants on the optimizer controller (latest commit) and refactored its own @RequestMapping to use BASE_PATH. The tables-side keeps its own literal because we don't want a tables → optimizer compile dep (optimizer pulls MySQL/JPA); the comment on the tables-side constant now names TableStatsController.TABLE_PATH_TEMPLATE as the canonical source.
| static final String OPT_IN_PROPERTY = "maintenance.optimizer.stats.enabled"; | ||
|
|
||
| /** Iceberg snapshot-summary keys we read. All values are decimal-string longs. */ | ||
| static final String SUMMARY_TOTAL_DATA_FILES = "total-data-files"; |
There was a problem hiding this comment.
Done. TableDto.currentSnapshot now carries a CurrentSnapshotInfo(snapshotId, summary) populated from Snapshot.snapshotId() and Snapshot.summary() at convertToTableDto. OptimizerStatsRequest.Snapshot carries snapshotId. The server-side dedupe wiring (use snapshot ID as an idempotency token on upsert and reject out-of-order replays) is intentionally deferred to a follow-up change so this PR stays focused on the framework + first operation.
There was a problem hiding this comment.
Done. TableDto.currentSnapshot now carries a CurrentSnapshotInfo(snapshotId, summary) populated from Snapshot.snapshotId() and Snapshot.summary() at convertToTableDto. OptimizerStatsRequest.Snapshot carries snapshotId. The server-side dedupe wiring is intentionally deferred to a follow-up change so this PR stays focused on the framework + first operation.
| * </ul> | ||
| */ | ||
| public void report(TableDto saved, Map<String, String> snapshotSummary) { | ||
| reportAsync(saved, snapshotSummary).subscribe(); |
There was a problem hiding this comment.
why async? If the thread dies we lose the data, no? There will never be a timeout ever with this impl as we don't actually.
Help me justify this complexity given we subscribe to the result.
There was a problem hiding this comment.
Justification, on record: a synchronous push converts an optimizer outage into a tables-service write outage. The push is a best-effort scheduling signal, not a write-correctness step — its blast radius must not include the write path. Crash-loss between commit and HTTP-completion is acceptable because operations are cumulative: the next commit re-pushes the current state and the consumer self-corrects from missing data.
On complexity: the async machinery now lives in one place (PostCommitDispatcher — timeout, swallow, subscribe). Operations describe payload + endpoint. Future reviewers reason about async semantics once. The optimizer-stats op itself dropped to prepare() + buildRequest() + a couple of static helpers — no .subscribe(), no outer timeout, no onErrorResume.
"There will never be a timeout" — the per-op timeout still fires; it bounds Reactor pipeline runtime (resource freeing, connection pool occupancy, the duration timer) even though the commit thread is no longer watching for it. The dispatcher's onErrorResume records outcome=timeout deterministically (covered by decorate_workExceedsPerOpTimeout_incrementsFailedWithTimeoutOutcome).
| .increment(); | ||
| return Mono.empty(); | ||
| } | ||
| if (snapshotSummary == null || snapshotSummary.isEmpty()) { |
There was a problem hiding this comment.
do not use null guards if this will never be null. In the case its empty we should not sent null to this function, but detect and immedtialty conver to an optional.
If it may be null, use optional instead. Apply this across the pr.
There was a problem hiding this comment.
Applied. TableDto.currentSnapshot is now read through getCurrentSnapshot(): Optional<CurrentSnapshotInfo> and InternalRepositoryUtils converts the nullable table.currentSnapshot() at the producer. The operation's prepare() no longer has any null/empty-map guards on the summary — it does savedDto.getCurrentSnapshot().isPresent() and returns Optional.empty() if not, which the dispatcher records as postcommit_op_skipped{op=optimizer_stats}.
| * </ul> | ||
| */ | ||
| public void report(TableDto saved, Map<String, String> snapshotSummary) { | ||
| reportAsync(saved, snapshotSummary).subscribe(); |
There was a problem hiding this comment.
If this is fully async timeouts can be substantially relaxed. we can have a timeout on the order of 3s overall then.
There was a problem hiding this comment.
Relaxed. Dropped optimizer.stats.total-timeout-ms. The dispatcher now owns the outer ceiling via tables.postcommit.per-op-timeout-ms (default 3000 ms). The op keeps its per-attempt timeout (default 1000 ms) so retries can fit inside the 3 s budget.
|
|
||
| OptimizerStatsRequest body = buildRequest(saved, snapshotSummary); | ||
| String tableUuid = saved.getTableUUID(); | ||
| Timer.Sample sample = Timer.start(meterRegistry); |
There was a problem hiding this comment.
Does the closure capture this variable correctly?
There was a problem hiding this comment.
Yes — sample is effectively final and captured by both lambdas; doOnSuccess and onErrorResume are mutually exclusive on a Mono, so there's no double-stop. After the refactor the Timer.Sample lives in PostCommitDispatcher.decorate(...) instead, and the same reasoning applies there. Covered by decorate_successfulWork_recordsDurationWithSuccessOutcome and decorate_workSignalsError_incrementsFailedWithClassifiedOutcomeAndSwallowsError.
| /** | ||
| * Iceberg {@code Snapshot.summary()} for the current snapshot at the time the {@code TableDto} | ||
| * was constructed (post-commit on save; post-load on read). Populated only when an Iceberg {@code | ||
| * Table} is available — i.e. by {@code InternalRepositoryUtils.convertToTableDto}. Not persisted, |
There was a problem hiding this comment.
don't reference private impls. Reference the conditions this is or is not present.
There was a problem hiding this comment.
Rewrote. The javadoc now describes the semantic condition: "Present whenever the underlying table has at least one committed snapshot at construction time; absent for tables with no committed data (e.g. CREATE TABLE with no rows yet)." No mention of InternalRepositoryUtils or any other private impl.
| TableDto savedDto = openHouseInternalRepository.save(tableDtoToSave); | ||
| // Fire-and-forget push of the post-commit snapshot summary to the optimizer. Returns | ||
| // immediately; failures are swallowed inside the client. Skipped at the client when the | ||
| // table is not opted in via maintenance.optimizer.stats.enabled or when there is no current | ||
| // snapshot (e.g. CREATE TABLE without a data commit). | ||
| optimizerStatsClient.ifPresent( | ||
| client -> client.report(savedDto, savedDto.getCurrentSnapshotSummary())); |
There was a problem hiding this comment.
Gate this block with a config for the tables service so we can enable the check one openhouse instance at a time and not cause a problem.
There was a problem hiding this comment.
Done — single gate at the tables-service level: tables.postcommit.enabled (default false). When false the PostCommitDispatcher bean isn't constructed; the service consumes Optional<PostCommitDispatcher> and the on-commit hook is a literal no-op. Per-OH-instance rollout is one config flip. The op also keeps its own optimizer.stats.enabled switch so we can disable one operation while leaving the framework on.
| /** | ||
| * Present only when {@code optimizer.stats.enabled=true}. When absent, no post-commit push is | ||
| * attempted. | ||
| */ | ||
| @Autowired(required = false) | ||
| Optional<OptimizerStatsClient> optimizerStatsClient = Optional.empty(); | ||
|
|
There was a problem hiding this comment.
This should be modeled as a postcommit operation. Postcommit operations must be bounded and async and are considered best-effort. We ahve one postcommit operation registered to the service which is optimizer stats. That way we have a generic behaviour and a specific impl.
There was a problem hiding this comment.
Done. New services/tables/.../services/postcommit/ package: PostCommitOperation interface (name() + Optional<Mono<Void>> prepare(TableDto)), PostCommitProperties (enabled, perOpTimeoutMs), PostCommitDispatcher (per-op timeout, error swallow, metric emission, fire-and-forget subscribe). OptimizerStatsPostCommitOperation is the first registered impl. IcebergSnapshotsServiceImpl now has one line: postCommitDispatcher.ifPresent(d -> d.dispatch(savedDto)). Adding a future operation = implement the interface and let @ConditionalOnProperty wire it; no service-layer change.
Restructures the post-commit stats push around a small generic dispatcher so async/timeout/swallow plumbing lives in one place. Addresses the review comments on PR #608. Framework (new): - PostCommitOperation interface: String name() + Optional<Mono<Void>> prepare(TableDto). Implementations describe what to do; the dispatcher owns how it runs. - PostCommitDispatcher (@ConditionalOnProperty tables.postcommit.enabled): per-op wall-clock timeout, error swallow, metric emission, fire-and- forget subscribe. Exposes package-private decorate() for synchronous testing. - PostCommitProperties: enabled (default false), per-op-timeout-ms (default 3000). - Metric keys renamed: POSTCOMMIT_OP_DURATION / POSTCOMMIT_OP_SKIPPED / POSTCOMMIT_OP_FAILED, tagged op={name} and outcome={success, timeout, network_error, server_error, client_error, prepare_threw, unknown_error}. Why async: a synchronous post-commit push converts an optimizer outage into a tables write outage. Stats are a best-effort scheduling signal and must not block the write path. Crash-loss is bounded because state is cumulative: the next commit re-pushes the current state. Operation (refactor): - OptimizerStatsClient -> OptimizerStatsPostCommitOperation implementing PostCommitOperation. Drops outer timeout, onErrorResume, .subscribe() (dispatcher owns these). Keeps per-attempt timeout + bounded retry on retryable errors only. - Adds snapshotId to OptimizerStatsRequest.Snapshot; tracked for server-side idempotency wiring in BDP-102985. - Path constant comment now cites TableStatsController.TABLE_PATH_TEMPLATE as the canonical source. TableDto: - currentSnapshotSummary (Map) replaced by Optional<CurrentSnapshotInfo> (snapshotId + summary). Stored nullable internally; consumers read through getCurrentSnapshot() which returns Optional. Javadoc rewritten to describe the semantic condition (presence ~ table has a committed snapshot), not the private impl that populates it. - New CurrentSnapshotInfo value class. - InternalRepositoryUtils populates the new field with both snapshotId and summary; no extra I/O (still purely in-memory). Service: - IcebergSnapshotsServiceImpl: Optional<OptimizerStatsClient> replaced by Optional<PostCommitDispatcher>; on-commit hook is one line. Optimizer side: - TableStatsController publishes BASE_PATH and TABLE_PATH_TEMPLATE as public constants; @RequestMapping refactored to use BASE_PATH. Config: - application.properties: drop optimizer.stats.total-timeout-ms; add tables.postcommit.enabled + tables.postcommit.per-op-timeout-ms. Tests: - New PostCommitDispatcherTest (6 cases, no sleeps; uses decorate() to block synchronously). - OptimizerStatsClientTest renamed -> OptimizerStatsPostCommitOperationTest (6 cases adapted to the new prepare() contract; timeout case moved to PostCommitDispatcherTest). - IcebergSnapshotsServiceTest swaps @MockBean OptimizerStatsClient for @MockBean PostCommitDispatcher and verifies dispatch(savedDto) once. - TableDtoMappingTest updated for the renamed field. - All :services:tables and :services:optimizer tests pass; spotless clean. Filed: BDP-102985 (Optimizer stats: use snapshot ID as idempotency token on upsert) under epic BDP-102026. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
| * Map a terminal error to a small set of outcome tags. Kept here so all operations share the same | ||
| * taxonomy. | ||
| */ | ||
| private static String classifyOutcome(Throwable e) { |
There was a problem hiding this comment.
this entire thing is stupid, remove it and report the error type / code. everyone knows 500 are sys error and 400 are user.
cbb330
left a comment
There was a problem hiding this comment.
lets pull from the kafka events emitted from the tableauditevent or CDC stream or poll the database periodically for last-updated
otherwise we're editting core openhouse commit() logic to plug table optimizer. this doesn't look right
|
@cbb330 this is for testing. NRT stats is the official design. I put up draft PRs for review and tell claude what to fix. |
This PR is not in-review, is for testing, and there is no design for this.
After a successful Iceberg snapshot commit, push the table's current
snapshot summary to the optimizer's PUT /v1/optimizer/stats/{tableUuid}
endpoint. Opt-in per table via the maintenance.optimizer.stats.enabled
table property; globally gated by optimizer.stats.enabled in
application.properties (default false).
Implementation notes:
- Stats sourced from the in-memory Snapshot.summary() map captured at
convertToTableDto(); no extra HDFS round-trip on the commit path.
- WebClient + Reactor pipeline runs fire-and-forget on the Netty event
loop. The hook on the commit thread is .subscribe() and returns
immediately, so a slow/down optimizer cannot impact commit latency.
- Per-attempt timeout 1000 ms, total budget 2000 ms, up to 3 attempts.
Retries only fire on retryable errors (network, 408, 429, 5xx,
TimeoutException). All terminal errors are swallowed; Micrometer
counters and a warn log preserve observability.
- Bean wiring (WebClient, client component) is @ConditionalOnProperty
on optimizer.stats.enabled=true. The service consumes the client as
an Optional, so disabled deployments construct nothing.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Restructures the post-commit stats push around a small generic dispatcher so async/timeout/swallow plumbing lives in one place. Addresses the review comments on PR #608. Framework (new): - PostCommitOperation interface: String name() + Optional<Mono<Void>> prepare(TableDto). Implementations describe what to do; the dispatcher owns how it runs. - PostCommitDispatcher (@ConditionalOnProperty tables.postcommit.enabled): per-op wall-clock timeout, error swallow, metric emission, fire-and- forget subscribe. Exposes package-private decorate() for synchronous testing. - PostCommitProperties: enabled (default false), per-op-timeout-ms (default 3000). - Metric keys renamed: POSTCOMMIT_OP_DURATION / POSTCOMMIT_OP_SKIPPED / POSTCOMMIT_OP_FAILED, tagged op={name} and outcome={success, timeout, network_error, server_error, client_error, prepare_threw, unknown_error}. Why async: a synchronous post-commit push converts an optimizer outage into a tables write outage. Stats are a best-effort scheduling signal and must not block the write path. Crash-loss is bounded because state is cumulative: the next commit re-pushes the current state. Operation (refactor): - OptimizerStatsClient -> OptimizerStatsPostCommitOperation implementing PostCommitOperation. Drops outer timeout, onErrorResume, .subscribe() (dispatcher owns these). Keeps per-attempt timeout + bounded retry on retryable errors only. - Adds snapshotId to OptimizerStatsRequest.Snapshot; tracked for server-side idempotency wiring in BDP-102985. - Path constant comment now cites TableStatsController.TABLE_PATH_TEMPLATE as the canonical source. TableDto: - currentSnapshotSummary (Map) replaced by Optional<CurrentSnapshotInfo> (snapshotId + summary). Stored nullable internally; consumers read through getCurrentSnapshot() which returns Optional. Javadoc rewritten to describe the semantic condition (presence ~ table has a committed snapshot), not the private impl that populates it. - New CurrentSnapshotInfo value class. - InternalRepositoryUtils populates the new field with both snapshotId and summary; no extra I/O (still purely in-memory). Service: - IcebergSnapshotsServiceImpl: Optional<OptimizerStatsClient> replaced by Optional<PostCommitDispatcher>; on-commit hook is one line. Optimizer side: - TableStatsController publishes BASE_PATH and TABLE_PATH_TEMPLATE as public constants; @RequestMapping refactored to use BASE_PATH. Config: - application.properties: drop optimizer.stats.total-timeout-ms; add tables.postcommit.enabled + tables.postcommit.per-op-timeout-ms. Tests: - New PostCommitDispatcherTest (6 cases, no sleeps; uses decorate() to block synchronously). - OptimizerStatsClientTest renamed -> OptimizerStatsPostCommitOperationTest (6 cases adapted to the new prepare() contract; timeout case moved to PostCommitDispatcherTest). - IcebergSnapshotsServiceTest swaps @MockBean OptimizerStatsClient for @MockBean PostCommitDispatcher and verifies dispatch(savedDto) once. - TableDtoMappingTest updated for the renamed field. - All :services:tables and :services:optimizer tests pass; spotless clean. Filed: BDP-102985 (Optimizer stats: use snapshot ID as idempotency token on upsert) under epic BDP-102026. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Per review feedback that the HTML tags made the diff ugly. Drop paragraph breaks where the comment can be one sentence/paragraph, drop bullet lists in favor of inline prose, drop <em>/<b> emphasis entirely. No behavior change. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Spotless's googlejavaformat rewrites multi-paragraph block javadoc to re-insert <p> tags. // comments are left untouched. Per review feedback that the source should read like prose with paragraph breaks, not like a webpage, swap all multi-paragraph descriptions on the new files to // blocks. Single-line descriptors that were javadoc become a single // line above the declaration. Complete sentences throughout. No behavior change. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Server-side idempotency wiring is a follow-up; the comment now states that without naming an internal tracker. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
13f1140 to
ecc65f8
Compare
Summary
After a successful Iceberg snapshot commit, the tables service runs registered
post-commit operations against the freshly-saved
TableDto. The first operation isthe optimizer stats push (
PUT /v1/optimizer/stats/{tableUuid}), opting in per tablevia
maintenance.optimizer.stats.enabled=trueand gated globally bytables.postcommit.enabled(defaultfalse).The framework is one generic dispatcher; individual operations describe payload +
endpoint and let the dispatcher own async / timeout / error swallow / metrics.
PostCommitDispatcher(
tables.postcommit.per-op-timeout-ms, default 3000 ms).never propagated to the commit caller.
Snapshot.summary()already in memory atload time, captured on
TableDto.currentSnapshotasOptional<CurrentSnapshotInfo>(snapshot id + summary).
OptimizerStatsPostCommitOperationretains its own per-attempt timeout +bounded retry on retryable errors (network, 408, 429, 5xx,
TimeoutException).Why async (the explicit justification)
A synchronous post-commit hook converts an optimizer outage into a tables-service
write outage. The stats push is a best-effort scheduling signal, not a
write-correctness step — its blast radius must not include the write path. The
crash-loss window (JVM dies between commit and HTTP push completion) is acceptable
because operations are cumulative: the next commit re-pushes the current state, and
the consumer self-corrects from missing data.
The complexity stays bounded by living in one place (
PostCommitDispatcher). Newoperations register without re-deriving async plumbing; reviewers reason about
async semantics once.
Configuration
tables.postcommit.enabledfalsetables.postcommit.per-op-timeout-msoptimizer.stats.enabledfalseoptimizer.stats.base-uritrue.optimizer.stats.per-attempt-timeout-msoptimizer.stats.max-attemptsMetrics (Micrometer, tagged
op={operation name})postcommit_op_durationop,outcome={success, timeout, network_error, server_error, client_error, unknown_error}postcommit_op_skippedoppostcommit_op_failedop,outcome={timeout, network_error, server_error, client_error, prepare_threw, unknown_error}Wiring
PostCommitDispatcher,OptimizerStatsConfig(WebClient bean), andOptimizerStatsPostCommitOperationare each@ConditionalOnProperty— when theirflag is
falseno bean is constructed.IcebergSnapshotsServiceImplconsumes thedispatcher as
Optional<PostCommitDispatcher>and the on-commit hook is a literalno-op when absent.
Follow-up
The wire body already carries
snapshotId. Server-side dedupe (use the snapshot IDas an idempotency token on upsert and reject out-of-order replays) is intentionally
deferred to a follow-up change so this PR stays focused on the framework + the
first operation.
Changes
Internal API: new
services/tables/.../services/postcommit/package(
PostCommitOperationinterface,PostCommitDispatcher,PostCommitProperties);new
services/tables/.../model/CurrentSnapshotInfo;TableDto.currentSnapshotreplaces
currentSnapshotSummary; new metric keys inMetricsConstant;TableStatsControllerpublishesBASE_PATH/TABLE_PATH_TEMPLATEconstants.New features: opt-in post-commit operation framework + the first operation
(optimizer stats push), each independently gated by config.
Refactor: optimizer-stats client recast as a
PostCommitOperationimpl; outertimeout,
onErrorResume, and.subscribe()removed from the operation and movedinto the dispatcher.
Tests: new
PostCommitDispatcherTest(6 cases);OptimizerStatsClientTestrenamedto
OptimizerStatsPostCommitOperationTest(6 cases adapted to theprepare()contract);
IcebergSnapshotsServiceTestupdated;TableDtoMappingTestupdated.Testing Done
PostCommitDispatcherTest— 6 cases, all synchronous viadecorate(...).block(BLOCK_MAX):dispatch_invokesEveryRegisteredOperationsPreparedecorate_emptyPrepare_incrementsSkippedAndReturnsEmptydecorate_successfulWork_recordsDurationWithSuccessOutcomedecorate_workSignalsError_incrementsFailedWithClassifiedOutcomeAndSwallowsErrordecorate_prepareThrowsSynchronously_incrementsFailedAndDispatchContinuesToLaterOpsdecorate_workExceedsPerOpTimeout_incrementsFailedWithTimeoutOutcomeOptimizerStatsPostCommitOperationTest— 7 cases (1 stable-name + 6 behavior):name_isStableTagValueprepare_optedIn_emitsRequestWithFullPayload(asserts wire body field-for-field, including newsnapshotId)prepare_notOptedIn_returnsEmptyprepare_noSnapshot_returnsEmptyprepare_5xxThenSuccess_retriesAndCompletesprepare_allAttemptsFail_propagatesUnderlyingErrorprepare_4xxNonRetryable_doesNotRetryIcebergSnapshotsServiceTest.testPostCommitDispatcherInvokedAfterSuccessfulCommitverifies
dispatcher.dispatch(savedDto)is invoked exactly once after a successfulsave (replaces the prior
OptimizerStatsClient.reportassertion).TableDtoMappingTest—currentSnapshotadded toFIELDS_UNMAPPABLE.Local build:
:services:tables:test+:services:optimizer:testBUILD SUCCESSFUL;:services:tables:spotlessCheck,:services:common:spotlessCheck,:services:optimizer:spotlessCheckclean.Additional Information
Both
tables.postcommit.enabledandoptimizer.stats.enableddefault tofalse,so existing deployments see no behavioural change. The feature only activates when
the global dispatcher flag, the operation's flag, and the per-table opt-in property
are all set.