Skip to content

Commit

Permalink
dbsp: Fix hidden dependency on vec-based batches in input_upsert.
Browse files Browse the repository at this point in the history
The vec-based batches set keys, values, and weights to their default
values when their builders accept tuples via `Builder::push` or
`Builder::push_vals`, which take mutable references.  The code for
`input_upsert` accumulates updates to keys in a `key_updates` vector,
and then adds them to a builder via `extend`, which internally uses
`Builder::push` and thus clears all the updates that were added to have
weight 0.

Before this commit, that code didn't later clear the `key_updates` vector,
which meant that anything added later was appended to a bunch of zeroed
tuples.  This was benign because, the next time `key_updates` was used,
it was first consolidated, which meant that all the zero-weight tuples
were dropped.

This failed with the file-based batches, which don't have any special
optimizations for mutable references and thus don't zero anything, which
meant that nothing in `key_updates` ever got cleared.  This commit fixes
the problem by properly clearing `key_updates` to empty after using it.

Signed-off-by: Ben Pfaff <blp@feldera.com>
  • Loading branch information
blp committed Jun 20, 2024
1 parent 41ac979 commit 4a5a3d6
Showing 1 changed file with 1 addition and 0 deletions.
1 change: 1 addition & 0 deletions crates/dbsp/src/operator/dynamic/input_upsert.rs
Original file line number Diff line number Diff line change
Expand Up @@ -389,6 +389,7 @@ where
}
key_updates.consolidate();
builder.extend(key_updates.dyn_iter_mut());
key_updates.truncate(0);
}

skip_key = false;
Expand Down

0 comments on commit 4a5a3d6

Please sign in to comment.