Skip to content

fix(query-engine): account for CountMinSketchWithHeap memory usage #684

Description

@milindsrivastava1997

Parent

#678

What to build

Make CountMinSketchWithHeap memory accounting reflect the actual allocation, including CMS cells, heap entries, and stored key strings. The estimate must use the backend’s actual cell representation rather than assuming f64 storage.

Acceptance criteria

  • CMS matrix memory is included.
  • Heap capacity and heap-entry metadata are included.
  • Key storage is included or explicitly bounded.
  • Accounting reflects the actual backend cell type.
  • Tests verify accounting for different dimensions and heap sizes.

Blocked by

None - can start immediately.

Concrete example

The current estimate is effectively:

size_of::<CountMinSketchWithHeapAccumulator>()
    + row_num * col_num * size_of::<f64>()

For a 3x1024 sketch with heap capacity 1000, this omits the heap’s 1000 entries plus their key strings and metadata. It also models cells as f64 even though the heap backend stores integer counts.

The corrected estimate should either measure actual allocations or document a conservative bound that includes the heap and key storage.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions