Skip to content

Enable parallel processing in synchronous pagination #404

@bevzzz

Description

@bevzzz

Java's Stream API features a built-in support for parallel stream processing: calling myList.stream().parallel() will distribute the workload across the available threads.

CursorSpliterator which underpins our pagination implementation is returning null from its trySplit() method, which the Stream model understands as "this cannot be processed in parallel".
We want to unlock this possibility for our users.

There are several possible strategies for splitting the stream of Weaviate objects. See this discussion for more details.

Metadata

Metadata

Assignees

Labels

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions