Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support a channel as a Sink #270

Open
V0ldek opened this issue Sep 18, 2023 · 1 comment
Open

Support a channel as a Sink #270

V0ldek opened this issue Sep 18, 2023 · 1 comment
Assignees
Labels
acceptance: go ahead Reviewed, implementation can start area: result Improvements in query result reporting type: feature New feature or request
Milestone

Comments

@V0ldek
Copy link
Member

V0ldek commented Sep 18, 2023

Is your feature request related to a problem? Please describe.
There is interest in using the engine as a pull-based iterator (#243). While that is hard to achieve, we can have a partial solution of supporting a channel between threads as a Sink. A user can then spin the engine in a separate thread and then use the other end of the channel as a source of results from which to pull.

It might be beneficial to check if one of those impls should be used in rq itself. If the overhead is small, we might default to it on large files where writing the results to output may be a heavy operation. This might also be an option for online output.

Describe the solution you'd like
We should have two implementations.

  1. std::sync::mpsc, the stdlib implementation of a channel.
  2. crossbeam_channel, the most popular channel crate.

The other should be behind a feature flag to enable the optional dependency – we won't force users to pay for it if they don't need it.

Describe alternatives you've considered
We could implement Sink on other primitives, like crossbeam's ArrayQueue, but that seems too specific. People trying to get a lower-level primitive to work with the engine probably would be better suited by a custom Sink impl anyway.

@V0ldek V0ldek added type: feature New feature or request mod: engine area: result Improvements in query result reporting labels Sep 18, 2023
@github-actions github-actions bot added the acceptance: triage Waiting for owner's input label Sep 18, 2023
@github-actions
Copy link

Tagging @V0ldek for notifications

@V0ldek V0ldek added this to the v1.0.0 milestone Sep 18, 2023
@github-actions github-actions bot added acceptance: go ahead Reviewed, implementation can start and removed acceptance: triage Waiting for owner's input labels Sep 18, 2023
@V0ldek V0ldek self-assigned this Sep 23, 2023
@V0ldek V0ldek modified the milestones: v1.0.0, v1.1.0 Sep 25, 2023
@V0ldek V0ldek removed the mod: engine label Oct 4, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
acceptance: go ahead Reviewed, implementation can start area: result Improvements in query result reporting type: feature New feature or request
Projects
Development

No branches or pull requests

1 participant