Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[C++] Add initial Acero design documents #32335

Closed
asfimport opened this issue Jul 8, 2022 · 1 comment · Fixed by #35320
Closed

[C++] Add initial Acero design documents #32335

asfimport opened this issue Jul 8, 2022 · 1 comment · Fixed by #35320

Comments

@asfimport
Copy link
Collaborator

As Acero grows in complexity it will be difficult for new developers to be able to contribute meaningfully. In addition, Acero should be open for extension by third party developers that wish to add new exec nodes. These 3rd party developers will need to know details on how Acero schedules work and operates and will appreciate advice on efficient development. At a minimum this first pass should explain:

  • Threading / Scheduling model for Acero (note, there are proposals to enhance the model we currently have)
  • Discussion of batch sizes and cache sizes and the morsel / batch model
  • General discussion / advice for writing operators in a column-major way
  • Design of current nodes, in particular, some more detail around how expression evaluation happens and how the hash-join node operates

Reporter: Weston Pace / @westonpace

Note: This issue was originally created as ARROW-17023. Please see the migration documentation for further details.

@asfimport
Copy link
Collaborator Author

Apache Arrow JIRA Bot:
This issue was last updated over 90 days ago, which may be an indication it is no longer being actively worked. To better reflect the current state, the issue is being unassigned per project policy. Please feel free to re-take assignment of the issue if it is being actively worked, or if you plan to start that work soon.

westonpace added a commit that referenced this issue May 19, 2023
### Rationale for this change

The documentation for Acero was incomplete.  This PR refactors the existing documentation and adds several entirely new sections to form a complete design document for Acero.

### What changes are included in this PR?

Some existing documentation is cleaned up.  Acero documentation is moved into its own folder and broken into several pages.

### Are these changes tested?

The documentation is built as part of the CI but I wouldn't say it is fully tested.

### Are there any user-facing changes?

There are not code changes (other than the removal of two legacy methods) but there are many user-facing documentation changes.
* Closes: #32335

Lead-authored-by: Weston Pace <weston.pace@gmail.com>
Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
Co-authored-by: Benjamin Kietzman <bengilgit@gmail.com>
Signed-off-by: Weston Pace <weston.pace@gmail.com>
@westonpace westonpace added this to the 13.0.0 milestone May 19, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants