You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
As Acero grows in complexity it will be difficult for new developers to be able to contribute meaningfully. In addition, Acero should be open for extension by third party developers that wish to add new exec nodes. These 3rd party developers will need to know details on how Acero schedules work and operates and will appreciate advice on efficient development. At a minimum this first pass should explain:
Threading / Scheduling model for Acero (note, there are proposals to enhance the model we currently have)
Discussion of batch sizes and cache sizes and the morsel / batch model
General discussion / advice for writing operators in a column-major way
Design of current nodes, in particular, some more detail around how expression evaluation happens and how the hash-join node operates
Apache Arrow JIRA Bot:
This issue was last updated over 90 days ago, which may be an indication it is no longer being actively worked. To better reflect the current state, the issue is being unassigned per project policy. Please feel free to re-take assignment of the issue if it is being actively worked, or if you plan to start that work soon.
### Rationale for this change
The documentation for Acero was incomplete. This PR refactors the existing documentation and adds several entirely new sections to form a complete design document for Acero.
### What changes are included in this PR?
Some existing documentation is cleaned up. Acero documentation is moved into its own folder and broken into several pages.
### Are these changes tested?
The documentation is built as part of the CI but I wouldn't say it is fully tested.
### Are there any user-facing changes?
There are not code changes (other than the removal of two legacy methods) but there are many user-facing documentation changes.
* Closes: #32335
Lead-authored-by: Weston Pace <weston.pace@gmail.com>
Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
Co-authored-by: Benjamin Kietzman <bengilgit@gmail.com>
Signed-off-by: Weston Pace <weston.pace@gmail.com>
As Acero grows in complexity it will be difficult for new developers to be able to contribute meaningfully. In addition, Acero should be open for extension by third party developers that wish to add new exec nodes. These 3rd party developers will need to know details on how Acero schedules work and operates and will appreciate advice on efficient development. At a minimum this first pass should explain:
Reporter: Weston Pace / @westonpace
Note: This issue was originally created as ARROW-17023. Please see the migration documentation for further details.
The text was updated successfully, but these errors were encountered: