[C++][Docs] Revise C++ Documentation

[asfimport]

Parent ticket for tasks that aim to revise C++ Arrow documentation.

 

Goal is to review the field of good documentation, and reformat/create pages as necessary to improve the C++ docs as a whole. Subtasks will tackle this iteratively, refer to those for more information. Suggestions from new users would be incredibly valued, due to their experiences being more likely to have been impacted by any lacking documentation.

Reporter: Kae Suarez / @ksuarez1423

Subtasks:

• [C++][Docs] Expand C++ Cookbook
• [C++][Docs] Research Documentation Formats
• [C++][Docs] Re-Organize the Existing Acero Streaming Engine Documentation
• [C++][Docs] Show examples of multiple Acero-compatible language APIs
• [C++][Docs] Verify Completeness of API Reference
• [C++][Docs] Plan Getting Started Section
• [C++][Docs] Create installation instructional documentation in Sphinx
• [C++][Docs] Create a 10 minutes to Arrow article
• [C++][Docs] Create tutorial content for basic Arrow, file access, compute, and datasets
• [C++][Docs] Create tutorial content for basic Flight usage
• [C++][Docs] Create tutorial content for Acero

_{Note: This issue was originally created as ARROW-17167. Please see the migration documentation for further details.}

[asfimport]

Antoine Pitrou / @pitrou:
I agree with the general sentiment, but one thing I would like to see first is an attempt at better architecting the C++ docs.
Right now they have two main sections:

• an auto-generated API section, which is fine

• a prose section that is more random and doesn't have a very delineated purpose (general introduction? user guide? detailed discussion of specific issues? exhaustive description of available functionality?)

  The prose sections turns out to be frustrating to maintain because there isn't a very consistent level of detail, and there's always a tension between making it immediately approachable and making it comprehensive enough.

[asfimport]

Kae Suarez / @ksuarez1423:
So we need to have a target for the level of detail and goal of the user guide, then? Would expanding the cookbook help make it so the user guide can be more targeted? I notice that right now it's stuck being a one-stop shop, which sometimes misses information due to the problem you're suggesting, probably.

I do think some things can be added that are generally of use regardless of how we change it in the future, like an installation section and a Getting Started section, just to get up to par with Python's Arrow documentation. However, I might be a bit over-eager, suggesting that, so I definitely want to keep this conversation open.

[asfimport]

Antoine Pitrou / @pitrou:
I think two main sections is probably too few for a proper separation of purposes. One nice example of clearly architected docs is the Dask docs : https://docs.dask.org/en/stable/

[asfimport]

Kae Suarez / @ksuarez1423:
I think the Dask docs are a pretty nice example, for sure, but that leads to the level of thinking about the docs as a whole – that kind of format change seems to lend itself well to the docs at a whole having this format duplicated, or applied at a higher level with usage for each language being in a section.

[asfimport]

Will Jones / @wjones127:
I agree the Dask model seems quite nice. It also seems like something we could do iteratively well. For example, we might start by focusing on a "Getting Started" section. And then separate our current user guide into a "Fundamentals" and "internals" section.

It doesn't seem necessary to try to apply such a framework across languages. For one thing, different languages have their own documentation conventions. For another, there are differing appetites for each Arrow language community to invest in docs at a given time; trying to convince each sub-community doesn't seem worth the effort IMO.

[asfimport]

Antoine Pitrou / @pitrou:
"Thinking about the docs as a whole" is precisely the kind of approach that I would like to encourage :-)

The more we pile up content under the current structure, the more painful it will be to migrate.

[asfimport]

Antoine Pitrou / @pitrou:
And, right, this is about C++ for now. If/when we're satisfied with the result, we can discuss apply learning to other language docs.

[asfimport]

Antoine Pitrou / @pitrou:
Also cc @jorisvandenbossche for other suggestions where to research.

[asfimport]

Kae Suarez / @ksuarez1423:
Just made https://issues.apache.org/jira/browse/ARROW-17170 to help organize this conversation further – for research, let's use that, and keep higher level discussion about direction here.

[asfimport]

Weston Pace / @westonpace:
Re: the query engine, in the "thinking about docs as a whole topic", I think there are a few different audiences, none of which are particularly well served at the moment.

• There seems to be (and I hope will continue to be) a certain audience that wants to extend Acero to fit their use case. This audience will be rather technical (often academic) and invested and want to know fairly low-level details of how the engine is designed, best practices for developing nodes, advice on profiling their extensions, etc. We have almost nothing here and this is the highest priority for me personally. I'm hoping to make some headway here next release.
• For user documentation, I am a little torn. There is some documentation today describing the different nodes and their options, and that could really be expanded. However, if Substrait ends up being the dominant interface to Acero then this API becomes internal and the audience changes (back to the people in the first bullet).
• Users that are planning on using the engine via Substrait also fall into two categories. I think there will always be some set of users that are planning on creating plans themselves. These users will need to know quite a bit about Substrait, but I don't know how much we want to re-document. These users will be very interested in knowing where & how Acero differs from Substrait (e.g. caveats & limitations).
• There will be another set of Substrait users that are using Acero via some kind of producer (e.g. Ibis). This set of users probably won't know much about Substrait. They won't be able to understand the limitations / caveats, but if there are significant limitations they will want to know about them in some way. They will be mostly interested in knowing how to configure Acero for performance, etc. and in the extensions that Acero adds to Substrait (e.g. integration with other APIs like datasets)

[asfimport]

Vibhatha Lakmal Abeykoon / @vibhatha:
@westonpace  I agree with the bullet point 1.  I think such users could be lesser compared to the users using the high-level details. Although, I think that's the part where ACERO would be adopted to other engines in a much deeper level. Sometimes, Substrait won't be an interest to a group working on a very much constrained and defined set of computations which uses in-house systems. Besides, for research purpose, I guess still the internal documentation would be very important to give a head-start to new users and existing users to track the modifications. 

So either way, I think we should document well what ACERO is doing internally. And make them avaialble in developer docs. 

I am not sure if we have pursue this avenue before. There are user docs and developer docs. User docs can include high level details and how it works with other systems in harmony. That's more or less-like tutorial oriented and API doc oriented piece. The developer docs can go to any depth as required. I would say if we can define a boundary and expose from that level, it would be good for new ACERO contributors and the existing community. 

Also, I think we have marked that the ACERO docs are experimental and this means a lot of changes are going to come. So if we try to document things as it is we must also make sure that each PR for ACERO must handle a follow up sub-task which updates the documentation as well. 

For instance if the ticket is to update a feature A, then there has to be a subtask to include the feature and another sub-task to update the documentation. If it comes to the same task, we could miss the documentation piece. But if there is a dedicated task that won't be missed. Again this is just a suggestion how to iteratively improve things. 

And as @pitrou  and @wjones127  suggested the Dask docs are pretty nicely organized. We could adopt a similar pattern. 

[asfimport]

Vibhatha Lakmal Abeykoon / @vibhatha:
As an initial step, I added a sub-task to clean-up the length execution-plan-documentation-examples. It is listed in the substaks for this jira. 

[asfimport]

Vibhatha Lakmal Abeykoon / @vibhatha:
@ksuarez1423  thank you for initiating this one. It is a much needed effort IMHO. 

[asfimport]

Kae Suarez / @ksuarez1423:
It seems that https://diataxis.fr/ as worked for multiple teams when it comes to writing documentation – in some cases, it's augmented for a given use case, but serves as a competent base. Thus, I propose we keep it in mind as we increment the docs towards a better form.

As a start, calling back to what @pitrou said at the start of this, we could stand to have better structure. To do that incrementally, I propose we have a proper introduction "getting started" section of the docs, that way the existing docs can shift towards being a larger tutorial or targeted tutorials – basically, break out some responsibility so we can be more precise with next steps. Of course, we can re-align targets over time, but I think that we'd at least get some parity with the other Arrow docs by adding the introduction section. Any thoughts before I throw that up as a task here on JIRA?

[asfimport]

Aldrin Montana / @drin:
I think a getting started section sounds great.

In my mind, it would be a nice place to put sub-sections such as:

• "getting data into Arrow"

• "ways to process Arrow data"

  I think as sub-sections these could introduce some concepts, how they relate, and point to places in the docs that describe them in more detail. IMO, a nice "getting started" also helps bootstrap enough knowledge to navigate the docs.