GH-41673: [Format][Docs] Add arrow format introductory page

[AlenkaF]

Rationale for this change

The documentation for Arrow Format could be improved:

• all types are not listed
• all layouts are not explained

What changes are included in this PR?

This PR includes:

• motivation behind the columnar format
• different physical layouts explained together with diagrams of example type in comparison to the physical layout
• Arrow terminology
• Extension types and sharing of Arrow data

in a separate "introduction" page with no technical details. Specifications index page is also restructured to include captions and make the left sidebar menu better organised.

Note: a table with all types listed together with their physical layout will be added in a separate PR to existing Columnar.rst page: #14752

Are these changes tested?

No, this is a docs change.

Are there any user-facing changes?

No.

• GitHub Issue: [Docs][Format] Add an introductory page to the Arrow Columnar Format #41673

[AlenkaF]

cc @amoeba this could use a look already. I think all I wanted to add is here. Will need to do a general look through one more time before marking it ready for review though.

[AlenkaF]

@github-actions crossbow submit preview-docs

[github-actions]

Revision: 3cdd97a

Submitted crossbow builds: ursacomputing/crossbow @ actions-4a1cc2326d

Task	Status
preview-docs

[AlenkaF]

@jorisvandenbossche have kept C Stream Interface in a separate file as the structure is nicer IMHO.

[AlenkaF]

@github-actions crossbow submit preview-docs

[github-actions]

Revision: 4d2bf8a

Submitted crossbow builds: ursacomputing/crossbow @ actions-cc7da250f4

Task	Status
preview-docs

[AlenkaF]

Not sure why the captions in the left sidebar menu are not visible in the crossbow preview build:

---

but are visible for me locally:

[AlenkaF]

Update: I have removed the change in docs/source/format/index.rst (captions for the Specifications section) and will move it to a separate PR, see 97e4217.

[AlenkaF]

@github-actions crossbow submit preview-docs

[github-actions]

Revision: 9f9bbff

Submitted crossbow builds: ursacomputing/crossbow @ actions-cee8fb4563

Task	Status
preview-docs

[AlenkaF]

Fresh link to the html version: http://crossbow.voltrondata.com/pr_docs/41593/format/Intro.html

[jorisvandenbossche]

Two comments here (cc @felipecrv as I think you suggested this paragraph):

• If we mention inter-process communication here, should we also mention zero-copy within-processing sharing? (i.e. what the C Data Interface provides). Also, I know we generally say about the IPC protocol that it is zero copy, but of course it's not entirely zero copy, so mentioning it twice in context of IPC is maybe a bit too much
• I find the mention of "more efficient reading and writing of file formats" a bit out of place now, because it's not really the format itself that enables to read eg a Parquet file more efficiently? (it's that many Arrow implementation will provide this functionality as well, and that we can more easily reuse such implementations if they read into Arrow format)

[jorisvandenbossche]

I would add here something like (to connect the array with buffer concepts): "An array consists of one or more buffers"

[jorisvandenbossche]

I would explicitly call out here that this is depicted that way in the diagrams

[jorisvandenbossche]

Can you expand this explanation a bit?

(I think the main point is that in addition to the offsets buffer, there is now also a sizes buffer. The offsets still indicate the start of each element, but the size is not inferred from the next offset value, but now coded explicitly in a separate sizes buffer. That allows to have out-of-order offsets.