[Docs] Document conventions for sending and receiving Arrow data over HTTP APIs

[ianmcook]

Describe the enhancement requested

The Arrow developer community intends to publish a set of conventions in the Arrow docs for how to send and receive Arrow-format data over HTTP APIs. There is a related discussion on the Arrow developer mailing list at https://lists.apache.org/thread/886cnx6ytjst3smmytz4r4ddcbv95191.

Tasks

This issue is an umbrella for tasks that are a part of this effort.

---

Simple HTTP GET client and server examples

📁 arrow-experiments/tree/main/http/get_simple

• [GLib] Create simple example of C GLib HTTP GET Arrow client #40466
• [C++] Create simple example of C++ HTTP GET Arrow client #40467
• [C++] Create simple example of C++ HTTP GET Arrow server #40472
• [C#] Create simple example of C# HTTP GET Arrow client #40480
• [C#] Create simple example of C# HTTP GET Arrow server #40481
• [Go] Create simple example of Go HTTP GET Arrow client #40468
• [Go] Create simple example of Go HTTP GET Arrow server #40469
• [Java] Create simple example of Java HTTP GET Arrow client #40470
• [Java] Create simple example of Java HTTP GET Arrow server #40471
• [JS] Create simple example of JavaScript HTTP GET Arrow client #40473
• [JS] Create simple example of JavaScript Node.js HTTP GET Arrow server arrow-js#64
• Arrow-over-HTTP client and server examples in Julia arrow-julia#502
• [MATLAB] Create simple example of MATLAB HTTP GET Arrow client #40489
• [Python] Create simple example of Python HTTP GET Arrow client #40475
• [Python] Create simple example of Python HTTP GET Arrow server #40476
• [Python] Create simple HTTP server example using Flask #40611
• [Python] Create simple HTTP server example using FastAPI #40869
• [R] Create simple example of R HTTP GET Arrow client #40477
• [Ruby] Create simple example of Ruby HTTP GET Arrow client #40478
• [Ruby] Create simple example of Ruby HTTP GET Arrow server #40479
• Prototype Arrow over HTTP in Rust arrow-rs#5496
• [Swift] Create simple example of Swift HTTP GET Arrow client arrow-swift#13
• [Docs][HTTP] Clarify when to use batch-at-a-time vs. one-shot approach for receiving data #40613

---

HTTP GET client and server examples demonstrating range requests

📁 arrow-experiments/tree/main/http/get_range

• [JS] Create examples of HTTP GET Arrow client/server supporting range requests #40597

---

Indirect response HTTP GET client and server examples

📁 arrow-experiments/tree/main/http/get_indirect

• [Python] Create Python examples of indirect HTTP GET Arrow client and server #40596

---

Multipart/mixed response HTTP GET client and server examples

📁 arrow-experiments/tree/main/http/get_multipart

• [Python] Create Python examples of HTTP GET Arrow client/server supporting multipart/mixed response #40598

---

HTTP GET examples to test different compression options

📁 arrow-experiments/tree/main/http/get_compressed

• [Python] Create Python examples of HTTP GET Arrow client and server using IPC buffer compression #40600
• [Python] Create Python examples of HTTP GET Arrow client and server using HTTP compression #40601
• [Docs][HTTP] Track status of IPC buffer compression support in Arrow libraries #40612

---

Simple HTTP PUT / POST client and server examples

📁 arrow-experiments/tree/main/http/post_simple

• [Python] Create simple example of Python HTTP POST Arrow client and server #40595

---

Multipart/form-data request HTTP PUT / POST client and server examples

📁 arrow-experiments/tree/main/http/post_multipart

• [Python] Create Python examples of HTTP POST Arrow client/server supporting multipart/form-data request  #40599

---

General issues and questions

• [Python] Verify that terminating sequences in binary IPC stream do not interfere with HTTP/1.1 transport #40581
• [Docs][HTTP] Recommend method(s) for resuming interrupted uploads? #44031

---

Component(s)

Documentation

[felipecrv]

Is this the main issue where I can track your HTTP+Arrow initiative @ianmcook?

[ianmcook]

Is this the main issue where I can track your HTTP+Arrow initiative @ianmcook?

Yes

[CurtHagenlocher]

Any thoughts on the "best" canonical way to return multiple record batches? Would that also be multipart/mixed or would it be better to avoid the delimiter problem and e.g. use an alternate Content-Type indicating that the response contains multiple streams?

[ianmcook]

Any thoughts on the "best" canonical way to return multiple record batches? Would that also be multipart/mixed or would it be better to avoid the delimiter problem and e.g. use an alternate Content-Type indicating that the response contains multiple streams?

@CurtHagenlocher Let's discuss this at #40596 (comment)