[Datasets] Datasets GA - Docs and UX Push

[clarkzinzow]

Tracker for docs and UX push for Datasets GA.

Docs

• [P0] [Datasets] [Docs] Refactor docs structure + Getting Started vs. User Guide content #23088
• [P0] [Datasets] [Docs] Create data ecosystem overview #23089
• [P0] FAQ for Datasets
• [P0] End-to-end examples: simple ML ingest, scalable batch inference, etc.
• [P0] User guide overhaul - concentrate on datasets creation since this is the starting point for many Ray ML workflows.
• [P0] API docs audit
• [P0] Knob tuning and debugging guides
• [P0] Document Datasets resource provisioning model (i.e. that Datasets resource allocation is implicit, using cluster resources in the margins of explicit allocations to other libraries like Tune).
• [P0] De-emphasize global/windowed shuffling.
• [P0] Document lazy mode.
• [P1] Make the simple and Arrow dataset distinction more clear, including: (a) what are the differences, (b) what makes a dataset simple vs. Arrow (method of creation, outputs of mappers, etc.), (c) examples of (a) and (b).
• [P1] [Datasets] Document comparison between NVTabular, Petastorm #18233
• [P1] Document that batch_size in .map_batches() doesn't limit working set size, that windowing should be used instead for that.
• [P1] Emphasize in the tensor support docs that explicit casts are necessary when creating tensor columns in e.g. .map_batches().
• [P1] [Datasets] [Docs] Better documentation for filesystem arg in I/O APIs #23090
• [P1] Need better docs for .map() and .map_batches(), indicating that the former is for row-based mapping, the latter is for batch-based mapping (i.e. vectorized column operations and the like). Should be clear how data scientist users can map pandas operations in distributed way.
• [P1] Explicitly note that column selection isn't possible on Datasets, funnel users to batch mapping for columnar transformations and aggregation API for columnar aggregations.
• [P1] Follow strict semantics about row/partition reordering, and document them: [Datasets] [Bug] Preserve row/partition order for non-shuffling operations; document row/partition reordering for non-obvious shuffle operations #20329
• [P2] Mention that datasets aren't indexed in the docs

UX (code changes)

• [P1] Allow user to set output column names for aggregation results.
• [P1] Better progress reporting for I/O methods, users frequently encounter what they think are "hangs" when the work is in progress.
• [P1] [Dataset] Progress Bar not rendering correctly inside Ray actor #18103
• [P2] [Datasets] [Bug/UX] Better error messages for aggregations on unsupported types #22411

[jianoaix]

The issues here are either resolved, or not relevant since GA wasn't a goal now. I'm closing it.