[Data][Docs] Revise Transforming data

[bveeramani]

Why are these changes needed?

Transforming data is verbose. It also contains information that's irrelevant for most users (for example, the fact repartition is an all-to-all operation). This PR revises the guide for brevity and discoverability.

Related issue number

Checks

• I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
  • I've added any new APIs to the API Reference. For example, if I added a
    method in Tune, I've added it in doc/source/tune/api/ under the
    corresponding .rst file.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[ericl]

I think we still need to cover batch_size and the notion of partitioning somehow. These are the two knobs that users most often get wrong and are both critical for performance, so it is hard to avoid these topics.

[ericl]

Looks pretty good! One comment on ensuring we address performance tuning.

[amogkam]

lgtm, thanks!

but +1 on not dropping the batch size configuration and repartitioning sections. In the repartitioning section, we should explicitly warn against using this unless necessary. And instead recommend to configure the parallelism as part of the read api.

[bveeramani]

In the repartitioning section, we should explicitly warn against using this unless necessary. And instead recommend to configure the parallelism as part of the read api.

@amogkam Wait, why do we prefer parallelism over repartition?

[amogkam]

@ericl can say more, but my understanding is that it essentially pushes the parallelism into the read operation. Instead of require an another repartition after.

[ericl]

Hmm I don't think we should recommend users set parallelism manually ever. The heuristic is pretty much always better than what users choose themselves.

However parallelism has one issue now where it cannot increase parallelism past the number of files. In this case, repartitioning is the only way you can actually increase the effective parallelism.

This is a bit unfortunate and I think we can fix this by auto repartitioning in these cases, but for now we probably still need to document repartitioning.

[amogkam]

nice lgtm!

[bveeramani]

Failing doctest test is unrelated

______________________________ [doctest] faq.rst _______________________________
--
  | 082    gets propagated to all training workers.
  | 083
  | 084 .. testcode::
  | 085
  | 086     import ray
  | 087
  | 088     # Add this at the top of your Ray application.
  | 089     runtime_env = {"env_vars": {"NCCL_SOCKET_IFNAME": "ens5"}}
  | 090     ray.init(runtime_env=runtime_env, ignore_reinit_error=True)
  | 091
  | UNEXPECTED EXCEPTION: AttributeError("'NoneType' object has no attribute 'address_info'")
  | Traceback (most recent call last):
  | File "/opt/miniconda/lib/python3.7/site-packages/pytest_sphinx.py", line 457, in _DocTestRunner__run
  | test.globs,
  | File "<doctest faq.rst[0]>", line 5, in <module>
  | File "/ray/python/ray/_private/client_mode_hook.py", line 103, in wrapper
  | return func(*args, **kwargs)
  | File "/ray/python/ray/_private/worker.py", line 1529, in init
  | return RayContext(dict(_global_node.address_info, node_id=node_id.hex()))
  | AttributeError: 'NoneType' object has no attribute 'address_info'
  | faq.rst:91: UnexpectedException