[Data] Enable read-only Datasets to be executed on new execution backend

[scottjlee]

Why are these changes needed?

• Enable read-only datasets to run on new execution backend. We achieve this by executing the InputDataBuffer in isolation to fetch ReadTasks and any known BlockMetadata from the input Datasource or Reader. This logic is in execute_read_only_to_legacy_lazy_block_list().
• By default, use the new execution backend for all Datasets unless otherwise configured by the user.
• Copy input ops when creating a new LogicalOperator: [Data] Copy input LogicalOperators to avoid mutating their output dependencies #41468 No longer needed, see the linked PR for more details.

Related issue number

Checks

• I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
  • I've added any new APIs to the API Reference. For example, if I added a
    method in Tune, I've added it in doc/source/tune/api/ under the
    corresponding .rst file.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[raulchen]

LGTM.
By the way, this PR just reminds me an issue I saw the other day. For read-only datasets, when we call ds.count(), we should get the count from the metadata. However, currently ds.count() will trigger the execution for the entire dataset. Not sure if this is still an issue after this PR. Could you verify that when you have a chance? (not blocking this PR)

[scottjlee]

@raulchen I used parquet_metadata_resolution release test to debug the case you mentioned, I found that there is spilling/eventual OOM with the previous approach. Instead of executing the plan containing only InputDataBuffer, I tried a new approach by directly using the Read logical op's _datasource_or_legacy_reader.get_read_tasks() to construct the output LazyBlockList (logic in get_legacy_lazy_block_list_read_only()). Any concerns overall with this approach?

Test run of parquet_metadata_resolution took 37.18 seconds, which is consistent with the current runtimes. Also checked runtime and throughput of torch_batch_inference_1_gpu_10gb_raw, they are also consistent with current results.