[data] Adding in batch inference mock pipeline release test

[omatthew98]

Why are these changes needed?

We want to add a release test that will be used to model performance on a batch inference image pipeline. This is meant to more accurately model a realistic user pipeline. This will be used to track our improvement on this workload and more broadly our improvement on batch inference with images with and without spot nodes.

Related issue number

Checks

• I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
  • I've added any new APIs to the API Reference. For example, if I added a
    method in Tune, I've added it in doc/source/tune/api/ under the
    corresponding .rst file.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[bveeramani]

(Optionally for future reference): we can use the variants syntax to minimize duplication

ray/release/release_data_tests.yaml

Lines 211 to 232 in 251a131

	- name: distributed_training
	working_dir: nightly_tests
	cluster:
	byod:
	post_build_script: byod_install_mosaicml.sh
	cluster_compute: dataset/multi_node_train_16_workers.yaml
	run:
	timeout: 3600
	script: >
	python dataset/multi_node_train_benchmark.py --num-workers 16 --file-type parquet
	--target-worker-gb 50 --use-gpu
	variations:
	- __suffix__: regular
	- __suffix__: chaos
	run:
	prepare: >
	python setup_chaos.py --kill-interval 200 --max-to-kill 1 --task-names
	"_RayTrainWorker__execute.get_next"

[omatthew98]

These changes are from @aslonnie's trouble shooting here #52687.

[aslonnie]

:) thank you

you might want to double check again if the release test works