[train] Speed up downloading of large checkpoints

[ericl]

Why are these changes needed?

This PR sets chunk_size and use_threads=True (download only) for pyarrow.fs.copy_files, speeding up the downloading by 2-3x.

Some benchmarks (bold is what this PR implements as the default for Train/Tune):

• pyarrow's default S3FileSystem, 64MiB chunk size, use_threads=F, download: ~50MiB/s
• pyarrow's default S3FileSystem, 64MiB chunk size, use_threads=F, upload: ~700MiB/s
• pyarrow's default S3FileSystem, 64MiB chunk size, use_threads=T, download: ~120MiB/s
• pyarrow's default S3FileSystem, 64MiB chunk size, use_threads=T, upload: hangs due to [Python] Python hangs when use pyarrow.fs.copy_files with "used_threads=True" apache/arrow#32372
• fsspec/s3_fs, 8MiB multipart chunk size, download: ~120MiB/s
• fsspec/s3_fs, 8MiB multipart chunk size, upload: ~300MiB/s

Note that aws s3 sync does better, and can hit ~700MiB/s for both upload and download, but this is out of the scope for this fix for 2.7:

awsv2 configure set default.s3.preferred_transfer_client crt
awsv2 configure set default.s3.target_bandwidth 100Gb/s
awsv2 configure set default.s3.multipart_chunksize 8MB

awsv2 s3 sync s3://... /tmp/ckpt

Completed 15.4 GiB/69.4 GiB (632.9 MiB/s) with 14 file(s) remaining

Closes #38612

[justinvyu]

Thanks! One comment

[justinvyu]

Thanks!