ARROW-15438: [Python] Flaky test test_write_dataset_max_open_files

[westonpace]

The test could fail when writing due to a race condition. If the batches were delivered AAAAABBBBBCCCCC... then by the time we need to close a file to make space we can close an already completed file (and so we won't have to open up a new one later) and we end up with 5 files for 5 partitions.

Adding use_threads=False to the write_dataset call was not sufficient. The arrow::dataset::FileSystemDataset::Write method was always using the CPU executor for the exec plan. In other scanner methods we base the CPU executor on the scan options (nullptr if scan_options->use_threads is false). Making both of these changes together seems to make the test reliably pass.

[github-actions]

https://issues.apache.org/jira/browse/ARROW-15438

[kszucs]

Thanks Weston!

@lidavidm could you please verify this locally?

[lidavidm]

Thanks for digging into this, Weston!

I ran it locally and it seems to be reliable now (at least, before it would fail within a few runs, now it runs for a while and I eventually just killed it)

[westonpace]

@kszucs Feel free to merge this if you want. This should not block RC6 as it is mostly a flaky test (the threading thing has some practical implications but they are minor)

[vibhatha]

Thanks for looking into this @westonpace 👍

[pitrou]

This fixes the failure for me while it could be reproduced quite reliably on master.

[ursabot]

Benchmark runs are scheduled for baseline = 79800d4 and contender = 5a51c6d. 5a51c6d is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2
[Finished ⬇️2.5% ⬆️0.0%] ursa-i9-9960x
[Finished ⬇️0.22% ⬆️0.04%] ursa-thinkcentre-m75q
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python. Runs only benchmarks with cloud = True
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java