ARROW-9458: [Python] Release GIL in ScanTask.execute

[maartenbreddels]

@jorisvandenbossche FYI I made you coauthor

[jorisvandenbossche]

Thanks for the PR! (OK, I was missing the with gil inside the for loop in my attempt ..)

(no worries about the authorship, I have my commits in arrow ;))

[jorisvandenbossche]

And I can confirm using scan tasks now gives a similar parallelization speed up as using fragments.

[maartenbreddels]

FYI, this:

## common code ##
import pyarrow as pa
import pyarrow.dataset as ds
import concurrent.futures
import glob
pool = concurrent.futures.ThreadPoolExecutor()
ds = pa.dataset.dataset(glob.glob('/data/taxi_parquet/data_*.parquet'))
## end common code ##

def process(f):
    scan_count = 0
    return len(f.to_table(use_threads=False))
sum(pool.map(process, ds.get_fragments()))

For me takes between 10 and 16 second (very irregular)

While this:

def process(fragment):
    scanned = 0
    for scan_task in fragment.scan(use_threads=False):
        for record_batch in scan_task.execute():
            scanned += record_batch.num_rows
    return scanned
sum(pool.map(process, ds.get_fragments()))

takes 7-9 seconds, more consistently.

Each file (fragment) has 1 million rows (1 rowgroup). Could be uninteresting details, but I though I'd share it.

[github-actions]

https://issues.apache.org/jira/browse/ARROW-9458

[wesm]

+1