ARROW-6165: [Integration] Run integration tests on multiple cores

On a local test with C++, Java, Flight and Gold files enabled, the runtime goes down from 1m56 (serial) to 35s (parallel).

Closes #6416 from pitrou/ARROW-6165-parallel-integration and squashes the following commits:

3ac06e5 <Antoine Pitrou> ARROW-6165:  Run integration tests on multiple cores

Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Wes McKinney <wesm+git@apache.org>

.github/workflows/integration.yml

@@ -22,7 +22,9 @@ on:
     paths:
       - '.github/workflows/integration.yml'
       - 'ci/**'
+      - 'dev/archery/**'
       - 'go/**'
+      - 'integration/**'
       - 'js/**'
       - 'cpp/**'
       - 'java/**'
@@ -31,7 +33,9 @@ on:
     paths:
       - '.github/workflows/integration.yml'
       - 'ci/**'
+      - 'dev/archery/**'
       - 'go/**'
+      - 'integration/**'
       - 'js/**'
       - 'cpp/**'
       - 'java/**'

ci/docker/conda-integration.dockerfile

@@ -39,6 +39,7 @@ ENV GOROOT=/opt/go \
 RUN wget -nv -O - https://dl.google.com/go/go${go}.linux-${arch}.tar.gz | tar -xzf - -C /opt
 
 ENV ARROW_BUILD_INTEGRATION=ON \
+    ARROW_BUILD_TESTS=OFF \
     ARROW_FLIGHT=ON \
     ARROW_ORC=OFF \
     ARROW_DATASET=OFF \

dev/archery/archery/cli.py

@@ -498,8 +498,10 @@ def _set_default(opt, default):
               help='Generate test JSON to indicated path')
 @click.option('--run-flight', is_flag=True, default=False,
               help='Run Flight integration tests')
-@click.option('--debug', type=bool, default=False,
+@click.option('--debug', is_flag=True, default=False,
               help='Run executables in debug mode as relevant')
+@click.option('--serial', is_flag=True, default=False,
+              help='Run tests serially, rather than in parallel')
 @click.option('--tempdir', default=None,
               help=('Directory to use for writing '
                     'integration test temporary files'))