[Python] Nightly or on-demand HDFS test builds

[asfimport]

We continue to acquire more functionality related to HDFS and Parquet. Testing this, including tests that involve interoperability with other systems, like Spark, will require some work outside of our normal CI infrastructure.

I suggest we start with testing the C++/Python HDFS integration, which will help with validating patches like ARROW-1643 #1668

Reporter: Wes McKinney / @wesm

_{Note: This issue was originally created as ARROW-2248. Please see the migration documentation for further details.}

[asfimport]

Wes McKinney / @wesm:
cc @nealrichardson

[asfimport]

Neal Richardson / @nealrichardson:
Doesn't crossbow already have this? I just created ARROW-6434 about fixing the broken HDFS integration test job.

[asfimport]

Wes McKinney / @wesm:
Yes, this is available in docker-compose and via Crossbow