Dockerize verify-release-candidate.{sh,bat}

[asfimport]

There are a number of issues with the linux version of this script that would disappear if the commands were all being run in a docker container.

Anyone with docker installed should be able to verify the release candidate

We could probably do the same for windows as well.

Reporter: Phillip Cloud / @cpcloud

_{Note: This issue was originally created as ARROW-2959. Please see the migration documentation for further details.}

[asfimport]

Phillip Cloud / @cpcloud:
Verifying the release candidate should be as simple as

docker pull apache/arrow-verify-release:latest
docker run apache/arrow-verify-release ./verify.sh

[asfimport]

Phillip Cloud / @cpcloud:
Obviously we might not be able to use that exact repo, but the idea remains the same

[asfimport]

Wes McKinney / @wesm:
The downside of this is that running in a Docker container will cause us to miss failures that are the result of issues on different platforms. Users may build and deploy Arrow in many different places

[asfimport]

Wes McKinney / @wesm:
As an example, I'm in the midst of trying to debug a failure on Xcode 6.4 while I'm using Ubuntu 14.04 and having no issues here apache/parquet-cpp#467

[asfimport]

Phillip Cloud / @cpcloud:
I don't think that it's reasonable to expect contributors to maintain a cross platform shell script that cannot be guaranteed to be run from a single common point of origin.

For example, the script doesn't work if you're inside of a non-base conda environment that has ARROW_HOME set to CONDA_PREFIX.

I should be able to verify the release for common platforms without having to think about what environment variables I have set or what state my shell is in.

Yes, arrow is built and deployed in a number of places but AFAICT most people who run the release verification script are working on Ubuntu or OS X so we're not catching anything (or people are running it and running into bugs and not reporting it) now that we would not catch by moving to docker for this.

[asfimport]

Phillip Cloud / @cpcloud:
Also, docker exists for Linux, OS X, and Windows, so I'm not sure what platforms we'd really be missing.

[asfimport]

Wes McKinney / @wesm:
I agree with you, but we should strive to make the script run on as many platforms as possible. It's good information to know whether there is a problem that is not found in the Linux distro used in the Dockerfile.

The script is intended to run in an environment that has not been polluted by any _HOME variables or anything else relating to Arrow development, ie simulating the experience of an average non-developer consumer.

[asfimport]

Uwe Korn / @xhochy:
I run this script on several different systems on the vote to see if there are any problems. Only running it in docker gives us the falsehood to believe it will run everywhere. But running the script only in docker will simply result in users reporting the same problems after the release as issues.

[asfimport]

Phillip Cloud / @cpcloud:
Unless we're actually testing this, the expectation to keep the script cross platform without making it known what systems this needs to run on and without providing a way for people to easily test across these platforms is unreasonable IMO.

Just to be clear, I personally am not going to invest time in bending over backwards to keep the script cross-undocumented-system compatible. I'm not going to stress out about keeping it that way for systems whose setup I have no way to reproduce.

I wasted a lot of time today only to figure out that the verification script can fail with a segfault when you are in a conda environment and have ARROW_HOME set. I don't think documenting this solves the problem, though it would have solved my immediate problem.

What is reasonable to me and what I think would solve the problem, is to provide a reliable way to set up each system we expect to be able to run on so that anyone can verify all supported platforms with minimal hassle.

Is there an alternative to docker for solving this problem in a way that is acceptable to everyone?

[asfimport]

Wes McKinney / @wesm:

Unless we're actually testing this

I think we should, in a nightly build

Just to be clear, I personally am not going to invest time in bending over backwards to keep the script cross-undocumented-system compatible. I'm not going to stress out about keeping it that way for systems whose setup I have no way to reproduce.

Other developers will be responsible for reporting problems and helping fix them. This is a shared responsibility and part of making sure Arrow runs in a lot of places.

I wasted a lot of time today only to figure out that the verification script can fail with a segfault when you are in a conda environment and have ARROW_HOME set. I don't think documenting this solves the problem, though it would have solved my immediate problem.

I wanted someone other than Uwe or me to suffer a bit as a release manager so that more problems like these can be better documented and fixed. So what needs to be said to prevent this particular issue is: "run this script in a clean, unadulterated environment" (I know this from my past experience as an RM but I didn't write this down)

What is reasonable to me and what I think would solve the problem, is to provide a reliable way to set up each system we expect to be able to run on so that anyone can verify all supported platforms with minimal hassle.

There's way too many possibilities. The best we can do is test the primary platforms (Ubuntu LTS releases, CentOS 6/7, Xcode 6.4 and higher, Windows MSVC 2015) and hope to catch more problems from users verifying the release on other platforms. For example, running these platforms would not have caught the Arch Linux issue that you reported from bleeding edge gcc.

Really all we are doing is cutting down on build/deploy problems that may occur after a release is cut.

Docker would help with automated testing a release candidate on the common platforms (Ubuntu, CentOS/RedHat/Fedora, Arch) but we still need to rely on others to verify the release on their platform and report problems.

[asfimport]

Krisztian Szucs / @kszucs:
We can do this in a similar fashion like #2755

[asfimport]

Wes McKinney / @wesm:
We still need to run the release verification in user environments, to make sure we haven't missed something that appears in a distribution-specific way. Finding a problem won't necessary invalidate a release, but it would be better to know than not know

[asfimport]

Wes McKinney / @wesm:
Closing as Won't Fix. The release verification process is regularly turning up issues that would be occluded by a Dockerized build (e.g. I found several problems on Ubuntu 14.04 for 0.12)

I think we should definitely make it more straightforward / simpler to set up the user environment to run the script though