[Packaging] Consider uploading tarballs to avoid naming conflicts

[asfimport]

@kou has suggested, that we should upload tarballs for linux package builds, to include all .deb and/or .rpm packages.
For conda packages currenlty we postfix the artifacts with the arch name to avoid naming conflicts. To remain consistent we should chose one of the strategies (postifx or tarballs) and apply for each tasks.

However tarballs would be a trivial solution to resolve naming conflicts, ensuring that a tarball contains all of the expected artifacts would require to download and extract it beforehand. This requires changes in status checks and artifact downloading / signing.

@cpcloud thoughts?

Reporter: Krisztian Szucs / @kszucs
Assignee: Krisztian Szucs / @kszucs

_{Note: This issue was originally created as ARROW-2792. Please see the migration documentation for further details.}

[asfimport]

Krisztian Szucs / @kszucs:
@wesm I need comments on this one.

[asfimport]

Wes McKinney / @wesm:
Uploading a tarball for each task seems reasonable. Since the deployment code is somewhat opaque for this https://github.com/apache/arrow/blob/master/dev/tasks/python-wheels/travis.linux.yml#L49, the tasks themselves would need to determine whether the set of produced artifacts is complete before uploading to GitHub

[asfimport]

Krisztian Szucs / @kszucs:
How about creating a release tag for each task (instead just one for the job)? That way We would avoid name collisions.
Should We be explicit about the artifact names? We can render the deploy segment with jinja2.

[asfimport]

Wes McKinney / @wesm:
That sounds fine to me (tag for each task). I don't have an opinion on whether to list the exact artifact names in the file; whatever's the simplest / most robust seems OK to me

[asfimport]

Krisztian Szucs / @kszucs:
@wesm We probably need to split the builds (define matrix in tasks.yml instead of the templates), because linux packages starts to timeout on travis and to prevent artifact collisions within a single task.

Something like:

  wheel-win:
    platform: win
    template: python-wheels/appveyor.yml
    matrix:
      cp35: # variant
        generator: Visual Studio 14 2015 Win64
        numpy: "1.13"
        python: "3.5"
      cp36:
        generator: Visual Studio 14 2015 Win64
        numpy: "1.13"
        python: "3.6"
    artifacts:
      - pyarrow-{version}-{variant}-{variant}m-win_amd64.whl
      - pyarrow-{version}-{variant}-{variant}m-win_amd64.whl

I'm not exactly sure on the matrix definition though.

[asfimport]

Phillip Cloud / @cpcloud:
@kszucs If we do that, do we lose the ability to run an individual artifact build? That is, would it still be possible to run just the "Python 3.5 windows wheel" build?

[asfimport]

Krisztian Szucs / @kszucs:
That's not possible currently :)
We can't specify to run only one variant of the matrix: https://github.com/apache/arrow/blob/master/dev/tasks/python-wheels/appveyor.yml

If We want to take over the matrix handling from the CIs, we need a good enough formalism to do that.

[asfimport]

Kouhei Sutou / @kou:
If we use different filenames for each distribution, we should do it by debian/changelog instead of outer script. We can do it by the following patch:

diff --git a/dev/tasks/linux-packages/apt/build.sh b/dev/tasks/linux-packages/apt/build.sh
index f489d026..fad5d16d 100755
--- a/dev/tasks/linux-packages/apt/build.sh
+++ b/dev/tasks/linux-packages/apt/build.sh
@@ -54,6 +54,13 @@ if [ -d "/host/tmp/${specific_debian_dir}" ]; then
 else
   run cp -rp "/host/tmp/debian" debian
 fi
+run \
+  sed \
+  -i"" \
+  -e"1 s/)/~${code_name}1)/" \
+  -e"1 s/unstable/${code_name}/" \
+  debian/changelog
+
# export DEB_BUILD_OPTIONS=noopt
 if [ "${DEBUG:-no}" = "yes" ]; then
   run debuild -us -uc

With this patch, we can generate
libarrow0_${VERSION}~${CODE_NAME}1_${ARCHITECTURE}.deb instead of
libarrow0_${VERSION}_${ARCHITECTURE}.deb.

[asfimport]

Krisztian Szucs / @kszucs:
Now each task has its own github release tag to avoid conflicts.

Resolved by ARROW-2836: [Packaging] Expand build matrices to multiple tasks
Github PR: #2259

Closing.