[Packaging] Fix linux package builds #19102

asfimport · 2018-06-15T08:45:41Z

Build configuration: https://github.com/kszucs/arrow/tree/0d9d89b7bff32823ab68e6ec1dc7ade52511f7ee/dev/tasks/linux-packages

Failing build: https://travis-ci.org/kszucs/crossbow/builds/391894564?utm_source=github_status&utm_medium=notification

Looks like it’s waiting for a user input? There might be some hardcoded version too, because the expected is 0.9.1 instead of 0.9.0.

ping @kou

Reporter: Krisztian Szucs / @kszucs
Assignee: Kouhei Sutou / @kou

_{Note: This issue was originally created as ARROW-2713. Please see the migration documentation for further details.}

asfimport · 2018-06-15T08:51:51Z

Kouhei Sutou / @kou:
For .deb:

We need an entry for the built version in debian/changelog. We can do this by running "rake version:update".
Can you try running "rake version:update" before "rake apt:build"?

asfimport · 2018-06-15T08:55:46Z

Krisztian Szucs / @kszucs:
Like this: 60f8b75#diff-66ceaeef0ef768c9dcd99e8da32ce261 ?

Build link: https://travis-ci.org/kszucs/crossbow/builds/392622254

asfimport · 2018-06-15T08:58:15Z

Kouhei Sutou / @kou:
For .rpm:

The error reason is:


/root/rpmbuild/BUILD/apache-arrow-0.10.0.20180613/c_glib/../cpp/build/release/libarrow.so.10: undefined reference to `typeinfo for google::protobuf::io::ZeroCopyOutputStream'

It may be caused by make -j3. We may have a parallel build problem in Apache Arrow C++.

asfimport · 2018-06-15T08:59:46Z

Kouhei Sutou / @kou:

Like this: 60f8b75#diff-66ceaeef0ef768c9dcd99e8da32ce261 ?

Yes!

asfimport · 2018-06-15T09:28:57Z

Krisztian Szucs / @kszucs:
Errored again https://travis-ci.org/kszucs/crossbow/builds/392622254

asfimport · 2018-06-15T09:43:01Z

Kouhei Sutou / @kou:
We've moved forward! The first "0.9.0 and 0.9.1 error" has gone.

The next error:


make[4]: Entering directory `/home/travis/build/kszucs/crossbow/arrow/c_glib_tmp/apache-arrow-0.10.0.20180614/c_glib/doc/reference'
cp: cannot stat ‘./arrow-glib.pdf’: No such file or directory
make[4]: [dist-hook] Error 1 (ignored)

Umm. Why is PDF document expected...
I need to look into it later.

asfimport · 2018-06-19T06:45:58Z

Kouhei Sutou / @kou:
I looked into CentOS 7 build failure:


/root/rpmbuild/BUILD/apache-arrow-0.10.0.20180613/c_glib/../cpp/build/release/libarrow.so.10: undefined reference to `typeinfo for google::protobuf::io::ZeroCopyOutputStream'

It's not related to RPM build. We can reproduce it with building with -DARROW_ORC=ON on CentOS 7. I don't know why. It isn't occurred on Debian GNU/Linux sid.

It should be worked in separated JIRA ticket.

asfimport · 2018-06-19T07:14:55Z

Kouhei Sutou / @kou:
For Debian GNU/Linux stretch:

FlatBuffers has build error:


cat /build/apache-arrow-0.10.0.20180617/cpp_build/flatbuffers_ep-prefix/src/flatbuffers_ep-stamp/flatbuffers_ep-build-err.log 
/build/apache-arrow-0.10.0.20180617/cpp_build/flatbuffers_ep-prefix/src/flatbuffers_ep/src/flatc.cpp:250:38: error: macro "__DATE__" might prevent reproducible builds [-Werror=date-time]
         printf("flatc version %s\n", FLATC_VERSION);
                                      ^~~~~~~~~~~~~
/build/apache-arrow-0.10.0.20180617/cpp_build/flatbuffers_ep-prefix/src/flatbuffers_ep/src/flatc.cpp:21:33: error: macro "__TIME__" might prevent reproducible builds [-Werror=date-time]
 #define FLATC_VERSION "1.9.0 (" __DATE__ " " __TIME__ ")"
                                 ^
/build/apache-arrow-0.10.0.20180617/cpp_build/flatbuffers_ep-prefix/src/flatbuffers_ep/src/flatc.cpp:250:38: note: in expansion of macro 'FLATC_VERSION'
         printf("flatc version %s\n", FLATC_VERSION);
                                      ^~~~~~~~~~~~~
cc1plus: all warnings being treated as errors

Debian GNU/Linux stretch adds -Wdate-time flag by default and FlatBuffers uses -Werror flag. It causes the error.

See also "timeless" value description of "reproducible" in http://man7.org/linux/man-pages/man1/dpkg-buildflags.1.html .

We can disable the feature by:

diff --git a/dev/tasks/linux-packages/debian/rules b/dev/tasks/linux-packages/debian/rules
index 5a3c0a31..5af70521 100755
--- a/dev/tasks/linux-packages/debian/rules
+++ b/dev/tasks/linux-packages/debian/rules
@@ -6,6 +6,8 @@
# This has to be exported to make some magic below work.
 export DH_OPTIONS
 
+export DEB_BUILD_MAINT_OPTIONS=reproducible=-timeless
+
 BUILD_TYPE=release
 
 %:

asfimport · 2018-06-19T07:54:34Z

Kouhei Sutou / @kou:
#2146 fixes the error on CentOS 7.

asfimport · 2018-06-20T00:16:24Z

Kouhei Sutou / @kou:
The pull request has been merged.
Can you rebase on master?

asfimport · 2018-06-20T16:14:38Z

Krisztian Szucs / @kszucs:
Thanks @kou!

I've rebased, there is another issue now: https://api.travis-ci.org/v3/job/394593664/log.txt
Build's url: https://travis-ci.org/kszucs/crossbow/builds/394593662

asfimport · 2018-06-21T02:50:18Z

Kouhei Sutou / @kou:
For deb:

Can you apply this patch?


diff --git a/dev/tasks/linux-packages/debian.ubuntu-trusty/rules b/dev/tasks/linux-packages/debian.ubuntu-trusty/rules
index 783b7a75..b1085d13 100755
--- a/dev/tasks/linux-packages/debian.ubuntu-trusty/rules
+++ b/dev/tasks/linux-packages/debian.ubuntu-trusty/rules
@@ -6,6 +6,8 @@
# This has to be exported to make some magic below work.
 export DH_OPTIONS
 
+export DEB_BUILD_MAINT_OPTIONS=reproducible=-timeless
+
 BUILD_TYPE=release
 
 %:
diff --git a/dev/tasks/linux-packages/debian/rules b/dev/tasks/linux-packages/debian/rules
index 5a3c0a31..5af70521 100755
--- a/dev/tasks/linux-packages/debian/rules
+++ b/dev/tasks/linux-packages/debian/rules
@@ -6,6 +6,8 @@
# This has to be exported to make some magic below work.
 export DH_OPTIONS
 
+export DEB_BUILD_MAINT_OPTIONS=reproducible=-timeless
+
 BUILD_TYPE=release
 
 %:

asfimport · 2018-06-21T05:08:26Z

Kouhei Sutou / @kou:
For RPM:

#2150 will fix this problem.

asfimport · 2018-06-22T00:23:07Z

Kouhei Sutou / @kou:
The pull request have been merged.
Can you rebase on master?

asfimport · 2018-06-26T06:57:51Z

Krisztian Szucs / @kszucs:
@kou Rebased. The current error is:

debuild: fatal error at line 1376:

dpkg-buildpackage -rfakeroot -D -us -uc failed

debuild: fatal error at line 1116:

dpkg-buildpackage -rfakeroot -us -uc failed

rake aborted!

Builds are here https://travis-ci.org/kszucs/crossbow/builds/396729506

asfimport · 2018-06-26T07:14:04Z

Kouhei Sutou / @kou:
Thanks. Can you apply the following patch?


diff --git a/dev/tasks/linux-packages/debian.ubuntu-trusty/rules b/dev/tasks/linux-packages/debian.ubuntu-trusty/rules
index 783b7a75..b1085d13 100755
--- a/dev/tasks/linux-packages/debian.ubuntu-trusty/rules
+++ b/dev/tasks/linux-packages/debian.ubuntu-trusty/rules
@@ -6,6 +6,8 @@
# This has to be exported to make some magic below work.
 export DH_OPTIONS
 
+export DEB_BUILD_MAINT_OPTIONS=reproducible=-timeless
+
 BUILD_TYPE=release
 
 %:
diff --git a/dev/tasks/linux-packages/debian/rules b/dev/tasks/linux-packages/debian/rules
index 5a3c0a31..5af70521 100755
--- a/dev/tasks/linux-packages/debian/rules
+++ b/dev/tasks/linux-packages/debian/rules
@@ -6,6 +6,8 @@
# This has to be exported to make some magic below work.
 export DH_OPTIONS
 
+export DEB_BUILD_MAINT_OPTIONS=reproducible=-timeless
+
 BUILD_TYPE=release
 
 %:

asfimport · 2018-06-26T08:17:19Z

Kouhei Sutou / @kou:
Can you also apply the following patch?


diff --git a/dev/tasks/linux-packages/yum/arrow.spec.in b/dev/tasks/linux-packages/yum/arrow.spec.in
index 65c357ef..72bc4b16 100644
--- a/dev/tasks/linux-packages/yum/arrow.spec.in
+++ b/dev/tasks/linux-packages/yum/arrow.spec.in
@@ -132,6 +132,7 @@ Libraries and header files for Apache Arrow C++.
%{_libdir}/libarrow.so
%{_libdir}/pkgconfig/arrow.pc
%{_libdir}/pkgconfig/arrow-compute.pc
+%{_libdir}/pkgconfig/arrow-orc.pc

%if %{use_python}
%package python-libs

asfimport · 2018-06-26T08:19:13Z

Krisztian Szucs / @kszucs:
Sure, thanks @kou!

For the previous patch the builds are running here https://travis-ci.org/kszucs/crossbow/builds/396755589

asfimport · 2018-06-26T08:35:25Z

Kouhei Sutou / @kou:
Thanks! Can you also apply the following patch?


diff --git a/dev/tasks/linux-packages/debian/libarrow-dev.install b/dev/tasks/linux-packages/debian/libarrow-dev.install
index 62de6ce5..18ea2e3a 100644
--- a/dev/tasks/linux-packages/debian/libarrow-dev.install
+++ b/dev/tasks/linux-packages/debian/libarrow-dev.install
@@ -3,3 +3,4 @@ usr/lib/*/libarrow.a
 usr/lib/*/libarrow.so
 usr/lib/*/pkgconfig/arrow.pc
 usr/lib/*/pkgconfig/arrow-compute.pc
+usr/lib/*/pkgconfig/arrow-orc.pc
diff --git a/dev/tasks/linux-packages/debian.ubuntu-trusty/libarrow-dev.install b/dev/tasks/linux-packages/debian.ubuntu-trusty/libarrow-dev.install
index 62de6ce5..18ea2e3a 100644
--- a/dev/tasks/linux-packages/debian.ubuntu-trusty/libarrow-dev.install
+++ b/dev/tasks/linux-packages/debian.ubuntu-trusty/libarrow-dev.install
@@ -3,3 +3,4 @@ usr/lib/*/libarrow.a
 usr/lib/*/libarrow.so
 usr/lib/*/pkgconfig/arrow.pc
 usr/lib/*/pkgconfig/arrow-compute.pc
+usr/lib/*/pkgconfig/arrow-orc.pc

asfimport · 2018-06-26T08:36:05Z

Kouhei Sutou / @kou:
Build failure will be fixed by #2169 .

asfimport · 2018-06-26T08:38:11Z

Krisztian Szucs / @kszucs:
Yes, I'm waiting for that to fix conda packages too :)

asfimport · 2018-06-27T01:12:12Z

Kouhei Sutou / @kou:
The pull request has been merged!

Can you rebase and apply the following patch?


diff --git a/dev/tasks/linux-packages/yum/arrow.spec.in b/dev/tasks/linux-packages/yum/arrow.spec.in
index 65c357ef..72bc4b16 100644
--- a/dev/tasks/linux-packages/yum/arrow.spec.in
+++ b/dev/tasks/linux-packages/yum/arrow.spec.in
@@ -132,6 +132,7 @@ Libraries and header files for Apache Arrow C++.
 %{_libdir}/libarrow.so
 %{_libdir}/pkgconfig/arrow.pc
 %{_libdir}/pkgconfig/arrow-compute.pc
+%{_libdir}/pkgconfig/arrow-orc.pc
 
 %if %{use_python}
 %package python-libs

asfimport · 2018-06-27T08:26:01Z

Krisztian Szucs / @kszucs:
@kou One of the build passes now: https://travis-ci.org/kszucs/crossbow/builds/396941927

asfimport · 2018-06-28T01:16:56Z

Kouhei Sutou / @kou:
Great!

Can you apply the following patch for https://travis-ci.org/kszucs/crossbow/jobs/396941930 ?


diff --git a/dev/tasks/linux-packages/yum/arrow.spec.in b/dev/tasks/linux-packages/yum/arrow.spec.in
index 65c357ef..72bc4b16 100644
--- a/dev/tasks/linux-packages/yum/arrow.spec.in
+++ b/dev/tasks/linux-packages/yum/arrow.spec.in
@@ -132,6 +132,7 @@ Libraries and header files for Apache Arrow C++.
 %{_libdir}/libarrow.so
 %{_libdir}/pkgconfig/arrow.pc
 %{_libdir}/pkgconfig/arrow-compute.pc
+%{_libdir}/pkgconfig/arrow-orc.pc
 
 %if %{use_python}
 %package python-libs

asfimport · 2018-06-28T10:07:42Z

Krisztian Szucs / @kszucs:
@kou One has left: https://travis-ci.org/kszucs/crossbow/builds/397700422 :)

asfimport · 2018-06-29T08:16:32Z

Kouhei Sutou / @kou:
Great!

It seems that the build is timed out.

Can you apply the following patch?


diff --git a/dev/tasks/linux-packages/travis.linux.yml b/dev/tasks/linux-packages/travis.linux.yml
index 8cc63772..3e2bcc79 100644
--- a/dev/tasks/linux-packages/travis.linux.yml
+++ b/dev/tasks/linux-packages/travis.linux.yml
@@ -30,9 +30,9 @@ env:
 matrix:
   include:
     - script:
-        - (cd arrow/dev/tasks/linux-packages && travis_wait 40 && rake version:update && rake apt:build APT_TARGETS=debian-stretch,ubuntu-trusty,ubuntu-xenial PARALLEL=yes DEBUG=no)
+        - (cd arrow/dev/tasks/linux-packages && rake version:update && travis_wait 40 rake apt:build APT_TARGETS=debian-stretch,ubuntu-trusty PARALLEL=yes DEBUG=no)
     - script:
-        - (cd arrow/dev/tasks/linux-packages && travis_wait 40 && rake version:update && rake apt:build APT_TARGETS=ubuntu-artful PARALLEL=yes DEBUG=no)
+        - (cd arrow/dev/tasks/linux-packages && rake version:update && travis_wait 40 rake apt:build APT_TARGETS=ubuntu-xenial,ubuntu-artful PARALLEL=yes DEBUG=no)
     - script:
         - (cd arrow/dev/tasks/linux-packages && rake version:update && rake yum:build PARALLEL=yes DEBUG=no)

asfimport · 2018-06-29T15:15:15Z

Krisztian Szucs / @kszucs:
Now all three are green https://travis-ci.org/kszucs/crossbow/builds/398207226
Thanks @kou!

I'll setup the deployments then.

asfimport · 2018-06-29T23:40:15Z

Kouhei Sutou / @kou:
Yay!

I want us to consider how to unify arrow-dist/cpp-linux/ and arrow/dev/tasks/linux-packages/ as the next step.
Should we move arrow-dist/cpp-linux/ to arrow/dev/tasks/linux-packages/?

asfimport · 2018-06-30T07:18:05Z

Krisztian Szucs / @kszucs:
Actually I've moved it, but there might be a couple of changes since then.

Yes, I think We should maintain arrow/dev/tasks onwards.

asfimport · 2018-06-30T07:23:43Z

Krisztian Szucs / @kszucs:
@kou What are exactly the expected artifacts from linux builds: https://github.com/kszucs/crossbow/releases/tag/build-154 ?

(ubuntu-xenial,ubuntu-artful have failed on this build somewhy)

asfimport · 2018-07-02T03:35:50Z

Kouhei Sutou / @kou:
OK. We'll use arrow/dev/tasks/linux-packages/ for 0.10.0 or later. arrow-dist refers them.

I've commetted about artifacts at #2162 (comment) . I think that tar.gz is better.

Failure on https://travis-ci.org/kszucs/crossbow/builds/398516100 was caused by filename conflict.

asfimport · 2018-07-18T19:09:27Z

Krisztian Szucs / @kszucs:
I'm closing this one, because all of the linux builds are working.
Thanks @kou !

asfimport closed this as completed Jul 18, 2018

asfimport assigned kou Jan 10, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Packaging] Fix linux package builds #19102

[Packaging] Fix linux package builds #19102

asfimport commented Jun 15, 2018

asfimport commented Jun 15, 2018

asfimport commented Jun 15, 2018

asfimport commented Jun 15, 2018

asfimport commented Jun 15, 2018

asfimport commented Jun 15, 2018

asfimport commented Jun 15, 2018

asfimport commented Jun 19, 2018

asfimport commented Jun 19, 2018

asfimport commented Jun 19, 2018

asfimport commented Jun 20, 2018

asfimport commented Jun 20, 2018

asfimport commented Jun 21, 2018

asfimport commented Jun 21, 2018

asfimport commented Jun 22, 2018

asfimport commented Jun 26, 2018

asfimport commented Jun 26, 2018

asfimport commented Jun 26, 2018

asfimport commented Jun 26, 2018

asfimport commented Jun 26, 2018

asfimport commented Jun 26, 2018

asfimport commented Jun 26, 2018

asfimport commented Jun 27, 2018

asfimport commented Jun 27, 2018

asfimport commented Jun 28, 2018

asfimport commented Jun 28, 2018

asfimport commented Jun 29, 2018

asfimport commented Jun 29, 2018

asfimport commented Jun 29, 2018

asfimport commented Jun 30, 2018

asfimport commented Jun 30, 2018

asfimport commented Jul 2, 2018

asfimport commented Jul 18, 2018

[Packaging] Fix linux package builds #19102

[Packaging] Fix linux package builds #19102

Comments

asfimport commented Jun 15, 2018

asfimport commented Jun 15, 2018

asfimport commented Jun 15, 2018

asfimport commented Jun 15, 2018

asfimport commented Jun 15, 2018

asfimport commented Jun 15, 2018

asfimport commented Jun 15, 2018

asfimport commented Jun 19, 2018

asfimport commented Jun 19, 2018

asfimport commented Jun 19, 2018

asfimport commented Jun 20, 2018

asfimport commented Jun 20, 2018

asfimport commented Jun 21, 2018

asfimport commented Jun 21, 2018

asfimport commented Jun 22, 2018

asfimport commented Jun 26, 2018

asfimport commented Jun 26, 2018

asfimport commented Jun 26, 2018

asfimport commented Jun 26, 2018

asfimport commented Jun 26, 2018

asfimport commented Jun 26, 2018

asfimport commented Jun 26, 2018

asfimport commented Jun 27, 2018

asfimport commented Jun 27, 2018

asfimport commented Jun 28, 2018

asfimport commented Jun 28, 2018

asfimport commented Jun 29, 2018

asfimport commented Jun 29, 2018

asfimport commented Jun 29, 2018

asfimport commented Jun 30, 2018

asfimport commented Jun 30, 2018

asfimport commented Jul 2, 2018

asfimport commented Jul 18, 2018