ARROW-9701: [CI][Java] Add a job for s390x Java on TravisCI #7938

kiszk · 2020-08-12T07:12:46Z

This PR adds a new test job for big-endian Java environment to TravisCI. It still keeps as

   allow_failures:
     - arch: s390x

github-actions · 2020-08-12T07:18:40Z

https://issues.apache.org/jira/browse/ARROW-9701

kou · 2020-08-12T21:42:26Z

.travis.yml

+        ARCH: s390x
+        ARROW_CI_MODULES: "JAVA"
+        DOCKER_IMAGE_ID: debian-java
+        UBUNTU: "20.04"


This is needless because we don't use Ubuntu based docker image.

kou · 2020-08-12T21:42:51Z

.travis.yml

+          ulimit -c unlimited || :
+        - |
+          archery docker run \
+            -e ARCH=${ARCH} \


This is needless because ARCH is already passed as environment variable.

kou · 2020-08-12T21:47:16Z

.travis.yml

@@ -81,24 +137,6 @@ before_install:
 install:
  - pip3 install -e dev/archery[docker]

-script:


How about using a common script for all jobs like the following?

diff --git a/.travis.yml b/.travis.yml index fa5d84c82..551d0a029 100644 --- a/.travis.yml +++ b/.travis.yml @@ -47,15 +47,23 @@ jobs: env: ARCH: s390x ARROW_CI_MODULES: "CPP" - ARROW_FLIGHT: "ON" - ARROW_PARQUET: "OFF" DOCKER_IMAGE_ID: ubuntu-cpp - PARQUET_BUILD_EXAMPLES: "OFF" - PARQUET_BUILD_EXECUTABLES: "OFF" - Protobuf_SOURCE: "BUNDLED" + DOCKER_RUN_ARGS: >- + -e ARROW_FLIGHT=ON + -e ARROW_PARQUET=OFF + -e PARQUET_BUILD_EXAMPLES=OFF + -e PARQUET_BUILD_EXECUTABLES=OFF + -e Protobuf_SOURCE=BUNDLED + -e cares_SOURCE=BUNDLED + -e gRPC_SOURCE=BUNDLED UBUNTU: "20.04" - cares_SOURCE: "BUNDLED" - gRPC_SOURCE: "BUNDLED" + - name: "Java on s390x" + os: linux + arch: s390x + env: + ARCH: s390x + ARROW_CI_MODULES: "JAVA" + DOCKER_IMAGE_ID: debian-java allow_failures: - arch: s390x @@ -89,13 +97,7 @@ script: ulimit -c unlimited || : - | archery docker run \ - -e ARROW_FLIGHT=${ARROW_FLIGHT:-OFF} \ - -e ARROW_PARQUET=${ARROW_PARQUET:-ON} \ - -e PARQUET_BUILD_EXAMPLES=${PARQUET_BUILD_EXAMPLES:-ON} \ - -e PARQUET_BUILD_EXECUTABLES=${PARQUET_BUILD_EXECUTABLES:-ON} \ - -e Protobuf_SOURCE=${Protobuf_SOURCE:-} \ - -e cares_SOURCE=${cares_SOURCE:-} \ - -e gRPC_SOURCE=${gRPC_SOURCE:-} \ + ${DOCKER_RUN_ARGS} \ --volume ${PWD}/build:/build \ ${DOCKER_IMAGE_ID}

kou · 2020-08-12T21:48:04Z

ci/scripts/java_build.sh

+  bintray_base_url="https://dl.bintray.com/apache/arrow"
+
+  bintray_dir="flatc-binary"
+  grp="com.github.icexelloss"


Suggested change

grp="com.github.icexelloss"

group="com.github.icexelloss"

kou · 2020-08-12T21:50:43Z

ci/scripts/java_build.sh

+  grp="com.google.protobuf"
+  artifact="protoc"
+  ver="3.7.1"
+  cls="linux-s390_64"


Suggested change

cls="linux-s390_64"

classifier="linux-s390_64"

kou · 2020-08-12T21:51:28Z

ci/scripts/java_build.sh

+  extension="exe"
+  target=${artifact}-${ver}.${extension}
+  ${wget} ${bintray_base_url}/${bintray_dir}/${ver}/${target}
+  ${mvn_install} -DgroupId=${grp} -DartifactId=${artifact} -Dversion=${ver} -Dpackaging=${extension} -Dfile=`pwd`/${target}


Generally, $(...) is preferred than `...`:

Suggested change

${mvn_install} -DgroupId=${grp} -DartifactId=${artifact} -Dversion=${ver} -Dpackaging=${extension} -Dfile=`pwd`/${target}

${mvn_install} -DgroupId=${grp} -DartifactId=${artifact} -Dversion=${ver} -Dpackaging=${extension} -Dfile=$(pwd)/${target}

kiszk · 2020-08-28T00:33:18Z

@kou Thank you. Now, a TravisCI job is finished correctly with many failures. Could you review this again?

kou · 2020-08-28T22:06:11Z

ci/scripts/java_build.sh

+  ${mvn_install} -DgroupId=${group} -DartifactId=${artifact} -Dversion=${ver} -Dclassifier=${classifier} -Dpackaging=${extension} -Dfile=$(pwd)/${target}
+
+  # Use `2 * ncores` threads
+  mvn_options="-T 2C -Dmaven.compiler.verbose=true"


Can we remove -Dmaven.compiler.verbose=true and remove the else clause?

kou · 2020-08-28T22:06:39Z

java/pom.xml

@@ -193,7 +193,7 @@
          <target>1.8</target>
          <maxmem>2048m</maxmem>
          <useIncrementalCompilation>false</useIncrementalCompilation>
-          <fork>true</fork>
+          <fork>false</fork>


Do we still need this?

No, use the original setting. Done

kou

+1

We hold off merging this until we come to consensus on https://lists.apache.org/thread.html/r3d4ab3ccc8db7a7a68baa727bf9b6911930b61609c0faaed14e2b773%40%3Cdev.arrow.apache.org%3E .

See also: #8047 (comment)

kiszk · 2020-10-03T04:40:55Z

@emkornfield @kou Is it ok to merge this PR now? Or, do we still hold off for a while?

emkornfield · 2020-10-05T03:49:44Z

Yes, I think so. I'd still like to update the contributor guidelines to make it clear what the level of support we are aiming for in each language (I hope to get to it tonight but if I don't it might take me a few days), but I think CI should be harmless. @kou it looks like you are happy with this PR?

kou · 2020-10-05T04:23:53Z

Yes.
But I want to confirm that this still works with the latest master before we merge.
@kiszk Could you rebase this on the latest master?

kiszk · 2020-10-05T05:16:04Z

@emkornfield @kou Thank you. I want to merge only this PR for CI. We should still hold off other PRs.

I rebased with master branch and confirmed it works well.

kou · 2020-10-05T05:26:39Z

Thanks.
I'll merge this.

This is a follow-up of #7938. #7938 forgot downloading a required dll `libprotobuf.so.18` at the build time. Closes #8368 from kiszk/ARROW-10200 Authored-by: Kazuaki Ishizaki <ishizaki@jp.ibm.com> Signed-off-by: Sutou Kouhei <kou@clear-code.com>

This PR adds a new test job for big-endian Java environment to TravisCI. It still keeps as ``` allow_failures: - arch: s390x ``` Closes apache#7938 from kiszk/ARROW-9701 Lead-authored-by: Kazuaki Ishizaki <ishizaki@jp.ibm.com> Co-authored-by: Sutou Kouhei <kou@clear-code.com> Signed-off-by: Sutou Kouhei <kou@clear-code.com>

This is a follow-up of apache#7938. apache#7938 forgot downloading a required dll `libprotobuf.so.18` at the build time. Closes apache#8368 from kiszk/ARROW-10200 Authored-by: Kazuaki Ishizaki <ishizaki@jp.ibm.com> Signed-off-by: Sutou Kouhei <kou@clear-code.com>

This PR adds a new test job for big-endian Java environment to TravisCI. It still keeps as ``` allow_failures: - arch: s390x ``` Closes apache#7938 from kiszk/ARROW-9701 Lead-authored-by: Kazuaki Ishizaki <ishizaki@jp.ibm.com> Co-authored-by: Sutou Kouhei <kou@clear-code.com> Signed-off-by: Sutou Kouhei <kou@clear-code.com>

This is a follow-up of apache#7938. apache#7938 forgot downloading a required dll `libprotobuf.so.18` at the build time. Closes apache#8368 from kiszk/ARROW-10200 Authored-by: Kazuaki Ishizaki <ishizaki@jp.ibm.com> Signed-off-by: Sutou Kouhei <kou@clear-code.com>

kou reviewed Aug 12, 2020

View reviewed changes

kiszk force-pushed the ARROW-9701 branch from a7ad1ec to 10aff55 Compare August 14, 2020 05:05

kiszk marked this pull request as draft August 15, 2020 12:49

kou mentioned this pull request Aug 19, 2020

ARROW-9803: [Go] Add initial support for s390x #8011

Closed

kiszk force-pushed the ARROW-9701 branch 2 times, most recently from ffbb910 to 6fd6dac Compare August 27, 2020 02:43

kiszk marked this pull request as ready for review August 28, 2020 00:31

kou force-pushed the ARROW-9701 branch from b4fe69b to aaca70b Compare August 28, 2020 21:36

kou reviewed Aug 28, 2020

View reviewed changes

kou approved these changes Aug 30, 2020

View reviewed changes

kiszk and others added 12 commits October 5, 2020 04:34

add a job for Java

5d6be25

download files from bintray and install them

825b210

address review comments

190c4fb

fix script error

c9eaed0

fix travis errors

56d7404

reduce mvn threads on s390x

d9b8986

show javac message

679dafe

Add missing quote

f91e10f

Fix merge miss

9db832d

Disable glog on arm64v8 again

c2ee041

Add JDK=11 again

ce12723

revert changes and use the original code

c67d6ee

kiszk force-pushed the ARROW-9701 branch from 844584f to c67d6ee Compare October 5, 2020 04:35

kou closed this in 8c75941 Oct 5, 2020

This was referenced Oct 6, 2020

ARROW-9704: [Java] TestEndianness.testLittleEndian supports little- and big-endian platforms #7942

Closed

ARROW-10200: [CI][Java] Fix a job failure for s390x Java on TravisCI #8368

Closed

asfimport mentioned this pull request Oct 5, 2020

[Java][CI] Add a test job on s390x #25757

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ARROW-9701: [CI][Java] Add a job for s390x Java on TravisCI #7938

ARROW-9701: [CI][Java] Add a job for s390x Java on TravisCI #7938

kiszk commented Aug 12, 2020

github-actions bot commented Aug 12, 2020

kou Aug 12, 2020

kou Aug 12, 2020

kou Aug 12, 2020

kou Aug 12, 2020

kou Aug 12, 2020

kou Aug 12, 2020

kiszk commented Aug 28, 2020

kou Aug 28, 2020

kiszk Aug 29, 2020

kou Aug 28, 2020

kiszk Aug 29, 2020

kou left a comment

kiszk commented Oct 3, 2020

emkornfield commented Oct 5, 2020

kou commented Oct 5, 2020

kiszk commented Oct 5, 2020

kou commented Oct 5, 2020

	${mvn_install} -DgroupId=${grp} -DartifactId=${artifact} -Dversion=${ver} -Dpackaging=${extension} -Dfile=`pwd`/${target}
	${mvn_install} -DgroupId=${grp} -DartifactId=${artifact} -Dversion=${ver} -Dpackaging=${extension} -Dfile=$(pwd)/${target}

ARROW-9701: [CI][Java] Add a job for s390x Java on TravisCI #7938

ARROW-9701: [CI][Java] Add a job for s390x Java on TravisCI #7938

Conversation

kiszk commented Aug 12, 2020

github-actions bot commented Aug 12, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kiszk commented Aug 28, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kou left a comment

Choose a reason for hiding this comment

kiszk commented Oct 3, 2020

emkornfield commented Oct 5, 2020

kou commented Oct 5, 2020

kiszk commented Oct 5, 2020

kou commented Oct 5, 2020