KAFKA-9857:Failed to build image ducker-ak-openjdk-8 on arm #8489

jiameixie · 2020-04-15T02:47:05Z

The default OpenJDK base image is openjdk:8. When building image on arm, no
matching manifest for linux/arm64/v8 in the manifest list entries error will
occur. For arm, the default OpenJDK should be set to arm64v8/openjdk:8

Change-Id: Ib450a36b3977a167743c24476ec1810f4830b66b
Signed-off-by: Jiamei Xie jiamei.xie@arm.com

More detailed description of your change,
if necessary. The PR title and PR message become
the squashed commit message, so use a separate
comment to ping reviewers.

Summary of testing strategy (including rationale)
for the feature or bug fix. Unit and/or integration
tests are expected for any behaviour change and
system tests should be considered for larger changes.

Committer Checklist (excluded from commit message)

Verify design and implementation
Verify test coverage and CI build status
Verify documentation (including upgrade notes)

The default OpenJDK base image is openjdk:8. When building image on arm, no matching manifest for linux/arm64/v8 in the manifest list entries error will occur. For arm, the default OpenJDK should be set to arm64v8/openjdk:8 Change-Id: Ib450a36b3977a167743c24476ec1810f4830b66b Signed-off-by: Jiamei Xie <jiamei.xie@arm.com>

jiameixie · 2020-04-23T02:00:23Z

@guozhangwang @ijuma @junrao PTAL, thanks

junrao · 2020-04-23T15:42:44Z

@cmccabe : Does this look good to you? Thanks.

cmccabe · 2020-04-23T18:29:30Z

This looks reasonable. What testing have we done so far?

jiameixie · 2020-05-06T01:18:47Z

@cmccabe I run it on arm, it works fine.

jiameixie · 2020-05-06T03:24:11Z

@cmccabe I got some failures by running "bash tests/docker/run_tests.sh" on ARM, but most of it can be run successfully after some modification, except some tests related to version compatibility. Following is the summary:

Fail due to timeout. Increasing timeout makes it run successfully.
kafkatest.tests.streams.streams_upgrade_test.StreamsUpgradeTest.test_metadata_upgrade.to_version=2.1.1.from_version=0.10.2.2
kafkatest.tests.core.replica_scale_test.ReplicaScaleTest.test_clean_bounce.partition_count=34.topic_count=500.replication_factor=3
kafkatest.tests.core.group_mode_transactions_test.GroupModeTransactionsTest.test_transactions.failure_mode=clean_bounce.bounce_target=clients
kafkatest.tests.streams.streams_broker_bounce_test.StreamsBrokerBounceTest.test_broker_type_bounce.num_threads=1.sleep_time_secs=120.failure_mode=clean_bounce.broker_type=controller
kafkatest.tests.streams.streams_broker_bounce_test.StreamsBrokerBounceTest.test_broker_type_bounce.num_threads=1.sleep_time_secs=120.failure_mode=clean_bounce.broker_type=leader
kafkatest.tests.streams.streams_broker_bounce_test.StreamsBrokerBounceTest.test_broker_type_bounce.num_threads=1.sleep_time_secs=120.failure_mode=clean_shutdown.broker_type=controller
Version compatibility
kafkatest.tests.core.downgrade_test.TestDowngrade.test_upgrade_and_downgrade.version=2.0.1.security_protocol=SASL_SSL.compression_types=.snappy
kafkatest.tests.core.upgrade_test.TestUpgrade.test_upgrade.from_kafka_version=0.8.2.2.to_message_format_version=None.compression_types=.snappy
kafkatest.tests.core.compatibility_test_new_broker_test.ClientCompatibilityTestNewBroker.test_compatibility.consumer_version=0.9.0.1.producer_version=0.9.0.1.compression_types=.snappy.timestamp_type=LogAppendTime
kafkatest.tests.core.compatibility_test_new_broker_test.ClientCompatibilityTestNewBroker.test_compatibility.consumer_version=0.9.0.1.producer_version=dev.compression_types=.snappy.timestamp_type=CreateTime
kafkatest.tests.core.compatibility_test_new_broker_test.ClientCompatibilityTestNewBroker.test_compatibility.consumer_version=2.0.1.producer_version=2.0.1.compression_types=.snappy.timestamp_type=CreateTime
kafkatest.tests.core.compatibility_test_new_broker_test.ClientCompatibilityTestNewBroker.test_compatibility.consumer_version=dev.producer_version=0.9.0.1.compression_types=.snappy.timestamp_type=None
kafkatest.tests.core.compatibility_test_new_broker_test.ClientCompatibilityTestNewBroker.test_compatibility.compression_types=.none.timestamp_type=None.producer_version=dev.new_consumer=False.consumer_version=0.9.0.1
kafkatest.tests.core.upgrade_test.TestUpgrade.test_upgrade.from_kafka_version=0.9.0.1.to_message_format_version=None.compression_types=.snappy
kafkatest.tests.core.upgrade_test.TestUpgrade.test_upgrade.from_kafka_version=2.0.1.to_message_format_version=None.compression_types=.snappy
Wrong parameter for StreamsSmokeTestJobRunnerService constructor. It has been fixed upstream.
kafkatest.tests.streams.streams_broker_bounce_test.StreamsBrokerBounceTest.test_all_brokers_bounce.failure_mode=clean_bounce.num_failures=3
kafkatest.tests.streams.streams_broker_bounce_test.StreamsBrokerBounceTest.test_all_brokers_bounce.failure_mode=hard_bounce.num_failures=3
network_degrade doesn't ready in time. I have a PR about it.
kafkatest.tests.core.network_degrade_test.NetworkDegradeTest.test_rate.rate_limit_kbit=1000000.device_name=eth0.task_name=rate-1000-latency-50.latency_ms=50

jiameixie · 2020-05-12T01:48:12Z

@cmccabe @junrao There were Jenkins errors. Does my PR need any more modification? Thanks.

OneCricketeer · 2020-05-14T21:53:27Z

Since Java 8 is EOL, why not upgrade to openjdk 11?

The default OpenJDK base image is openjdk:8. When building image on arm, no matching manifest for linux/arm64/v8 in the manifest list entries error will occur. For arm, the default OpenJDK should be set to arm64v8/openjdk:8. Java 8 is end of life, so upgrade to openjdk 11. Change-Id: Ib450a36b3977a167743c24476ec1810f4830b66b Signed-off-by: Jiamei Xie <jiamei.xie@arm.com>

jiameixie · 2020-05-15T03:35:39Z

@OneCricketeer Yes, you're right. I recommitted the pr and upgraded openjdk 8 to openjdk 11.

OneCricketeer · 2020-05-15T03:45:12Z

tests/docker/ducker-ak

@@ -42,7 +42,14 @@ docker_run_memory_limit="2000m"
 default_num_nodes=14

 # The default OpenJDK base image.
-default_jdk="openjdk:8"


So, you could've kept this

OneCricketeer · 2020-05-15T03:45:52Z

tests/docker/ducker-ak

-default_jdk="openjdk:8"
+case "$(uname -m)" in
+  aarch64)
+    default_jdk="arm64v8/openjdk:11"


Then done default_jdk=arm64v8/${default_jdk}

OneCricketeer · 2020-05-15T03:46:19Z

tests/docker/ducker-ak

+    default_jdk="arm64v8/openjdk:11"
+    ;;
+  *)
+    default_jdk="openjdk:11"


Then the wild card wouldn't be necessary

The default OpenJDK base image is openjdk:8. When building image on arm, no matching manifest for linux/arm64/v8 in the manifest list entries error will occur. For arm, the default OpenJDK should be set to arm64v8/openjdk:8. Java 8 is end of life, so upgrade to openjdk 11. Openjdk:11 is a multi-arch image. Change-Id: Ib450a36b3977a167743c24476ec1810f4830b66b Signed-off-by: Jiamei Xie <jiamei.xie@arm.com>

jiameixie · 2020-05-15T05:00:20Z

@OneCricketeer I tested just now. Openjdk:11 is a multi-arch. X86 and arm use the same image name.

jiameixie · 2020-05-15T05:11:10Z

@OneCricketeer Openjdk:11 is multi-arch, and openjdk:8 is not multi-arch. It's ok to run command "docker run -it openjdk:11 bash" on both arm and x86, while running command "docker run -it openjdk:8 bash" will bring an error.

jiameixie · 2020-05-15T05:15:13Z

@OneCricketeer So if update to OpenJDK 11, there is no need to set default_jdk according to machine architecture. Thanks for your reminding

jiameixie · 2020-05-15T05:21:19Z

@cmccabe I updated it to openjdk:11. Is that ok?

jiameixie · 2020-06-01T02:59:23Z

@cmccabe What tests should be done? Thanks

JunHe77 · 2020-09-18T08:04:34Z

@junrao @cmccabe Ping... is there any way we can help you merge this? Thanks.

chia7712 · 2020-10-20T03:21:29Z

there is an existent bug after updating base image from openjdk:8 to openjdk:11 (see #9324). feel free to merge the fix to your PR :)

…openjdk:11. Merge remote-tracking branch 'chia/MINOR-9324' into Failed-to-build-image-ducker-ak-openjdk-8-on-arm

…n-arm

jiameixie · 2020-10-21T04:17:10Z

@chia7712 Thanks. I have merged it.

lizthegrey · 2020-10-27T17:59:34Z

Honeycomb doesn't directly have an interest in this as we use Confluent's packages, but we do run Confluent's Kafka distro on ARM64 (successfully).

JunHe77 · 2020-10-28T05:35:53Z

Really appreciated for the feedback, @lizthegrey . Glad to know there are actual use cases to run and test Kafka on Arm64, and it seems that merging this will help users/developers from community perpective.

martin-g · 2020-12-29T08:46:45Z

I've tested this PR on my ARM64 machine and at TravisCI. At both environments it fails with:

...
Collecting pynacl>=1.0.1
  Downloading PyNaCl-1.4.0.tar.gz (3.4 MB)
  Installing build dependencies: started
  Installing build dependencies: finished with status 'error'
  ERROR: Command errored out with exit status 1:
   command: /usr/bin/python3 /usr/local/lib/python3.7/dist-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-wmhbzpt1/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- 'setuptools>=40.8.0' wheel 'cffi>=1.4.1; python_implementation != '"'"'PyPy'"'"''
       cwd: None
  Complete output (14 lines):
  Traceback (most recent call last):
    File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
      "__main__", mod_spec)
    File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
      exec(code, run_globals)
    File "/usr/local/lib/python3.7/dist-packages/pip/__main__.py", line 23, in <module>
      from pip._internal.cli.main import main as _main  # isort:skip # noqa
    File "/usr/local/lib/python3.7/dist-packages/pip/_internal/cli/main.py", line 5, in <module>
      import locale
    File "/usr/lib/python3.7/locale.py", line 16, in <module>
      import re
    File "/usr/lib/python3.7/re.py", line 143, in <module>
      class RegexFlag(enum.IntFlag):
  AttributeError: module 'enum' has no attribute 'IntFlag'
  ----------------------------------------
ERROR: Command errored out with exit status 1: /usr/bin/python3 /usr/local/lib/python3.7/dist-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-wmhbzpt1/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- 'setuptools>=40.8.0' wheel 'cffi>=1.4.1; python_implementation != '"'"'PyPy'"'"'' Check the logs for full command output.
WARNING: You are using pip version 20.2.2; however, version 20.3.3 is available.
You should consider upgrading via the '/usr/bin/python3 -m pip install --upgrade pip' command.
The command '/bin/sh -c pip3 install --upgrade cffi virtualenv pyasn1 boto3 pycrypto pywinrm ipaddress enum34 && pip3 install --upgrade ducktape==0.8.0' returned a non-zero code: 1
docker failed
ducker-ak up failed

martin-g · 2020-12-29T11:31:17Z

I've created #9794. It applies the changes from this PR and also fixes the issue in the comment above.
It also adds an extra job for Travis to run the build and the tests on ARM64 at AWS Graviton2 node.

OneCricketeer reviewed May 15, 2020

View reviewed changes

MINOR: Install "iproute2" explicitly in Dockerfile

8e0c208

jiameixie added 2 commits October 21, 2020 04:00

There is an existent bug after updating base image from openjdk:8 to …

1fa7b5b

…openjdk:11. Merge remote-tracking branch 'chia/MINOR-9324' into Failed-to-build-image-ducker-ak-openjdk-8-on-arm

Merge branch 'trunk' into Failed-to-build-image-ducker-ak-openjdk-8-o…

a0c7a50

…n-arm

martin-g mentioned this pull request Dec 29, 2020

Add a job to build on ARM64 at Amazon Graviton2 nodes #9794

Closed

jiameixie closed this Dec 23, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

KAFKA-9857:Failed to build image ducker-ak-openjdk-8 on arm #8489

KAFKA-9857:Failed to build image ducker-ak-openjdk-8 on arm #8489

jiameixie commented Apr 15, 2020

jiameixie commented Apr 23, 2020

junrao commented Apr 23, 2020

cmccabe commented Apr 23, 2020

jiameixie commented May 6, 2020

jiameixie commented May 6, 2020

jiameixie commented May 12, 2020

OneCricketeer commented May 14, 2020

jiameixie commented May 15, 2020

OneCricketeer May 15, 2020

OneCricketeer May 15, 2020

OneCricketeer May 15, 2020

OneCricketeer May 15, 2020

jiameixie commented May 15, 2020

jiameixie commented May 15, 2020

jiameixie commented May 15, 2020

jiameixie commented May 15, 2020

jiameixie commented Jun 1, 2020

JunHe77 commented Sep 18, 2020

chia7712 commented Oct 20, 2020

jiameixie commented Oct 21, 2020

lizthegrey commented Oct 27, 2020

JunHe77 commented Oct 28, 2020

martin-g commented Dec 29, 2020

martin-g commented Dec 29, 2020

KAFKA-9857:Failed to build image ducker-ak-openjdk-8 on arm #8489

KAFKA-9857:Failed to build image ducker-ak-openjdk-8 on arm #8489

Conversation

jiameixie commented Apr 15, 2020

Committer Checklist (excluded from commit message)

jiameixie commented Apr 23, 2020

junrao commented Apr 23, 2020

cmccabe commented Apr 23, 2020

jiameixie commented May 6, 2020

jiameixie commented May 6, 2020

jiameixie commented May 12, 2020

OneCricketeer commented May 14, 2020

jiameixie commented May 15, 2020

OneCricketeer May 15, 2020

Choose a reason for hiding this comment

OneCricketeer May 15, 2020

Choose a reason for hiding this comment

OneCricketeer May 15, 2020

Choose a reason for hiding this comment

OneCricketeer May 15, 2020

Choose a reason for hiding this comment

jiameixie commented May 15, 2020

jiameixie commented May 15, 2020

jiameixie commented May 15, 2020

jiameixie commented May 15, 2020

jiameixie commented Jun 1, 2020

JunHe77 commented Sep 18, 2020

chia7712 commented Oct 20, 2020

jiameixie commented Oct 21, 2020

lizthegrey commented Oct 27, 2020

JunHe77 commented Oct 28, 2020

martin-g commented Dec 29, 2020

martin-g commented Dec 29, 2020