aarch64 distribution

[nvisser]

• Program: dnsdist
• Issue type: Feature request

Short description

Currently there seems to be no distribution of dnsdist for the aarch64 platform.

Usecase

Running dnsdist on modern ARM aarch64 based hardware.

Description

Modern ARM based platforms are based on aarch64 so having the ability to use dnsdist (or any other powerdns program, really) on these platforms without having to spend a long time compiling or fiddling with cross-compiling would be ideal.

[Habbie]

#4663 requests this for 'debian', closing that one as this ticket asks it more widely.

Are there specific distributions for which you'd like this?

[dkowis]

I'd like to see it for Armbian. Armbian seems to be super useful when installing to pine64's. It'd also be handy for building an aarch64 docker container for similar purposes.

[kpfleming]

I'd use this too, I've switched my Raspberry Pis over to plain Debian and the aarch64 kernel (arm64 Debian package flavor). If there's anything I can do to help with this let me know.

[Habbie]

I understand that https://drone.io/ has arm64 CI runners. Perhaps it makes sense to delegate arm64 builds for a few distributions to them, as they have ARM64 hardware. We do not, which means our arm builds (like the Raspbian builds we do today) run under QEMU which is very slow.

I'd welcome a PR for doing this with Drone as an experiment :)

[kpfleming]

Travis-CI has aarch64 support available in beta form: https://docs.travis-ci.com/user/multi-cpu-architectures/

Using that will be tremendously easier than trying to use any other CI platform, given how much work has gone into build-travis.sh.

[arthurpetitpierre]

Travis-CI can now build on Graviton2/arm64 instances: https://blog.travis-ci.com/2020-09-11-arm-on-aws
There's a program to get credits for open-source projects on AWS: https://pages.awscloud.com/AWS-Credits-for-Open-Source-Projects , supporting such a build pipeline would be a perfect use-case.

If a PR would help to get that bootstrapped, please let me know.

[kpfleming]

I took a look at the pipeline for building packages on Travis-Ci, and it's definitely non-trivial. Most importantly it has an assumption of just one architecture, so adding a second one will require teaching which target distributions are supported on each architecture in the build matrix. At a minimum CentOS 6 and Ubuntu 16.04 are unlikely to be usable on aarch64 (or even desired).

[pieterlexis]

I took a look at the pipeline for building packages on Travis-Ci, and it's definitely non-trivial. Most importantly it has an assumption of just one architecture, so adding a second one will require teaching which target distributions are supported on each architecture in the build matrix.

We are planning to look at GH actions (no promises though), perhaps this might make things easy

[kpfleming]

I don't believe GitHub Actions offers native aarch64 builds at this time; it can be done via QEMU, but Travis-CI offers native builds.

[kpfleming]

I've taken a look at the build process, and it seems relatively understandable (although it's fairly complex!). The one thing I can't seem to find is the script which actually iterates over all the targets in builder-support/Dockerfiles and executes the builds (and then extracts the resulting packages from the images). This script will need to understand the list of targets which are available on each architecture.

[Habbie]

If you init & update git submodules, builder/build.sh appears. It is called like builder/build.sh centos-7 or builder/build.sh -m authoritative debian-buster etc.

It probably makes sense to hardcode the list of aarch64 targets (probably way shorter than our full list) instead of trying to iterate over a directory listing.

[kpfleming]

Yep, I found that part, I was just wondering what actually invokes build.sh for each of the available targets :-) It doesn't seem to be in the .travis.yml or .circleci configurations. The tool that is doing the iterating is the one that will have to be taught which distro/arch combinations are valid.

[Habbie]

We don't build packages on Travis or CircleCI currently. The only place -we- call build.sh is in the configs for https://builder.powerdns.com/, so it makes sense that you could not find that :)

[Habbie]

In other words, you'll have to write that five line shell script.

[kpfleming]

Got it... then I can propose a new script which is aware of the host architecture and chooses the targets to build for it, and you can decide how to integrate that into the real build process.

[Habbie]

Yes - I see two open questions there:
(1) where do we put the packages after Travis has built them
(2) do we trust Travis enough to make these packages 'official' and sign them with a pdns key

[Habbie]

By the way, we have no need (or use) for a script around build.sh for amd64, because we list those targets in our buildbot config already. So feel free to underengineer the script.

[kpfleming]

Another option is to leverage the "AWS for Open Source" link above and get AWS aarch64 compute resources that builder.powerdns.com can use.

In either case I can get to the point where I can prove that the existing build processes run properly on an aarch64 machine and produce usable packages for most of the distros that are in the list today.

[Habbie]

Oh! I missed that comment! Indeed that would also make sense, but we wouldn't get to it soon. Getting packages out of Travis would still be a great start.

[kpfleming]

Well, at least some simple testing produced good results:

• launched a t4g.large instance in AWS EC2
• cloned pdns repo master branch and initialized/updated submodule
• installed Docker engine
• ran builder/build.sh -m recursor debian-buster

The build ran to completion with no errors visible. One small issue: the default build uses only one CPU, which is somewhat annoying when you are running builds manually :-) Adding an appropriate DEB_BUILD_OPTIONS in build-debs.sh solves that problem, although based on the debhelper documentation that should not be necessary... This could be an issue if the jobs are run in Travis-CI because they'll use more wall-clock time than necessary, and the jobs may hit the maximum time limit.

Unsurprisingly, builder/build.sh -m recursor raspbian-buster also works just fine, and of course no QEMU is required.

[Habbie]

Unsurprisingly, builder/build.sh -m recursor raspbian-buster also works just fine, and of course no QEMU is required.

That is not entirely unsurprising! When we tested on aarch64 last year, we had a box with -no- arm 32 bit support. So this is excellent news!

[Habbie]

One small issue: the default build uses only one CPU, which is somewhat annoying when you are running builds manually

I noticed the same last year, but I did not dig in to find the -right- solution.

[kpfleming]

One small issue: the default build uses only one CPU, which is somewhat annoying when you are running builds manually

I noticed the same last year, but I did not dig in to find the -right- solution.

My hack of a fix was to set DEB_BUILD_OPTIONS='parallel=4' in the script line which calls fakeroot. Clearly this is suboptimal as it should be configurable (or at least default to the number of CPU cores on the build machine), and based on the debhelper documentation shouldn't even be necessary as if debian/compat is set to 10 or higher (it is for recursor and dnsdist, but not yet for authoritative) and the version requirement for debhelper is set to 10 or higher in debian/control (it is for recursor and dnsdist, but not yet for authoritative), then parallel building is supposed to be the default.

I've got some debhelper-knowledgeable colleagues at $dayjob so I'll ask them for guidance on that front.

[kpfleming]

First build failure: building the authoritative packages from the 4.3.0 tag produced some test failures.

testrunner: ../ext/luawrapper/include/LuaContext.hpp:107: LuaContext::LuaContext(bool)::<lambda(lua_State*)>: Assertion `false && "lua_atpanic triggered"' failed.
unknown location(0): fatal error: in "lua_auth4_cc/test_prequery": signal: SIGABRT (application abort requested)
test-lua_auth4_cc.cc(20): last checkpoint: "test_prequery" test entry
testrunner: ../ext/luawrapper/include/LuaContext.hpp:107: LuaContext::LuaContext(bool)::<lambda(lua_State*)>: Assertion `false && "lua_atpanic triggered"' failed.
unknown location(0): fatal error: in "lua_auth4_cc/test_updatePolicy": signal: SIGABRT (application abort requested)
test-lua_auth4_cc.cc(47): last checkpoint: "test_updatePolicy" test entry

[Habbie]

ah yes, luajit is broken on aarch64. We have workarounds in #6512 but they are not acceptable for general consumption (i.e. they might create slowdowns for other architectures).

Cleanest would probably be to build against lua 5.3 instead.

[kpfleming]

Confirmed; switching to liblua5.3 allows the build to complete and the tests to pass. This means we'll end up having different Debian configuration files (at least) for amd64 and aarch64 I suppose.

I've also apparently succeeded in getting parallel builds to work using the documented mechanism (at least for versions of Debian which support debhelper 10.x and higher), but I'll not yet claim success there until I've tested it with dnsdist and recursor too :-)

[kpfleming]

Ahh, well... hmm. The EPEL page says that EPEL-7 is no longer available for aarch64. That means we either drop CentOS 7 from the aarch64 package list, or we allow the fallback to ucontexts.

At this point, I think we could just go with this distro list for aarch64:

• Debian Buster
• Ubuntu Bionic
• Ubuntu Focal
• CentOS 8

If the AWS crew want to work on support for Amazon Linux 2 they are certainly welcome to do so!

The aarch64 builders can also be used for the Raspbian Stretch and Buster packages, but those produce armhf packages, so aarch64 is not a concern there.

If this plan works for the maintainers, I'll start cleaning up my branch of changes for the builder-support tree and get a PR opened.

[Habbie]

If this plan works for the maintainers, I'll start cleaning up my branch of changes for the builder-support tree and get a PR opened.

Yes please!

[andypost]

Alpinelinux is building dnsdist for 7 arches https://pkgs.alpinelinux.org/packages?name=dnsdist&branch=edge

Latest update to 1.5.1 shows few failed tests

test-suite.log {{{
     1  =====================================
     2     dnsdist 1.5.1: ./test-suite.log
     3  =====================================
     4  
     5  # TOTAL: 1
     6  # PASS:  0
     7  # SKIP:  0
     8  # XFAIL: 0
     9  # FAIL:  1
    10  # XPASS: 0
    11  # ERROR: 0
    12  
    13  .. contents:: :depth: 2
    14  
    15  FAIL: testrunner
    16  ================
    17  
    18  Running 93 test cases...
    19  unknown location(0): ^[[4;31;49mfatal error: in "dnsdistlbpolicies/test_lua": LuaContext::ExecutionErrorException: bad light userdata pointer^[[0;39;49m
    20  test-dnsdistlbpolicies_cc.cc(455): ^[[1;36;49mlast checkpoint: "test_lua" test entry^[[0;39;49m
    21  unknown location(0): ^[[4;31;49mfatal error: in "dnsdistlbpolicies/test_lua_ffi_rr": LuaContext::ExecutionErrorException: bad light userdata pointer^[[0;39;49m
    22  test-dnsdistlbpolicies_cc.cc(511): ^[[1;36;49mlast checkpoint: "test_lua_ffi_rr" test entry^[[0;39;49m
    23  unknown location(0): ^[[4;31;49mfatal error: in "dnsdistlbpolicies/test_lua_ffi_hashed": LuaContext::ExecutionErrorException: bad light userdata pointer^[[0;39;49m
    24  test-dnsdistlbpolicies_cc.cc(569): ^[[1;36;49mlast checkpoint: "test_lua_ffi_hashed" test entry^[[0;39;49m
    25  unknown location(0): ^[[4;31;49mfatal error: in "dnsdistlbpolicies/test_lua_ffi_whashed": LuaContext::ExecutionErrorException: bad light userdata pointer^[[0;39;49m
    26  test-dnsdistlbpolicies_cc.cc(626): ^[[1;36;49mlast checkpoint: "test_lua_ffi_whashed" test entry^[[0;39;49m
    27  unknown location(0): ^[[4;31;49mfatal error: in "dnsdistlbpolicies/test_lua_ffi_chashed": LuaContext::ExecutionErrorException: bad light userdata pointer^[[0;39;49m
    28  test-dnsdistlbpolicies_cc.cc(681): ^[[1;36;49mlast checkpoint: "test_lua_ffi_chashed" test entry^[[0;39;49m
    29  
    30  ^[[1;31;49m*** 5 failures are detected in the test module "unit"
    31  ^[[0;39;49mFAIL testrunner (exit status: 201)
    32  
}}} test-suite.log

[kpfleming]

That looks like the aarch64/luajit problem.

[andypost]

As I got it should use lua5.3 but disable jit for aarch64? 6afb693#diff-b1dc70edc0e96e8e68616ec2894bb235R22-R27

[kpfleming]

Yep, that's it.

[andypost]

@kpfleming thank you, excluded jit from aarch64

[muhlba91]

Any chances of getting official arm64 (buster) builds in the apt repositories soon-ish? :)

[Habbie]

I'm doing some testing on an AWS t4g.medium (2 vCPU, 4 GiB, 28 USD/month) with Debian Buster.

builder/build.sh -m dnsdist raspbian-buster (spotted to do make -j1) takes 17 minutes. It also emits a weird warning WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested.

builder/build.sh -m dnsdist debian-buster (also make -j1) takes 14 minutes.

[Habbie]

On a t4g.large (2 vCPU, 8 GiB, 55 USD/month) (I ran out of memory on the medium during this experiment), our Debian Buster builds, doing auth/rec/dnsdist in parallel, take 22-35 minutes each, which also totals to 35 minutes. This is, as I understand it, a likely situation for our Buildbot scheduling. Available memory (the available column in free) did not drop below 3.9GB in the process, so there's plenty of headroom there.

As a secondary data point, this grows /var/lib/docker by 11GB.

[kpfleming]

I used a t4g.xlarge I think, since I only needed it for a day. If you do decide to use a t4g.large, a 12-month reserved instance is under US$350 for the year, which is a substantial discount.

[mhill8]

Where does this stand? I was hoping to migrate my current Cloud PDNS server to a t4g (Graviton2) instance by the end of this month, for which Amazon Linux 2 and Debian 10 are probably my best candidates. Having an official repo for either of those, on that architecture, would be great.

[Habbie]

This is on my TODO, but many things are above it. You can build your own packages with builder/build.sh if you check out pdns.git and install Docker, by the way.

[kpfleming]

If you're willing to run Debian 11 pre-release (not far from actual release), there are arm64 packages in the distribution repository. I've got two Graviton 2 public auth servers running that way now.

[mhill8]

kpfleming, that sounds like a great option. I'm not too experienced with Debian (my exposure is mainly from maintaining a few Ubuntu and Mint laptops), so say I installed the "debian-11-arm64-daily-20210702-691" AMI (ami-03bcabcaeb252fa69) and updated it regularly; once 11 was released, would this instance be just like any fresh 11 install, or would I need to create a new instance from the 11 release image to be able to keep it up to date?

[kpfleming]

I did it by launching a Debian 10 AMI, then doing an in-place upgrade to 11 (which is quite easy to do on Debian). I don't know how the 'daily' images are configured, so can't provide any direct experience there.

[egarcia-r]

Hi,
Actually there is any intention to add "debian-buster-arm64" option on the "4.4.x" branch on the official repo?

Bye the way, how can I create a 4.4.1 "debian-buster-arm64" package? On 4.4.x branch there is not available this option.

[Habbie]

Bye the way, how can I create a 4.4.1 "debian-buster-arm64" package? On 4.4.x branch there is not available this option.

Building 'debian-buster' with build.sh on 4.4.x, when done on an arm64 box, might work. If not, you'll have to backport some things. Note that 4.5.1 is out now.

Actually there is any intention to add "debian-buster-arm64" option on the "4.4.x" branch on the official repo?

It's unlikely we'll do this for 4.4. I do hope we get around to it for 4.5 soon.

[egarcia-r]

Building 'debian-buster' with build.sh on 4.4.x, when done on an arm64 box, might work.

It worked!

It's unlikely we'll do this for 4.4. I do hope we get around to it for 4.5 soon.

I think I will go with building 4.4.x by now until I upgrade to 4.5 in near future and the official repo include the debian-buster-arm64.

Thanks for your great support and fast reply ;)

[zeha]

I'd like to echo the bullseye/arm64 packages request :-)

[MestreLion]

Came here following the instruction "Only packages for 64-bit Intel (amd64/x86_64) distributions and Raspbian armhf (32-bits) are provided. Should you miss your favorite Operating System or Release, please let us know. "

So... letting you know an Ubuntu Server 20.04 arm64/aarch64 build would be very welcome! 🥇

[cernoel]

i would love to see arm builds for all powerdns apps, oracle is offering (free) Ampere servers with Ubuntu, maybe one can build that on them?

edit: ... nevermind, i found thread: #10499 .. and i could build it via your docker build script .. using ubuntu focal target.. /builder/build.sh ubuntu-focal-arm64

[dkowis]

I was able to build nicely for arm64 docker using simple docker buildx

How far away are arm64 packages/docker containers? This software is perfect for an rpi (armbian, or pine64) in my homelab to give me some more powerful DNS availability. Thanks!