Adding riscv64 to the list of supported architectures

[rvs]

Since I ended up being on lucky owner of a BeagleV https://beagleboard.org/beaglev I've naturally started porting https://github.com/lf-edge/eve to it. This in turn required porting of Alpine Linux (which I'm almost done with) but now I'm faced with the following problem: there's a few linuxkit packages that EVE depends on wholesale (just pointing at linuxkit/XXX artifacts on Dockerhub) and obviously those don't have riscv64 as a supported architecture. To be exact, here's the list:

  - linuxkit/init:a68f9fa0c1d9dbfc9c23663749a0b7ac510cbe1c
  - linuxkit/runc:f79954950022fea76b8b6f10de58cb48e4fb3878
  - linuxkit/containerd:1ae8f054e9fe792d1dbdb9a65f1b5e14491cb106
  - linuxkit/getty:v0.5
  - linuxkit/memlogd:v0.5
  - linuxkit/sysctl:v0.5
  - linuxkit/modprobe:v0.5
  - linuxkit/openntpd:v0.5

Now, this is probably 1/4th (1/6th?) of all the packages that linuxkit has, so the question I'm going to ask has a loaded subtext of (and if we agree -- somebody will have to help me with the rest of 3/4th (5/6th?) of packages): is there interest in making Linuxkit available on riscv64 at this point?

The good news is that BeagleV is the first example of a very practical, tangible RaspberryPi-like SBC that I'm sure a lot of folks are going to get excited about once it hits GA in a quarter or so. We are no longer stuck with eval boards and qemu. The bad news, of course, is that we're still probably a year or so away from a flood of these (so I can understand if the linuxkit community has other priorities).

Thoughts?

[deitch]

I am going to take the first stab at answering this. Expect others may have divergent opinions.

First, from the technical perspective. The packages you listed mostly are our own (although why are you running v0.5, which is quite old?), and thus should port over fairly easily. runc and containerd obviously just leverage the upstream, so as long as those compile to risc-v, these should go without a problem. It still is possible to run into issues, of course.

Second, from the process perspective. If we do want to support it properly, we need a build server. Every supported architecture has a dedicated build server of the right architecture. There are so many packages to build and support that it isn't realistic to do everything emulated. If we could get a fixed device that we could use as a build server - has to support docker - that would make this discussion much easier.

A possible alternative path might be native cross-compilation, but we haven't quite taken the next step yet. We went from "must ssh to each arch to build for that arch and push it out" to "can build on each arch remotely or via qemu locally, launching from one centralized location", (i.e. lkt pkg build builds for all via qemu or remote runners), which greatly simplified things, but not yet to "build for all archs in one place using native cross-compilation" (i.e. lkt pkg build builds for all locally via cross-compilation).

Even if we had that, I think the kernel would require a build server. @rn could comment more intelligently on that.

Third, from a dependencies perspective. Base alpine, i.e. linuxkit/alpine, which is built based on docker.io/library/alpine, needs that to be available natively on risc-v. Here is the manifest for alpine:3.13

{
  "manifests": [
    {
      "digest": "sha256:def822f9851ca422481ec6fee59a9966f12b351c62ccb9aca841526ffaa9f748",
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "platform": {
        "architecture": "amd64",
        "os": "linux"
      },
      "size": 528
    },
    {
      "digest": "sha256:ea73ecf48cd45e250f65eb731dd35808175ae37d70cca5d41f9ef57210737f04",
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "platform": {
        "architecture": "arm",
        "os": "linux",
        "variant": "v6"
      },
      "size": 528
    },
    {
      "digest": "sha256:9663906b1c3bf891618ebcac857961531357525b25493ef717bca0f86f581ad6",
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "platform": {
        "architecture": "arm",
        "os": "linux",
        "variant": "v7"
      },
      "size": 528
    },
    {
      "digest": "sha256:8f18fae117ec6e5777cc62ba78cbb3be10a8a38639ccfb949521abd95c8301a4",
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "platform": {
        "architecture": "arm64",
        "os": "linux",
        "variant": "v8"
      },
      "size": 528
    },
    {
      "digest": "sha256:5de788243acadd50526e70868b86d12ad79f3793619719ae22e0d09e8c873a66",
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "platform": {
        "architecture": "386",
        "os": "linux"
      },
      "size": 528
    },
    {
      "digest": "sha256:827525365ff718681b0688621e09912af49a17611701ee4d421ba50d57c13f7e",
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "platform": {
        "architecture": "ppc64le",
        "os": "linux"
      },
      "size": 528
    },
    {
      "digest": "sha256:a090d7c93c8e9ab88946367500756c5f50cd660e09deb4c57494989c1f23fa5a",
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "platform": {
        "architecture": "s390x",
        "os": "linux"
      },
      "size": 528
    }
  ],
  "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
  "schemaVersion": 2
}

No manifest for risc-v. Of course, you know that, as you had to work around this for eve-alpine. This team is most likely to want to wait until docker.io/library/alpine has support for risc-v. If you want to offer to upstream your cool work on bootstrapping it, people will look, but I suspect limited appetite for carrying extra bootstrap code, that in any case will go away once the Alpine team has it all ready natively.

I believe, this is the alpine issue for risc-v support

Fourth, it is just a question of resources. Once we have:

• a place to build packages (or native cross-compilation without emulation)
• a place to build kernel
• alpine support

then it just becomes a question of time to port the packages.

I do want to hear other maintainers weigh in on this.

[deitch]

Also, I want that BeagleBoard!

[rvs]

A few comments @deitch

although why are you running v0.5, which is quite old?

'cuz... they work just fine? ;-)

Second, from the process perspective. If we do want to support it properly, we need a build server. Every supported architecture has a dedicated build server of the right architecture. There are so many packages to build and support that it isn't realistic to do everything emulated. If we could get a fixed device that we could use as a build server - has to support docker - that would make this discussion much easier.

Actually we went through exact same discussion on Alpine side and came back with the conclusion that for now, a 32-way Xeon (or EPYC) server running qemu-user-static is much faster (although still pretty slow compared to native runners) than any RISCV "server" that one can get (there are no actual servers around - more like evaluation boards). Hence on Alpine side we're going to produce the next release of Alpine with that infrastructure. It isn't actually half as bad as people fear and it gets us going.

Even if we had that, I think the kernel would require a build server. @rn could comment more intelligently on that.

The kernel for Alpine is just fine building on an x86 GH action in under 90 minutes -- so that's not that big of a deal (much faster on a beefy machine)

Third, from a dependencies perspective. Base alpine, i.e. linuxkit/alpine, which is built based on docker.io/library/alpine, needs that to be available natively on risc-v. Here is the manifest for alpine:3.13

Sure, and the position linuxkit can take is wait for that. Right now it seems that 3.15 will be the first official Alpine distro to support riscv64 -- that'll be quite a bit towards the end of 2021 and waiting that long may not be ideal given that the boards are now showing up as we speak (see below) and the developer's interest is as high as it gets.

What I would suggest instead is that linuxkit follows what EVE is doing and it publishes its own equivalent of the build environment that is based on the Alpine riscv64 bits. There's absolutely not reason to wait.

I believe, this is the alpine issue for risc-v support

The actual discussion for Alpine 3.15 riscv64 support is on the mailing list

then it just becomes a question of time to port the packages.

That's fundamentally the biggest question of all. Like I said -- I'll have to do the riscv64 port ASAP of those packages we need in EVE -- once those get effectively forked, I'm all open for people to pull the changes back into linuxkit -- but my only interest in upstream work at the linuxkit level will only be as much people are willing to see this happen upstream (EVE is ambivalent on whether to repatriate those packages or not).

[rn]

I agree with the points @deitch made. I personally would not be keen supporting a new architecture without proper support in upstream Alpine and a usable build infrastructure. For the new architectures we introduced (first arm64, then s390x), Alpine support was already there and we got access to build servers from (now) equinix metal and IBM, and even patches for supporting these new architectures from ARM and IBM all the way through (equinix metal is now also supporting with a x86 build server, which is great).

With the current setup of LinuxKit, where Alpine is the base for both builds and quite a few packages, to me, upstream Alpine Linux support is a must for riscv64 support. Maintaining our own riscv64 fork of Alpine seems like a lot of effort, and even pulling in bits from elsewhere then introduces special cases and is a hassle to debug when it doesn't work.

For the build server, we could create a VM on our x86 server and see how that goes in terms of performance. How good is the distro support for riscv64? Looks like Debian has some experimental support ditto for Fedora. How is support for Docker, manifest tool and other things? Do you have instructions on how to set up such a dev environment that you could share?

Lastly, what sort of build targets are required? I'm admitting a ignorance to the riscv ecosystem here, but what sort of things need to be generated to boot a image on qemu and/or real hardware? For arm64 this was a real PITA, esp support for SBCs like RPI3. IIRC it took me a week or two full time to get it booting on a RPI3, with many iterations on kernel builds, uboot, device tree etc. What is required for even a simple linuxkit run with qemu?

[rvs]

Hey @rn a couple of comments:

I personally would not be keen supporting a new architecture without proper support in upstream Alpine and a usable build infrastructure.

Well, then it puts EVE in a precarious position of effectively forking linuxkit and sort of reversing the upstream/downstream relationship here. It is fine for you guys to trail behind -- just wanted to know what does the community want.

For the new architectures we introduced (first arm64, then s390x), Alpine support was already there and we got access to build servers from (now) equinix metal and IBM, and even patches for supporting these new architectures from ARM and IBM all the way through (equinix metal is now also supporting with a x86 build server, which is great).

And sometimes you've got to lead ;-)

Maintaining our own riscv64 fork of Alpine seems like a lot of effort, and even pulling in bits from elsewhere then introduces special cases and is a hassle to debug when it doesn't work.

Nobody's talking about the fork -- I'm the guy doing Alpine riscv64 port -- so all we're talking about is latching onto edge instead of a release. A release is likely to be 3.15 -- but that'll be too late for EVE needs (not that you guys should care that much -- just saying).

For the build server, we could create a VM on our x86 server and see how that goes in terms of performance.

It does pretty well -- that's how we plan to do Alpine 3.15 release.

How good is the distro support for riscv64?

It is reasonable -- Fedora is there and so is Debian.

Do you have instructions on how to set up such a dev environment that you could share?

Yup ;-)

$ docker build -t riscv64 https://raw.githubusercontent.com/lf-edge/eve/master/build-tools/src/scripts/Dockerfile.alpine.bootstrap
$ docker run -it --rm riscv64 sh

Lastly, what sort of build targets are required? I'm admitting a ignorance to the riscv ecosystem here, but what sort of things need to be generated to boot a image on qemu and/or real hardware? For arm64 this was a real PITA, esp support for SBCs like RPI3. IIRC it took me a week or two full time to get it booting on a RPI3, with many iterations on kernel builds, uboot, device tree etc. What is required for even a simple linuxkit run with qemu?

I already have full support (including kernel) for this SBC https://beagleboard.org/beaglev in both Alpine Edge and soon EVE. Works like a charm. qemu is also quite functional.

What is required for even a simple linuxkit run with qemu?

Well, arguably -- this is already done in EVE -- I can boot into existing bits and pieces in the latest master. Hop on Slack and I can walk you through the process -- but it really is just our make file targets at this point.

[rn]

@rvs frankly I'm a little annoyed by your comments

LinuxKit maintainers are volunteers who do this work in their spare time. Adding a new platform might be good fun, but maintaining it is quite a bit of effort.

Well, then it puts EVE in a precarious position of effectively forking linuxkit and sort of reversing the upstream/downstream relationship here. It is fine for you guys to trail behind -- just wanted to know what does the community want.

I seriously do not understand this talk about "forking" and "trailing behind" . If you as a downstream consumer of LinuxKit would like to see new features and support for a new architecture, we are happy to work with you, review patches etc. But, you also have to understand that, esp adding a new architecture does impose an additional strain on maintainers and you simply do not address this with your comments. Both ARM and IBM when they contributed support for arm64 and s390x worked closely with the maintainers, did most of the work in terms of submitting PRs and provided resources to maintain it afterwards. Now, since there are no riscv64 servers, I'd be happy to reuse one of the existing build servers but you will have to provide the instructions on how to set up the dev environment.

Both deitch@ and me have pointed out that in order to make the maintain part manageable we need a stable base to work off and a working development environment.

What I mean by working development environment for a new architecture is something I can ssh into, git clone and type make push and it just works. The Dockerfile you reference above is far from this and also just pulls random stuff from a random S3 bucket. I found instructions for creating a riscv64 VM for Debian but they are quite pedestrian and experimental (though it did seem to work). Still quite a bit away from a straight forward dev environment. We need docker and manifest-tool as well. So, again, we need instructions on how to set up a dev environment for riscv64.

Nobody's talking about the fork -- I'm the guy doing Alpine riscv64 port -- so all we're talking about is latching onto edge instead of a release. A release is likely to be 3.15 -- but that'll be too late for EVE needs (not that you guys should care that much -- just saying).

First time you mention that the Alpine port will be in Alpine edge. This certainly makes it easier. I'd be happy to consider building linuxkit/alpine from Alpine Edge for riscv64 (but keep the other arches on stable). However this still requires to have access to a development environment.

Well, arguably -- this is already done in EVE -- I can boot into existing bits and pieces in the latest master. Hop on Slack and I can walk you through the process -- but it really is just our make file targets at this point.

Happy to review PRs for linuxkit run...

To move forward, I suggest:

• Provide patches to introduce riscv64 as an optional architecture to linukit pkg build. It has to be optional for now since some build.yml don't specify a list of architectures and then riscv64 builds would fail.
• Provide instructions for setting up a dev environment
• Once riscv64 is in edge, provide patches to linuxkit/alpine to build the alpine base for riscv64 from edge
• Make sure the packages you care about build for riscv64
• Move riscv64 to a supported arch for linuxkit pkg build and adjust build.yml as needed
• Add kernel builds for riscv64. Initially I'd suggest to just use the latest (LTS) kernel.
• Contribute linuxkit run code to run riscv64 images in qemu, assuming kernel+initrd works, otherwise we also need linuxkit build support to build appropriate images

As you can see, this is a non-trivial amount of work.

[dave-tucker]

This is going in the opposite direction to #3674 #3675 #3676 -adding new things vs. removing things.

My $0.02 (GBP equivalent) is that I'd love to see linuxkit on riscv but given that we're stretched even with amd64/arm64 support (assuming s390x is removed) I'm not sure that we can do it alone.

@rn @deitch I've opened #3677 to try and distil some of what has been discussed here in to documentation so we don't lose it. feel free to provide suggestions and i'll get them folded in.

@rvs also re: ☝️ i've tried to call out what support you can expect from us at each step of the way.
forking is certainly one option to bootstrap riscv support, however (if the other maintainers agree) we could also open up a feature branch for you here, or look at adding support to our build system CI such that the new arch remains optional - so we could release v0.9 without it for example. ultimately though I'd want some commitment either from yourself or a member of the community that someone with some risc-v experience is willing to help with the ongoing maintenance should we promote it to being a "first-class supported" arch.

to give some concrete examples of "arch-specific issues" and some of the reasoning behind #3676

1. the go s390x support is limited to a specific version of the instruction set
2. cross buillds for s390x (using binfmt_misc) fail with illegal instruction errors because of a mismatch between the go language support and runtime cpu type in qemu - i've yet to find one that works
3. because of this we've had to disable testing s390x in CI

one of the things that i've been trying to find time to do for a while is to make the contribution process easier + reduce the burden on maintainers to have to build/push changes to pkg for them to be tested. having a ci do this for us would be great, but it also means we need to avoid issues like the ones mentioned above... or have a person on hand to help resolve them.