[FAB-9629] support for building multi-arch docker images

[paleozogt]

Type of change

• New feature

Description

I want to be able to use Hyperledger Fabric on various architectures, including arm64. Since HLF is written in Go I figured this should be straightforward to support.

Sadly the Docker Hub images are x86_64 only. However, using Docker's new buildx makes building multi-arch images easy.

I've modified the HLF makefiles so that the various xxx-docker targets build for multiple architectures. So that building the images doesn't take forever, cross-compilation on the host arch is used.

Additional details

Building images is much the same as it currently is, with some additional setup:

• Setup qemu and buildx:

  $ docker run --rm --privileged multiarch/qemu-user-static --reset -p yes
  $ docker buildx create --name mybuilder
  $ docker buildx use mybuilder
  

• Build peer images:

  $ make peer-docker
  ...
  
  $ docker images --format "{{.Repository}}:{{.Tag}}" | grep peer
  hyperledger/fabric-peer:2.1
  hyperledger/fabric-peer:2.1.0
  hyperledger/fabric-peer:amd64-2.1.0-snapshot-67a821897
  hyperledger/fabric-peer:latest
  hyperledger/fabric-peer:s390x-2.1.0-snapshot-67a821897
  hyperledger/fabric-peer:ppc64le-2.1.0-snapshot-67a821897
  hyperledger/fabric-peer:arm64-2.1.0-snapshot-67a821897
  

• Publish peer multi-arch manifest:

  make peer-publish-images
  

The orderer, tools and ccenv images are built similarly.

I've published these multi-arch images to my own repo and have been using them. I can't find how the makefiles are invoked by CI, so I can't test this any further. However, I'm willing to work on integrating these changes into any CI scripts.

Related issues

HLF Jira has issue FAB-9629, but it seems to be closed as "Won't Do" without any comments. Hopefully it was closed due to no one wanting to work on it and not for some other reason.

While this PR is on master, I've also implemented this on the release-2.0 and release-2.1 branches.

[sykesm]

HLF Jira has issue FAB-9629, but it seems to be closed as "Won't Do" without any comments. Hopefully it was closed due to no one wanting to work on it and not for some other reason.

The issue was closed with a comment of 'stale' along with hundreds of others that received no meaningful update in more than 9 months. If you look at it, you will see that all of the associated items were completed in the Fabric 1.1 time frame.

Fabric created multi-arch images for amd64 and s390x until very recently. We stopped when we moved off of the LF CI infrastructure and then eventually removed the multi-arch script when we were only doing amd64.

At this point we have no CI for anything other amd64-linux. Without CI for all of the platforms, we are not in a position to maintain or publish images for any other platform.

Basically, it's not the tools that are the issue, it's the infrastructure.

[paleozogt]

@sykesm Azure can do multi-arch CI by using qemu. The image-building I've already implemented. But if you need to do multi-arch tests I can implement that as well.

[sykesm]

Azure can do multi-arch CI by using qemu. The image-building I've already implemented. But if you need to do multi-arch tests I can implement that as well.

It's not just building; it's maintaining, documenting, and debugging. Every new platform and every new layer adds more scope to the work we do and new platforms tend to have a long tail.

If you're interested in formally proposing this work, please take a look at https://github.com/hyperledger/fabric-rfcs and have a conversation with @denyeart to determine the best way to propose this.

[mastersingh24]

Azure can do multi-arch CI by using qemu. The image-building I've already implemented. But if you need to do multi-arch tests I can implement that as well.

It's not just building; it's maintaining, documenting, and debugging. Every new platform and every new layer adds more scope to the work we do and new platforms tend to have a long tail.

If you're interested in formally proposing this work, please take a look at https://github.com/hyperledger/fabric-rfcs and have a conversation with @denyeart to determine the best way to propose this.

Prior to the 2.x release, the maintainers decided that we were only going to publish x86_64 Docker images. The images we publish are really only for use in our tutorials and getting started demos. They are not built or maintained for other uses.

But the code is open source so people are free to clone it and use it however they like.

[paleozogt]

@mastersingh24 The HLF Docker images aren't intended to be used in production? I find that surprising. It seems like anyone using kubernetes would just use the Docker hub images rather than building their own.

[lindluni]

"Anyone who uses Kubernetes" is a pretty general statement. Many organizations who are serious about running Fabric purchase support contracts from third parties, or use Blockchain-as-a-Service offerings (which most cloud providers have now), where the images, and service are built and maintained by these third parties.

Consider a few other open source projects:

• Docker - Offers Docker Enterprise as its "supported" version
• Kubernetes - AKS, EKS, GKS, OpenShift, Rancher and Mirantis are what people run in production and support comes from these companies. In these offerings the companies tweak the software to harden it and then create their offering around their enterprise supported implementation.

An open source license expressly states that software is granted "as is" (sections 7, 8 and 9 of our LICENSE, https://github.com/hyperledger/fabric/blob/master/LICENSE#L143), and we do, on a best effort, keep up with security and bug fixes, but if you are running in production, waiting a few weeks for our next release may not be an option when you are staring down a potential exploit in your Docker images, and you have no warranty to fall back on.

[paleozogt]

@btl5037 I'm maintaining my own fork of HLF in order to have multi-arch images. I figured the community would prefer to go to an official source for such things, so I tried to contribute it back. I'm rather surprised by the pushback.

I'm hearing two things. One, it needs to be aggressively tested on CI in order for it to be accepted (which I can appreciate). But I'm also hearing that the docker images aren't intended to be used (!), so why bother expanding the arches.

[lindluni]

As I mentioned in RocketChat, you're also missing quite a few pieces, in order for us to run our tests across these images it also requires you to create multi-arch images of the fabric-ca, fabric-javaenv, fabric-nodeenv, fabric-couchdb, fabric-zookeeper, and fabric-kafaka` images. Both Fabric and Fabric-CA also have sets of binaries we maintain in our GitHub releases and our Artifactory instance. We've now spread the surface area across 4 more repositories, and I can tell you from experience, as I've implemented arm on these images, it's a monumental feat as there exist almost no baseimages for any of the architectures you are proposing support for.

As the person who maintains our DockerHub repositories, Artifactory repositories, and our GitHub releases, the thought of adding support for all (or any one) of these would make me lose sleep. It's a monumental undertaking to keep x86 images in line across all of our repos, multiplying that by 4 would be near impossible to maintain, as we don't only have to maintain the images, but we also have to perform bug fixes that may affect only one of these platforms.

I can absolutely appreciate where you are coming from, and at its face, it seems like a great idea. But given a finite set of resources, and the fact no one is screaming from the hills for this, its just not something we can undertake without a strong proposal for continued support and maintenance (of the images and the Fabric code itself), and a really, really good reason for why open source should host this work.

[mastersingh24]

@btl5037 I'm maintaining my own fork of HLF in order to have multi-arch images. I figured the community would prefer to go to an official source for such things, so I tried to contribute it back. I'm rather surprised by the pushback.

I'm hearing two things. One, it needs to be aggressively tested on CI in order for it to be accepted (which I can appreciate). But I'm also hearing that the docker images aren't intended to be used (!), so why bother expanding the arches.

You bring up some interesting points .... I'll net it out this way for you:

Hyperledger Fabric is a project not a product. The official artifact for each release is actually the tarball of the source code. This is actually the same as every Apache Foundation project as well (although some do also provide binaries). Fabric also provides binaries for 3 platforms which are also under each release.

And it's not that we do not appreciate your contribution; it's simply that we made a decision not to publish multi-arch images as many people were assuming that these were "product releases".

And when I say they are not production images, I mean that we do not constantly scan them and update vulnerabilities. Almost any company which runs containers in production requires vulnerability scans and timely patching. We do, however, fix any security issues in the actually source as quickly as possible. We cannot and do not prevent people from using the images we publish however they like, but we also do not treat them like an official product either.

[lindluni]

Closing, as the decision to only produce x86 artifacts was decided by the Fabric maintainers as a part of the CICD migration several months ago. Users who wish to create artifacts for other platforms are free to do so independently.

[ramesh-m99]

Apple m1 Pro requires arm-based docker fabric images. fabric-samples chain code installation is failing. its better to start this PR