bitcoin/bitcoin:alpine-static on hub.docker.com

[tegila]

This pull request is one possible solution to issue #18110.

After a long time waiting for a official image from bitcoin/bitcoin on hub.docker.com,
which both can be small and affordable to small clouds and be deployed to thousands of
nodes on container providers at the same time maintaining a clean and non obstructive code,
and maintained by the most used code for bitcoin.

It was a hard work of 3 days compiling and recompiling until I found a way to compile the
binary inside a alpine:latest container, and his different libc, musl and linking the binary as static,
avoiding any possible dependencies opening door to use even smaller images.

The current image has 34Mb but still have enough doors for improvement.
It runs a full node very easily as > docker run tegila/bitcoin-static-alpine
I hope it can be done soon using > docker run bitcoin:latest
And many cloud providers had the same feeling.

milestone/52 by @laanwj also add issue #18110

I'm using the building system provided by hub.docker.com which also open doors to tests and
other CI's but for now only a full node image was my target as I have servers to deploy.

Hope you guys like it as I did, and hope we can raise more full nodes making easy to cloud providers deploy bitcoin.

I'll left it here:

docker run tegila/bitcoin-static-alpine

[maflcko]

See bitcoin-core/packaging#55

[tegila]

Hi @MarcoFalke,

I didn’t got your point on the refereed packaging as this pull request isn’t a concurrent for the current ‘continuous integration’ gitian in place, but it’s also one more solution to heavily increase the user base by putting an official Bitcoin-core image on docker hub.

it also isn’t a random dockerfile as I hardly managed to produce a static binary without any dependencies (you can check with ldd) and my following plans are to run this right after kernel boot.

By your comments I seen some misunderstood about how docker images work because after we find a good dockerfile we don’t have to actively maintain it, as the build process will stay the same for a long time. We don’t have to keep changing it’s contents every time we deploy a new version but instead docker hub will detect the change in our GitHub, fetch the code, build and deploy direct into the biggest marketplace to cloud deployers.

It’s a totally different approach to download some random guy dockerfile, who could put some random code inside the container, so you have to check his source file, line by line. And them you have to build yourself and it takes a lot of time, and the worst, most images you referred in your packaging will end up with more than 600mb (1.2Gb during building phase) just to deploy what is supposed to be a very simple deployment.

as you can see by my files it produces a very simple image using a very simple build structure and build the code using the own GitHub source code and tracking the versions without any human interaction.

if I push some new code into GitHub it will reflect in maybe 40min into docker.hub official image.

I hope you can understand that as BITCOIN grows, more deployers will want to use BITCOIN as a service and they didn’t spect to get into Bitcoin .com .org .net or something searching for instructions on how to build (Slackware 2001 feeling) but instead they want deploy automatically using their kubernetes infrastructure and Bitcoin-core is far behind this kind of technology.

Well, I’m a long time BITCOIN enthusiastic and I’m very happy to be able to deploy BITCOIN inside my clouds without that kind of hassle using a minimal image from a trusted source.

I can understand the need of everyone build their own code from source as I have been user of Slackware and gentoo for a long time, but this approach is more suitable to enthusiasts than for cloud formations.

Also I want keep the attention that the current pull request didn’t change any significant code, build process, or any previous work that were being done.

Having a official docker hub image for Bitcoin-core is an important step to reduce the distance between the users and the developers and can raise significantly the numbers of nodes in the network.

if you have any more questions I’m glad to help.

[maflcko]

We absolutely do see builds breaking when the dependency packages change, so there is additional maintenance to keep Bitcoin Core running with the alpine packages.

Also, with dependencies changing from down under us without our control, the "moving" dockerfile will only be tested in production (whenever the docker cache is cleared and the new dependency package versions are downloaded).

Finally, I am not familiar how and who maintains the alpine packages. It wouldn't be ideal if one of the packages was updated (even intermittently) to include a key-stealer.

Our static release builds fix all those issues, which is why most other docker images use them: bitcoin-core/packaging#55 (comment)

[tegila]

We absolutely do see builds breaking when the dependency packages change, so there is additional maintenance to keep Bitcoin Core running with the alpine packages.

I've been building bitcoin since 2015 in the same way, cd depends;make....cd ..; ./configure; make; make install, so not much change. As the dependency packages change they will probably be added to depends folder to pass the tests and if it's too much of a problem I can maintain it myself for the best of our community.

Also, with dependencies changing from down under us without our control, the "moving" dockerfile will only be tested in production (whenever the docker cache is cleared and the new dependency package versions are downloaded).

Finally, I am not familiar how and who maintains the alpine packages. It wouldn't be ideal if one of the packages was updated (even intermittently) to include a key-stealer.

well, alpine docker is an official docker image from docker hub team.
Docker, Inc. sponsors a dedicated team that is responsible for reviewing and publishing all content in the Docker Official Images. This team works in collaboration with upstream software maintainers, security experts, and the broader Docker community. https://docs.docker.com/docker-hub/official_images/

It has 1bi downloads and it's embedded in most cloud servers this days, like: nginx, mysql, mongo, etc.
https://hub.docker.com/_/alpine

So if someone can inject something inside an alpine package they can shutdown the entire internet this days. Not saying it's not possible but maybe you're being toooo strict with something helpful.

Our static release builds fix all those issues, which is why most other docker images use them: bitcoin-core/packaging#55 (comment)

I mean, I got your point in defend what you have working with and I was carefully when pulling this request to NOT change any of that.

I can understand the importance of a 'continuous deployment' focused only on development as a point of trust to check tests and reproducibility. BUT, you can't put all your energy in one single point of failure. In the same way someone can inject a key stealing inside any CI environment and get advantage of that.

The point here is create a NEW alternative, which have been expected for many years now, by many people in our community.

If you aren't happy by my work here you can simply put your pre-build packages inside docker hub. But arging alpine isn't trustable...

but please put the bitcoin thing inside docker hub.

[sipa]

Note that this isn't about trust in maintainers; it's about verifiability. Our current gitian (soon guix) based build process can be repeated by anyone, allowing them to verify that the published binaries do in fact correspond to the source code (which hopefully has many eyes on it already).

I'd support extending our build process to also create deterministic docker images, if that's possible, if someone wants to take that on.

[maflcko]

I've been building bitcoin since 2015 in the same way, cd depends;make....cd ..; ./configure; make; make install, so not much change. As the dependency packages change they will probably be added to depends folder to pass the tests and if it's too much of a problem I can maintain it myself for the best of our community.

Depends are also used by our release builds in Guix, so yes this is an improvement. Though the dockerfile attached to this pull request doesn't use depends, unless I am missing something?

In the same way someone can inject a key stealing inside any CI environment and get advantage of that.

We don't use a CI environment in any of our release process, so this shouldn't be a problem.

If you aren't happy by my work here ...

I appreciate the work, and I don't reject it in any way. I am trying to explain why a docker image didn't exist in the past and why using the official release binaries might be preferable and potentially could be moving this effort forward.

There is also a good point by @sipa that maybe the release process itself can(?) be used to create the docker image.

I am not a docker expert myself, so I can't comment on the implementation of this, but I think the current dockerfile in this pull is not going to be accepted as is. It should be easy to at least use the depends system, which also shouldn't come with any downside.

Note that this isn't about trust in maintainers; it's about verifiability.

Our current "build from source" instructions are not verifiable either, so at least in theory it is thinkable that a dockerfile that mimics this documentation is added and left for the users at their own risk, just as the build docs are left for them at their own risk.

However, given that the goal here seems to be to ship release builds, it should be easier and without downsides to ship the existing releases via docker.

[tegila]

Though the dockerfile attached to this pull request doesn't use depends, unless I am missing something?

https://github.com/tegila/bitcoin/blob/master/Dockerfile#L12

However, given that the goal here seems to be to ship release builds

Yes.

it should be easier and without downsides to ship the existing releases via docker.

Agree.

The image itself start at L34 until L42: https://github.com/tegila/bitcoin/blob/master/Dockerfile#L34

I just removed the alpine image from the shipped image in favor of scratch image.
So now we removed some need to trust on alpine team. Alpine still in the building process, and we also have to trust on toolchain suppliers.

The shipped image now has 24Mb and only bitcoin binaries inside.

Could you please add milestone/52 to this pull request as this also solve the problem of building bitcoin against musl instead of glibc.

There is also a good point by @sipa that maybe the release process itself can(?) be used to create the docker image.

You could push the image to docker hub but it's a hard task and not how it's supposed to be done.
Docker hub can give a free account to build public projects. So you can use them as part of your CI/CD basically putting the Dockerfile on the / of your project and setting up the integration with github on docker hub settings.

[maflcko]

Could you please add milestone/52 to this pull request as this also solve the problem of building bitcoin against musl instead of glibc.

Features don't have milestones in our project. Stuff is merged when it is reviewed and ready.

[maflcko]

For reference @dongcarl used this configure to build from depends on alpine: #18110 (comment) and it resulted in ~10Mb. (Edit: Though my preference would still be to just wget the release binary and be done with it).

So you can use them as part of your CI/CD basically putting the Dockerfile on the / of your project and setting up the integration with github on docker hub settings

Again, we are not using CI/CD as part of the release process and this also not going to change.

I presume it is impossible to create a docker image (binary file) and distribute that via our own website? The only way to distribute images is via docker hub?

[tegila]

For reference @dongcarl used this configure to build from depends on alpine: #18110 (comment) and it resulted in ~10Mb.

Thank you for referencing @dongcarl as he provided the solution on how to create the static binary and the configure on my Dockerfile was based on his comment on issue#18110.

As I used the same build as him, probably I got the same results.
My image is bigger than the bitcoind binary itself as there are other binaries being deployed to /usr/local and man pages, share, etc. I could remove them if necessary as well as many other improvements can be done after some agreement on the topic.

in #18110 (comment) @laanwj wrote:

I think having a static bitcoind that runs directly on top of the kernel and doesn't depend on any libraries that happen to be on the system would be a good aim, at least for the release binaries. It removes a aource of variability and possible bugs. It would allow getting rid of all the glibc compatibility hacks.

DONE.

Again, we are not using CI/CD as part of the release process and this also not going to change.

I hope you guys change your mind about that as docker hub can maintain all the versions at same time and if you don't want to deploy the lastest version automatically you can do it by click, we can even run the tests and see the results on the build system of docker hub.

I presume it is impossible to create a docker image (binary file) and distribute that via our own website? The only way to distribute images is via docker hub?

It's not impossible but its counter-intuitive as the system is design to assit the developer and not give him more repetitive work.
BTW you can build the image locally and use > docker push image:version
One can also > docker save busybox > busybox.tar but again not close to the best choice.

To keep the confidence and trust you guys talk, docker hub has versioned deployment, and one can write like bitcoin:trusted_version or any other branch and leave the last commit on master to bitcon:latest and write on the readme.md that :latest isn't trustable, and use at your own risk.

[tegila]

Just for reference, when I search for bitcoin on the marketplace today, the first result is something which stands for bitcoin-sv, 50k downloads.

After some scrolling I found something relevant with 8.9M downloads, which sounds like real bitcoin, build from unknown provider, kylemanna. Ruimarino also has 1M downloads, but I didn't want trust.

A big project like bitcoin should have it's own repository, connected to the official github repository, build from scratch, following the tags and branches on github, with all images from all times available with one click.

The project can run its tests during the build process and show the result as bonus, because we can browse and understand better how the tests are made, and if something is wrong with the binary.

The search result has more than 10 pages, with many altcoins wrong named bitcoin, which lead to confusion and distrust and not a single build from this repository.

For me it's a major risk to keep the things in this way.
When someone want deploy a cloud application after finish the amatour phase, goes direct to docker hub, and there's only two bad choices:
1- trust one unknown
2- find and build a good image

As you can see by yourself, more than 10M people had to download and run this images that can easily contain a key stealer and the providers can easily hide the current Dockerfile for the image present on the docker hub. They can even fake which source was used to deploy.

So, please, consider you're putting many users at risk when you choose to keep the things as it's.

My current image was designed using milestone/52, musl and removing totally the external dependencies. But other images can contain alpine, or ubuntu, using glibc etc.
This will give the user the power to choose, and will remove the need to trust. Mostly because after you put the build process there we can check the tests and see how the image was build and which errors/warnings.

All this without need to change the current deployment for hackers and contributers.
For me it's a win-win.
Hope you all understand, and I'll be here to answer/help and also on CT.

[ghost]

Just for reference, when I search for bitcoin on the marketplace today, the first result is something which stands for bitcoin-sv, 50k downloads.

SMH

After some scrolling I found something relevant with 8.9M downloads, which sounds like real bitcoin, build from unknown provider, kylemanna. Ruimarino also has 1M downloads, but I didn't want trust.

Third link in search results also looks legit and related to Bitcoin Core although I am not sure who maintains it.

https://hub.docker.com/r/bitcoindevelopernetwork/bitcoind-regtest

[fanquake]

Concept ~0. Building Bitcoin Core on Alpine (statically or not) has been done by many people for a long time. I'm not sure how wrapping that up in a Docker container and putting it into this repository, and/or the Docker marketplace is a solution to #18110.

What would be a solution, would be adding an optional musl build to depends, so that users on Linux platforms could opt into compiling against that when building using depends, or so that it can be used during release builds.

[fanquake]

Thanks for you changes here. However I'm going to close this in favour of #23203, which is more like an approach we would take to work towards solving #18110.