Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Docker container support for Raspberry Pi #109

Closed
tjaadvd opened this issue May 16, 2019 · 65 comments
Closed

Docker container support for Raspberry Pi #109

tjaadvd opened this issue May 16, 2019 · 65 comments
Assignees
Labels
Milestone

Comments

@tjaadvd
Copy link

@tjaadvd tjaadvd commented May 16, 2019

Hi guys,

I love where this project is going and I'm eager to try it on my Raspberry Pi 3b+. Are you planning on making the docker container suitable for ARM devices?

Greetings,
Thomas

@lastzero

This comment has been minimized.

Copy link
Member

@lastzero lastzero commented May 16, 2019

You might have to reduce quality settings and of course you need to compile it for ARM. Since we don't own a Raspberry, some help would be appreciated.

@lastzero lastzero added the question label May 16, 2019
@Redwid

This comment has been minimized.

Copy link

@Redwid Redwid commented May 18, 2019

Your docker base architecture is x86. So it's not possible right now to run it on ARM.

@lastzero

This comment has been minimized.

Copy link
Member

@lastzero lastzero commented May 19, 2019

Your docker base architecture is x86. So it's not possible right now to run it on ARM.

@Redwid This is where pull requests come into play 💎

Sent with GitHawk

@graciousgrey graciousgrey added the idea label Oct 26, 2019
@guysoft

This comment has been minimized.

Copy link
Collaborator

@guysoft guysoft commented Dec 23, 2019

Hey guys,
Developer of CustomPiOS and maintainer from ownphotos.
It is possible to have arm32v7/arm64v8 builds on dockerhub.
I have recently implemented this for CustomPiOS, and would love to see this also in photoprism.
It requires some dark magic involving qemu and dockerhub hooks. Example here. CustomPiOS implementation here.
I would love to help write this, but I need some help understanding what is amd64 bit, and I can see your docker build is split to several images. We will need to make them arm-compatible one by one.
I vote for working on arm32v7 first because that would give all Raspbian users the ability to run photoprism. We could even make a dedicated image.

@lastzero

This comment has been minimized.

Copy link
Member

@lastzero lastzero commented Dec 23, 2019

@guysoft There are two main challenges regarding ARM:

  1. We need the TensorFlow C library which we previously had to compile from source, especially for specific ARM-based devices like the Nvidia Jetson. Not sure if / how you would be able to compile (and test) it with qemu or any other emulation, see #83 and https://dl.photoprism.org/tensorflow/nvidia-jetson/

  2. We need a RAW to JPEG converter, currently this is Darktable on Linux (alternatively Sips on OS X), see #145 and https://github.com/photoprism/photoprism/wiki/Converting-RAW-to-JPEG - Darktable however can not be compiled for arm32v7, only 64 bit on ARM (x86 also 32 bit). So we can either find a good alternative or disable RAW to JPEG conversion.

When this is solved, we can start to think about build automation.

@lastzero

This comment has been minimized.

Copy link
Member

@lastzero lastzero commented Dec 23, 2019

Ubuntu 18.04 and other build dependencies are available for ARM 32/64, so that's the easy part... had that already but didn't seem to push it.

@guysoft

This comment has been minimized.

Copy link
Collaborator

@guysoft guysoft commented Dec 24, 2019

@lastzero

  1. Looks well documented:
  1. Looks like it could build. I can give that a try (actually started).
    Repo I am looking at is at:
    https://github.com/darktable-org/darktable

Assuming I have managed to build both of these - will you lend me a hand with the Dockerfiles?

@guysoft

This comment has been minimized.

Copy link
Collaborator

@guysoft guysoft commented Dec 24, 2019

Update: the great news is that it works on ARM, the bad is only ARMv8, which is a little more tricky to setup.

/darktable/src/is_supported_platform.h:64:2: error: #error "Unfortunately we only work on the 64-bit architectures amd64, ARMv8-A and PPC64."
 #error "Unfortunately we only work on the 64-bit architectures amd64, ARMv8-A and PPC64."

So I guess I can try build an env for that but wont be. Still this possible. I can try setting up a 64bit docker in a 32bit env.

@lastzero

This comment has been minimized.

Copy link
Member

@lastzero lastzero commented Dec 24, 2019

You can run 64bit Docker on a 32bit Kernel? Didn't find a way to do that. We use TensorFlow 1.4 not 2.x... These scripts didn't work for me, ended up using Bazel... But you can try. Also it was tricky to find the right compiler with the right settings for C and C++... In fact I contributed a patch to TensorFlow because it didn't compile on ARM. Should be part of the latest releases.

@lastzero

This comment has been minimized.

Copy link
Member

@lastzero lastzero commented Dec 24, 2019

Check out the Nvidia Jetson forums regarding TensorFlow... Took me one week to compile it... And I had fans afterwards:

Hi,

Thank you for your response. I am successful in building the pip package and .SO for TF v1.13.1 on Jetson TX2 after around 3 weeks!!

Thank you

@guysoft

This comment has been minimized.

Copy link
Collaborator

@guysoft guysoft commented Dec 26, 2019

@lastzero Recent Rasbian ships a 64bit kernel in a 32bit userland. Using a chroot you can run 64bit applications. I am trying to build your darktable docker container in such a chroot.

I can build you a raspberrypi distro that has arm64 docker running on it. I think from there it just a matter of updating the Dockerfiles.

@lastzero

This comment has been minimized.

Copy link
Member

@lastzero lastzero commented Dec 26, 2019

There is a pre-built darktable package for Ubuntu 18.04... No need to build it from source

@guysoft

This comment has been minimized.

Copy link
Collaborator

@guysoft guysoft commented Dec 26, 2019

@lastzero prebuilt for aarch64?
If so ill divert my efforts to making an arm64 bit distro with docker, is that all that is needed?

@lastzero

This comment has been minimized.

Copy link
Member

@lastzero lastzero commented Dec 26, 2019

Yes, Ubuntu has one for arm64

@guysoft

This comment has been minimized.

Copy link
Collaborator

@guysoft guysoft commented Jan 5, 2020

Hey, not image yet, but updating you I have a repo that builds something. Hopefully this week if life is not too much in the way:
https://github.com/guysoft/Raspbian64

@guysoft

This comment has been minimized.

Copy link
Collaborator

@guysoft guysoft commented Jan 8, 2020

Aaand I am stuck:
https://forums.docker.com/t/running-docker-from-within-a-chroot/86883
Help would be great.

I have docker running, but it will not chroot. Not sure how to solve this yet.

@lastzero

This comment has been minimized.

Copy link
Member

@lastzero lastzero commented Jan 8, 2020

Is there anything we can change within our environment or software that would help?

@guysoft

This comment has been minimized.

Copy link
Collaborator

@guysoft guysoft commented Jan 8, 2020

@lastzero Well it would have been easier if it was possble to run it on 32bit arm. aka arm32v7. But AFAIK Darktable wont let us do that.
I am pretty close to getting this to work, so let see if we get an answer on that. Also posted here:
https://serverfault.com/questions/998150/running-docker-from-within-a-chroot

@lastzero

This comment has been minimized.

Copy link
Member

@lastzero lastzero commented Jan 8, 2020

You could search for an alternative, any command that does the same will do...

@guysoft

This comment has been minimized.

Copy link
Collaborator

@guysoft guysoft commented Jan 8, 2020

I am looking at ubuntu server as an alternative now

@guysoft

This comment has been minimized.

Copy link
Collaborator

@guysoft guysoft commented Jan 13, 2020

Ubuntu server distro looks promising, I am adding CustomPiOS support for it.
Its kinda ridiculous they ship the distro without dhcpcd, ifconfig and iwconfig installed. Fixing that.
Should have an image can flash and build docker aarch64bit images from.

@lastzero As the next part of this - what is the Dockerfile that i need to build to have a aarch64 container?
Start with the final one, and then list if there are smaller ones that it depends on.

@lastzero

This comment has been minimized.

Copy link
Member

@lastzero lastzero commented Jan 13, 2020

Our development base image for x86:

https://github.com/photoprism/photoprism/blob/develop/docker/development/Dockerfile

For deployment, we use a two stage build (copy the binary files from our development image to a new image without dev tools):

https://github.com/photoprism/photoprism/blob/develop/docker/photoprism/Dockerfile

guysoft added a commit to guysoft/CustomPiOS that referenced this issue Jan 13, 2020
@lastzero

This comment has been minimized.

Copy link
Member

@lastzero lastzero commented Jan 16, 2020

Can you check if the patch was really applied? Looks like it is missing, that's exactly what the patch is for.

@guysoft

This comment has been minimized.

Copy link
Collaborator

@guysoft guysoft commented Jan 16, 2020

@lastzero Dang, missed that I comment the patching. Fixed, building.

@lastzero

This comment has been minimized.

Copy link
Member

@lastzero lastzero commented Jan 16, 2020

@guysoft Mentioned your efforts on the Project-Status page

@guysoft

This comment has been minimized.

Copy link
Collaborator

@guysoft guysoft commented Jan 17, 2020

Success!

                                       ^
tensorflow/core/grappler/optimizers/data/map_vectorization.cc:313:66: warning: 'new_map_node' may be used uninitialized in this function [-Wmaybe-uninitialized]
     graph_utils::CopyAttribute(key, new_map_node, &prefetch_node);
                                                                  ^
tensorflow/core/grappler/optimizers/data/map_vectorization.cc:541:14: note: 'new_map_node' was declared here
     NodeDef* new_map_node;
              ^
Target //tensorflow:libtensorflow.so up-to-date:
  bazel-bin/tensorflow/libtensorflow.so
INFO: Elapsed time: 21082.844s, Critical Path: 1028.63s
INFO: 5011 processes: 5011 local.
INFO: Build completed successfully, 5158 total actions

real    351m24.113s
user    0m5.710s
sys     0m6.036s

Uploaded to:
http://unofficialpi.org/photoprisim/libtensorflow-aarch64-1.14.0.tar.gz

@lastzero

This comment has been minimized.

Copy link
Member

@lastzero lastzero commented Jan 17, 2020

Congratulations 🎊💐🎉

@negbie

This comment has been minimized.

Copy link

@negbie negbie commented Jan 17, 2020

This is awesome! Thank you all who were involved!

@guysoft

This comment has been minimized.

Copy link
Collaborator

@guysoft guysoft commented Jan 18, 2020

I have now all the binaries to build the image. Should have time to start next week.

@guysoft

This comment has been minimized.

Copy link
Collaborator

@guysoft guysoft commented Jan 18, 2020

Built, haven't tested yet:
https://hub.docker.com/repository/docker/guysoft/photoprisim

Need to set docker-compose

@guysoft

This comment has been minimized.

Copy link
Collaborator

@guysoft guysoft commented Jan 19, 2020

Ok, tested and it works and adds images.

You need to:

  1. Use a Raspberrypi 3/4 where you have set in /boot/config.txt the parameter arm_64bit=1
  2. Update docker-compose.yml to use the image: guysoft/photoprisimpi
  3. Make sure your docker compose has the setting for the image:
  photoprism:
    security_opt:
      - seccomp:unconfined

You can use UbuntuDockerPi as well to run the image, or Raspbian once its on 64bit kernel.

docker-compose file in full that you can use right now:
https://github.com/guysoft/photoprism/blob/rpi/docker/photoprism/docker-compose-aarch64.yml

docker-compose up -f docker-compose-aarch64.yml

@lastzero I guess we should talk how to set up a PR from https://github.com/guysoft/photoprism/tree/rpi
Which has all the Dockerfiles to build this.

@lastzero

This comment has been minimized.

Copy link
Member

@lastzero lastzero commented Jan 19, 2020

@guysoft Epic! Love it. Works (almost) out of the box.

This is merged although GitHub shows a strange merge error/conflict status...

Maybe we can automatically build images for aarch64 on Travis CI: https://blog.travis-ci.com/2019-10-07-multi-cpu-architecture-support

Two questions:

  1. Is it normal that I have to run docker-compose as root (sudo)? Can't connect to the Docker service otherwise. Storage directories are created in /root/Pictures because of this. I'm using your Ubuntu Server image: https://github.com/guysoft/UbuntuDockerPi

  2. Do you think it makes sense to move ARM64 related files to a separate directory or even repository? Or is it better to use our existing directories and image names with aarch64 suffixes / tags?

@guysoft

This comment has been minimized.

Copy link
Collaborator

@guysoft guysoft commented Jan 19, 2020

  1. You should be able to get it to build on docker hub, using hooks for multiatch. Check out what I did for CustomPiOS:
    https://github.com/guysoft/CustomPiOS/tree/devel/src/hooks
  2. Yes. docker-compose should either run as root, or you need to put the user in the docker group. I could add that to the disto.
  3. I'd put them where they are. Because you want it infront of your eyes when you are editing the amd64 Dockerfiles too. Definitely the same repo, because you want a new version tag to include those filea, they are part of the release no less than your other files. Its also easy to seach for them all, they all end with aarch64.
@lastzero

This comment has been minimized.

Copy link
Member

@lastzero lastzero commented Jan 19, 2020

Any reason to favor aarch64 instead of arm64? Docker Hub seems to use arm64. Refactoring the directory structure at the moment.

What Dockerfile did you use to build the actual image? Can only find the development image (production Dockerfile probably looks very similar to the one for x86)

backwardn pushed a commit to backwardn/photoprism that referenced this issue Jan 19, 2020
…#207)

* Inital work for raspberrypi arm64 docker container photoprism#109

* Add aarch64 docker container to build tensorflow

* Add tensorflow for aarch64

* Add docker compose for raspberrypi running on 64bit kernel
backwardn pushed a commit to backwardn/photoprism that referenced this issue Jan 19, 2020
…otoprism#207)

* Inital work for raspberrypi arm64 docker container photoprism#109

* Add Dockerfile.aarch64 for tensorflow

* Add aarch64 docker container to build tensorflow

* Add tensorflow build for aarch64

* Add  tensorflow, rename

* Add docker compose for raspberrypi running on 64bit kernel
lastzero added a commit that referenced this issue Jan 19, 2020
Signed-off-by: Michael Mayer <michael@liquidbytes.net>
lastzero added a commit that referenced this issue Jan 19, 2020
Signed-off-by: Michael Mayer <michael@liquidbytes.net>
lastzero added a commit that referenced this issue Jan 20, 2020
Signed-off-by: Michael Mayer <michael@liquidbytes.net>
@lastzero

This comment has been minimized.

Copy link
Member

@lastzero lastzero commented Jan 20, 2020

Put the docker-compose.yml on our download server to shorten the download url:

wget https://dl.photoprism.org/docker/arm64/docker-compose.yml
sudo docker-compose up

Using arm64 instead of aarch64 now, simply because it's shorter. Seems like aarch64 is common in GNU land whereas Apple prefers arm64.

Should probably create a manifest for our architectures with https://github.com/estesp/manifest-tool. Shall we add a ticket for this and do it later? Wouldn't consider that a high priority task. Any experience with this?

@lastzero lastzero added please-test and removed in-progress labels Jan 20, 2020
lastzero added a commit that referenced this issue Jan 20, 2020
Signed-off-by: Michael Mayer <michael@liquidbytes.net>
@guysoft

This comment has been minimized.

Copy link
Collaborator

@guysoft guysoft commented Jan 20, 2020

Any reason to favour aarch64 instead of arm64? Docker Hub seems to use arm64. Refactoring the directory structure at the moment.

Its the same thing:

AArch64 is the 64-bit state introduced in the Armv8-A architecture. The 32-bit state which is backwards compatible with Armv7-A and previous 32-bit Arm architectures is referred to as AArch32. Therefore the GNU triplet for the 64-bit ISA is aarch64. The Linux kernel community chose to call their port of the kernel to this architecture arm64 rather than aarch64, so that's where some of the arm64 usage comes from.

As far as I know the Apple backend for aarch64 was called arm64 whereas the LLVM community-developed backend was called aarch64 (as it is the canonical name for the 64-bit ISA) and later the two were merged and the backend now is called aarch64.

So aarch64 and arm64 refer to the same thing.

source

I went with aarch because that seems to be the name in most debian repos and when you download binaries.
It does not really matter unless we are consistent.
I also find it better because arm64 is too similar to amd64. aarch64 is easier to spot visually.

@guysoft

This comment has been minimized.

Copy link
Collaborator

@guysoft guysoft commented Jan 20, 2020

Should probably create a manifest for our architectures with https://github.com/estesp/manifest-tool. Shall we add a ticket for this and do it later? Wouldn't consider that a high priority task. Any experience with this?

Yes I recommend a septate ticket. This one is really long as it is. and you want people to look back and have it easier to follow.

@lastzero

This comment has been minimized.

Copy link
Member

@lastzero lastzero commented Jan 20, 2020

Perfect, I'll close this then if that's OK for you? Thank you very much for the effort you put into this!

Maybe you or someone else can quickly test this:

wget https://dl.photoprism.org/docker/arm64/docker-compose.yml
sudo docker-compose up
@guysoft

This comment has been minimized.

Copy link
Collaborator

@guysoft guysoft commented Jan 20, 2020

Close, open new issues. Including documentation on README :)

@lastzero lastzero added done and removed please-test labels Jan 20, 2020
@lastzero lastzero added this to the MVP milestone Jan 20, 2020
@lastzero lastzero closed this Jan 20, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
6 participants
You can’t perform that action at this time.