Feat: Tracim/Tracim docker support for non-AMD64 platform with Debian Bullseye

[inkhey]

Feature description and goals

Currently Tracim and Tracim docker is only working/tested on AMD64 platform.
To make it functional on other plateform some change should be done, as some dependencies doesn't exist on other platform.

All these point are closely related for compatibility, so it make not much sense to split them.

Which Plateforms to support ?

Docker support multiple plateform:
https://github.com/docker-library/official-images#architectures-other-than-amd64

AMD64 is the most common and the default one for intel/amd processor, this architecture is already supported by tracim. It covered almost all server.

ARM is the challenger, there is many version:

Official debian version support armv5 (armel), arm v7(armhf), armv8 (arm64)

• support for armv5 will target: some plug computers like sheevaplugs.
• support for armv6 will target: raspberry pi zero, raspberry pi 1 and some other single board computers. Note that raspbian support armv6 but not debian, so the easiest to support them may be to create a armv5 image.
• support for armv7 will target: raspberry pi 2, 3 and some others single board computers.
• support for armv8 will target: some servers, raspberry pi 4, modern sbc like helios64, etc…

i386 is the 32bit version on AMD64, all current x86 processor are 64bit(AMD64), it's not very useful to support this one.

Others architectures (ppc64le,s390x,mips64le) are exotics enough to not be really useful to check now.

One other architecture that may be useful to support later is riscv64.

I do suggest to focus on armv8 and (maybe) armv7.

Current issues with arm build:

• VTK (for preview_generator) is not provided for arm in pypi: https://pypi.org/project/vtk/#files, workaround: removing vtk, building vtk for specific arch is a possible solution, check aarch64 support  KitwareMedical/VTKPythonPackage#42.
• cairosvg builder for preview rely on cffi, with support only armv8 (aarch64) as arm arch https://pypi.org/project/cffi/#files -> replaced by inkscape
• new tracim docker image is both stretch based and install gocryptfs, which is not supported for arm64 in stretch : https://packages.debian.org/fr/stretch/gocryptfs : not fixed but workaround with not using the feature.
• pushpin repository doesn't handle arm architecture, we do need to use debian package for this, see Bug: pushpin sometimes does not seem to start correctly with the docker image #3372 Fixed by using debian bullseye repository
• etc.

Prerequisites

These 2 PR should be merged:

• https://github.com/algoo/preview-generator/tree/fix/xxx__make_some_builders_not_required
• Feat/4784 update dependencies #4785 replaced by Bug: installation of Tracim does not work on Debian 11 (Bullseye) #4874

Implemented solution:

#5081

• use new bullseye docker images and new python dependencies that support python 3.6-3.9.
• create a new docker image based on the non-encrypted docker that remove vtk as dependencies.
• improve the setup_default_backend.sh for minimal dependencies case.
• add documentation about building arm64 docker image.

deprecated ticket informations:

New Working Proof of Concept (04/07/2021) 🎉:

After the merge of https://github.com/tracim/tracim/issues/4751. I do success to generate a "minimal" tracim image for arm64 (didn't try 32 bit arm build). I did also a rebase the code of #4785.

source code is here: https://github.com/tracim/tracim/tree/POC/ARM_docker
container build for both arm64 and amd64 are available here (for now): https://github.com/inkhey/tracim/pkgs/container/tracim (old version !)

What's work:

• building arm64 and amd64 image with minimal dependencies (mostly removing preview builder stuff).
• building amd64 with full dependencies.
• running this image on real hardware including:
  • Fonctionnal frontend.
  • Fonctionnal API.
  • Fonctionnal preview for png/jpeg/gif.
  • TLM with pushpin.🎉
• New feature due to bulleyes: out of the box HEIC file preview support.
• New feature if we use the latest colalbora version: csv file support , see also Feat: Tracim/Tracim docker support for non-AMD64 platform with Debian Bullseye #3985 (comment)
  note:
• for the amd64, the image was tested with docker (debian sid and ubuntu 18.04) without reverse-proxy.
• for the arm64, the image was tested on a Helios64 (on debian buster) with podman in rootless mode with nginx as reverse proxy + https, http/2 support (*)

What is not included/tested:

• docker test. Tested.
• support for encryption with gocryptfs Tested on amd64
• building with full preview-generator dependencies (should work for amd64 only as vtk and probably others are not build for arm64). Tested on amd64. should find a solution for arm64
• webdav. partially tested.
• collabora. partially tested. tested on amd64 only with docker test process.
• non-sqlite database
• s3 storage
• tracimcli
• caldav. partially tested: only checked caldavzap.
• probably other complex features.
• running backend test on debian bulleyes.

What doesn't work for now :

• caldav (probably due to dependencies update) fixed
• RAW preview is broken dues to missing ufraw_batch in bulleyes. : Repair Raw Preview on Imagemagick builder: ufraw-batch does no longer exists in Debian algoo/preview-generator#154, Feat: Better Raw preview builder with Darktable-cli or rawpy ? algoo/preview-generator#249, see also this comment: Feat: Tracim/Tracim docker support for non-AMD64 platform with Debian Bullseye #3985 (comment)
• SVG preview is broken dues to the change in inkscape 1.0 command line.: Bug: Inkscape builder broken with inkscape 1.0+ algoo/preview-generator#263, see also this comment: Feat: Tracim/Tracim docker support for non-AMD64 platform with Debian Bullseye #3985 (comment)
• running the arm64 image with qemu (i do get weird issue with apache related to https://serverfault.com/questions/1014620/permission-denied-ah02478-failed-to-create-proxy-mutex and get issue then with uwsgi…) . It work great on real arm64 hardware.

What has changed:

• code from develop branch with unfinished features ⚠️ .
• Most python dependencies are updated to last version due to Feat/4784 update dependencies #4785 . Some package like lxml get binary package for arm64 only in last release. It's was easier to update most things than trying to make thing work.
• Docker image use Bullseye (future stable !) which use python 3.9. To get latest python binary package available for arm64 and to get pushpin > 0.31 (there is no official repository for arm64, so we need to use debian package).
• Dockerfile was heavily modified:
  • optional options to disable installation of packages + debug mode.
  • apt package list outside of dockerfile
  • option to install tracim from local dir install/compressed archive.
  • remove the old docker to keep only one.
  • fix pushpin support with this new package (override default config).
  • split front and backend build as frontend build
  • use buildx by default in documentation of docker build
  • some cleanup in each stage of the build.
  • document how to build arm64 and amd64 minimal build using buildx and qemu.
  • use node 12 with node 12 build docker: no need to download node/npm/yarn.

TODO:

• make preview_generator able to handle optional previews builder, see poc here: https://github.com/algoo/preview-generator/tree/fix/xxx__make_some_builders_not_required
• update python dependencies: see Feat/4784 update dependencies #4785
• update docker image to bulleyes
• update docker to allow full/minimal install with or without all the dependencies for preview-generator
• update bash script (setup_default_backend) to allow optionals dependencies for preview-generator
• fix other dependencies issue with tracim by either updating dependencies or make them optionals.
  • fix RAW support: Repair Raw Preview on Imagemagick builder: ufraw-batch does no longer exists in Debian algoo/preview-generator#154, Feat: Better Raw preview builder with Darktable-cli or rawpy ? algoo/preview-generator#249, see also this comment: Feat: Tracim/Tracim docker support for non-AMD64 platform with Debian Bullseye #3985 (comment)
  • fix svg support: Bug: Inkscape builder broken with inkscape 1.0+ algoo/preview-generator#263

Deprecated previous poc comment:

I just tried a POC here:
(https://github.com/tracim/tracim/tree/poc/ARM_build
but failed to success now as there are some complex issue related to dependencies version of requirements.txt that cannot be solved properly on other platform. As this file is handled by pip-tools and there is currently not preview-generator with optional dependency support, it's complicated to properly fix dependencies issues. Fixing preview-generator optional dependencies support and create and official release with it will make finding the good dependencies version to support arm much easier.

Another point which may help too, is upgrading docker from stretch to buster.

To test this poc, check the documentation:
https://github.com/tracim/tracim/blob/593f74c1e5100495b6e0dd23a4771f5d2bcd7735/tools_docker/README.md#arm-build-experimental)

[inkhey]

This is related to previous issue related to arm support:
#1458
#3727
#3729

[inkhey]

I updated the issue with my new working arm64 build ! 🎉

PS: here a working config for using nginx as reverse-proxy (ssl is defined somewhere else), the proxy buffering is disabled for making tlm to work as expected.

upstream tracim_local_apache{
    server 127.0.0.1:8080;
}
server {
  listen 443 http2 ssl;
  listen [::]:443 http2 ssl;
  server_name tracim.domainname;
  include common.conf;
  location / {
    client_max_body_size 300M;
    include proxy_params;
    proxy_pass http://tracim_local_apache;
    proxy_http_version 1.1;
    proxy_set_header Connection "";
    proxy_buffering off;
  }
}

[inkhey]

Side notes when trying to generate a full dependency docker using docker code for manual testing #4785 ):

• RAW file preview is fully broken due to the fact that ufraw-batch is removed from latest debian (unmaintained library). The good way to solve this could be either building imagemagick with libraw support or add a darktable builder. i do suggest to go to the darktable road as the cli is easy to use and there is no need for recompilation. Note: this also avoid creating big png, if we do use for direct preview (without intermediate png to jpeg builder). Feat: Better Raw preview builder with Darktable-cli or rawpy ? algoo/preview-generator#249. Another potential advantage of doing so, is better preview generation perf for raw files.
• using slim image of bulleyes broke libreoffice install due to issue with doc files during jre install.
• i have fixed few minors issue testing this.

[inkhey]

Other side notes:

• HEIC preview work out of the box !!
• SVG preview using inkscape fail (command line has changed ?)

[inkhey]

Others side notes:

• Collabora work with latest collabora version. It work for csv too but ui is not perfect (the ui propose to convert to odt but it doesn't work).
• caldav work (from web ui)
• Webdav is working
• encryption work (amd64 full version)

[grignards]

With the changes from #4874 a build for arm64 stops as the python package vtk==9.0.2 is not available.

[inkhey]

With change from #4874, the change required to handle arm64 look like:

• find some way to make some PG dependencies optional like VTK which is not builded for linux_arm64.
• clarify how to deal with the encryption feature, it has been known to not work on arm64 (make only one of the image compatible, create another image, other ?)
• reintroduce documentation about how to build arm64 image.

[grignards]

My opinion on your points:

• setup_default_backend.sh should get an argument to only install minimal dependencies
• discuss with @PhilippeAccorsi which of a dedicated Dockerfile or an option to the current one is preferred (the option only impacts one line of the Dockerfile)
• ok for the documentation, don't forget to add the link to the official documentation for installing buildx.

[inkhey]

I can confirm that removing vtk is enough to make arm64 docker image work.

[gbonaspetti]

Done