Why so many volumes?

[pwalkr]

Hello! Gramps is a fantastic application. I have been self-hosting an HTML export of my tree for years and I'm excited to see an official hosted solution with docker. Thank you!

Is it safe/possible to use fewer volume mounts? Per docker-compose.yml there's 8 volumes, 6 of which are subdirectories of /app.

Would it be bad to simply mount a gramps_data on /app?

Is mounting /tmp necessary?

I'd love for a single volume to backup/restore, relocate as needed. If /app doesn't contain anything that should be discarded on restart, /tmp isn't necessary, and grampsdb is relocatable based on ENV then it would be possible.

*I see /tmp was added to share with celery - I've no experience with celery, but it seems like redis should be the only necessary link between containers. Is that resolvable?

[DavidMStraub]

Hi! Thanks for the nice words.

You can use less volumes if you like - the main reason for splitting them was actually that not all of them are equally relevant for a backup.

Would it be bad to simply mount a gramps_data on /app?

Yep, that would be bad, because /app contains the source code for Web API: https://github.com/gramps-project/gramps-web-api/blob/master/Dockerfile#L67

Is mounting /tmp necessary?

It's necessary, yes, because that's how the Web API container and the Celery container share temporary files (e.g., the celery container creates an export and the flask container serves it). Using redis for that doesn't make sense - imagine generating a media archive ZIP, which can be GB in size.

I'd love for a single volume to backup/restore, relocate as needed. If /app doesn't contain anything that should be discarded on restart, /tmp isn't necessary, and grampsdb is relocatable based on ENV then it would be possible.

It's still possible. Nothing stops you from changing the config as well, see https://www.grampsweb.org/Configuration/. Then you can move media, search index, and Gramps database into the same folder. However it might require you to create some subdirectories which are not present in the image yet, so not 100% sure it's possible without creating your own image or modifying the entrypoint.

If you want to move the Gramps database away from /root/.gramps/grampsdb, you can probably do so by setting the GRAMPSHOME env variable.

[pwalkr]

Ah, there is a lot in /app besides the user-specific data. Thank you for clarifying. I will look at relocating the folders so they can be kept together without clobbering everything else. Maybe just a subvolume type mount.

Have you considered moving gunicorn and celery into the same container with init/tini? It would eliminate the tmp volume, but it may be useful to keep them separate for scaling.

[DavidMStraub]

Have you considered moving gunicorn and celery into the same container with init/tini? It would eliminate the tmp volume, but it may be useful to keep them separate for scaling.

I haven't, and think it's cleaner to keep them separate.

[pwalkr]

After gramps-project/gramps-web-api#538, my docker-compose.yml is down to two mounts, and the whole instance seems to have survived simply mving the folders into a single /data dir with the associated env being set (existing instance. I have not tried on a net-new install). Copied below to have all the path keys in one place.

Not included: /app/cache - I believe this would be relocated via GRAMPSWEB_REPORT_DIR and/or GRAMPSWEB_EXPORT_DIR (gramps-web-api/gramps_webapi/config.py). But I plan to run this remotely and Gramps Web Sync to local for development and any reporting so I don't think I need it. Please correct me if wrong.

Bonus: this works as non-root. After chown -R 1000:100 data/ it started back up with user: '1000' (again, likely to have issues on a net-new installation)

version: "3.7"

services:
  grampsweb: &grampsweb
    image: grampsweb-no-app-secret  # Locally built, uses https://github.com/gramps-project/gramps-web-api/pull/538
    user: '1000'
    restart: always
    ports:
      - "80:5000"  # host:docker
    environment:
      GRAMPSWEB_TREE: "Gramps Web"  # will create a new tree if not exists
      GRAMPSWEB_CELERY_CONFIG__broker_url: "redis://grampsweb_redis:6379/0"
      GRAMPSWEB_CELERY_CONFIG__result_backend: "redis://grampsweb_redis:6379/0"
      GRAMPSWEB_RATELIMIT_STORAGE_URI: redis://grampsweb_redis:6379/1
      GRAMPSHOME: /data
      GRAMPSWEB_SEARCH_INDEX_DB_URI: sqlite:////data/search_index.db
      GRAMPSWEB_MEDIA_BASE_DIR: /data/media
      GRAMPSWEB_USER_DB_URI: sqlite:////data/users.sqlite
      GRAMPSWEB_SECRET_KEY: '*****************'
      GRAMPSWEB_THUMBNAIL_CACHE_CONFIG__CACHE_DIR: /data/thumbnail_cache
    depends_on:
      - grampsweb_redis
    volumes:
      - /home/user/grampsweb/data:/data
      - /home/user/grampsweb/tmp:/tmp

  grampsweb_celery:
    <<: *grampsweb  # YAML merge key copying the entire grampsweb service config
    ports: []
    container_name: grampsweb_celery
    depends_on:
      - grampsweb_redis
    command: celery -A gramps_webapi.celery worker --loglevel=INFO

  grampsweb_redis:
    image: docker.io/library/redis:7.2.4-alpine
    container_name: grampsweb_redis
    restart: always

[DavidMStraub]

EXPORT_DIR is needed for Gramps Web Sync to work as it exports the tree as XML to detect changes.

[jondycz]

Doesn't work. The GRAMPSHOME directory contains other "preinstalled" directories that are lost when the directory is changed through the environment variable (relation tree for instance) and running as non-root will probably cause other random issues. It's a real shame :(

truenas/apps#2544

[DavidMStraub]

I'm not ashamed of anything.

[letsjustfixit]

It seems to me that all hell breaks loose right at the start (installing 3GB of dependencies right away)
https://i.imgur.com/QgKu4D3.png
The chatbot stuff is over an additional gigabyte so it would probably worth it to create a "lite" version without it.

[letsjustfixit]

As a oneoff experiment I've removed the things I currently do not use:

• chat
• tesseract
• postgres client
  and changed debian:bookworm (117MB) to debian:bookworm-slim (74.8MB)
  Result: 2.55GB
  --
  I'll look into the feasability of an alpine based image as well. The main issue above was retaining the python packages so that runtime dependencies don't break.

[jondycz]

The chatbot stuff is over an additional gigabyte so it would probably worth it to create a "lite" version without it.

Yes, that would certainly make sense - but I don't want to maintain even more stuff. Someone could set up custom images of different flavours.

Would it be really that hard to add a new build to the pipeline that doesn't include the AI stuff and publish new releases under a "-light" tag? I'd like to deploy this on TrueNAS and that has a 10ish minute timeout for docker apps to start up. This takes about 30 minutes for me to deploy even on a 1Gbit network (due to HDD bottleneck)

[DavidMStraub]

I don't want to commit to something I will have to personally maintain perpetually. It would also increase CI time, which is already > 1 hour now. I also don't get the downtime argument, the pull can be done while the container is running.

[jondycz]

It's not a downtime thing. The issue is that after a certain period (say 10 minutes) the container gets flagged as unhealthy and is killed.

It's strange to require almost 3GB download because of an optional feature most people won't use, or at least would offload to an open-ai compatible API (be it third party or their selfhosted instance running elsewhere)

[DavidMStraub]

Look, I get the point of the discussion, and I have made the choice for myself that I do not want to take the additional maintenance burden of offering even more image flavors in addition to

• dmstraub/gramps-webapi
• ghcr.io/gramps-project/grampsweb
• ghcr.io/gramps-project/grampsjs

each in two versions (stable and develop) and for 3 architectures, that's 3*2*3=18 different images.

Please accept this choice.

There is absolutely nothing stopping you, or anyone else, from setting up a fork with a custom build pipeline and provide light images to the community.