Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

data persistence is missing in docs #3

Closed
max-tet opened this issue Jul 23, 2022 · 13 comments
Closed

data persistence is missing in docs #3

max-tet opened this issue Jul 23, 2022 · 13 comments
Assignees
Labels
docker help wanted Extra attention is needed

Comments

@max-tet
Copy link

max-tet commented Jul 23, 2022

Hi.
I was wondering how to mount a directory such that all relevant data is persisted outside of the container itself. The docker commands in the readme don't do that.
Would be great if you could add that information. Otherwise, all data would be lost if the container is removed.
Thanks!

@JulianPrieber
Copy link
Member

This is not something we really support or is something we test.

You can mount a directory with:
--volume /docker/data/www:/htdocs \

Again, not something we test or officially support.
I'm not sure if this will work without modifying the dockerfile.

Hope this helps 👍

@max-tet
Copy link
Author

max-tet commented Jul 31, 2022

Thanks for the hint. I tried to get it to work, but unsuccessfully. Mounting /htdocs is too broad, because the whole directory then is empty and the web interface just shows an empty directory listing from nginx.

I tried to change the location of the database with -e DB_DATABASE=/data/db.sqlite and mount /data but it seems like the database must be initialized. If the file does not exist, the application does not work.

I would reccomend to create the database file on application startup if it does not exist yet and apply any migrations that might be needed.

Also, are there other directories that should be mounted? Can a user upload files that are stored somewhere on the filesystem?

In general, when publishing a docker-based application that is not explicitely meant to be stateless, it is really best practice to offer a method to persist data. Usually documenting which directories should be mounted is sufficient.

@JulianPrieber
Copy link
Member

Our docker image is just an Apache web server running the web version of LittleLink Custom.
For that reason, our two versions don't differ for the simple reason of keeping everything manageable to maintain.

Everything is stored in the htdocs folder in the docker container, including the database.

When creating the docker, I essentially move everything from here into the htdocs folder and then do some permission management for the web server.

COPY littlelink-custom /htdocs
RUN chown -R apache:apache /htdocs
RUN find /htdocs -type d -print0 | xargs -0 chmod 0755
RUN find /htdocs -type f -print0 | xargs -0 chmod 0644 

If you want storage independent of the docker, you'd have to move files into this folder again and probably have to redo the file permission, then everything should work as normal.

@max-tet
Copy link
Author

max-tet commented Aug 1, 2022

Why don't you build the docker image directly from the littlelink-custom repo? Just put the dockerfile there. You wouldn't have the problem of maintaining two versions and could have a pipeline that builds a new image with every release.

@JulianPrieber
Copy link
Member

Why don't you build the docker image directly from the littlelink-custom repo? Just put the dockerfile there. You wouldn't have the problem of maintaining two versions and could have a pipeline that builds a new image with every release.

We do build the docker image from the littlelink-custom repo. Everything is automated when I release a new version.
It is actually easier to have the two versions separate. This repository only contains docker files, our main repository contains the application itself.

@max-tet
Copy link
Author

max-tet commented Aug 1, 2022

I'm not sure I understand why that is easier, but as long as it works for you that's great.

But still, I think for a docker release, it is really important that there is an easy way for data persistence that is well documented. Noone wants their data to be lost when the container is removed. A missing piece in an otherwise very nice project.

@bonelifer
Copy link

bonelifer commented Aug 11, 2022

I agree. Just reorganizing my instance of dockers. And didn't realize there wasn't persistence on this container. Lost everything. :(

@JulianPrieber
Copy link
Member

I think I worked out something that works and is easy to set up:

  1. Create volume:
    docker volume create llc

  2. Running the Docker image normally while mounting the volume:
    docker run --detach --name littlelink-custom --publish 80:80 --publish 443:443 --restart unless-stopped --mount source=llc,target=/htdocs julianprieber/littlelink-custom

Something like this


This should move the important files to the volume for persistent data.

If you could please check that this is performing as expected, @max-tet, I'll add it to the documentation.

@max-tet
Copy link
Author

max-tet commented Aug 12, 2022

Thanks for your effort. It is only half a solution though, sorry. When inspecting the volume llc, you can see that basically the whole application is copied there, not just the user data. You can use e.g. docker run -v llc:/srv -it --rm -p 80:80 filebrowser/filebrowser to have a look.

This will most probably become a problem when upgrading the app. Since lots of parts of the app itself are now persisted in the volume, they will not be upgraded when a new docker image is used.

The correct way to implement data persistence would be to:

  1. change the paths of all user data files (sqlite db, uploaded media, etc.) such they are inside their own directory, e.g. /data and not mixed between application files.
  2. Make sure that the app starts correctly if the data dir is initially empty. I.e. create and initialize the database if it does not exist.
  3. Mount a dir or volume only for the user data directory.

@JulianPrieber
Copy link
Member

Thanks for the quick response and help on this! I will look into the method you described and get a third opinion on this.

Granted, this is probably not the best solution.

Copying the whole application was actually intentional. As it is, right now this is the easiest to implement solution I can think of for the user. Copying every directory and file that contains user data that should be persisted is much easier this way. If I'd copy individual folders, I would have to future prove the thing, so it would still include anything that might get added later on.

You might be right on the upgrade part, I will do some testing and see what works best.

@LeoColman
Copy link

I don't think there's an issue with copying the whole application.

We can have multiple volumes, for example one for data and one for /var/www.

In my docker-compose setup I'd do something like:

services:
  littlelink-custom:
    image: julianprieber/littlelink-custom
    volumes:
      - data:/var/data
      - app:/var/www/app
      - database:/var/www/app/data

However I don't know which paths I should pay attention to. Can you enlist what the project uses as folders and files, @JulianPrieber ?

@LeoColman
Copy link

In the 'copying the whole application': Nextcloud does that as well, it puts everything in /var/www/html and you can still add volumes to it.

More on that in
https://hub.docker.com/_/nextcloud >> Persistent data

docker run -d \
	-v nextcloud:/var/www/html \
	-v apps:/var/www/html/custom_apps \
	-v config:/var/www/html/config \
	-v data:/var/www/html/data \
	-v theme:/var/www/html/themes/<YOUR_CUSTOM_THEME> \
	nextcloud

@JulianPrieber
Copy link
Member

JulianPrieber commented Aug 16, 2022

However I don't know which paths I should pay attention to. Can you enlist what the project uses as folders and files, [...] ?

@LeoColman Everything is saved in the /htdocs folder including the database.

JulianPrieber added a commit that referenced this issue Sep 6, 2022
@JulianPrieber JulianPrieber unpinned this issue Sep 6, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
docker help wanted Extra attention is needed
Projects
None yet
Development

No branches or pull requests

4 participants