Docker image cannot be used with SQLite in a mounted volume

[schlamar]

Data

• Shiori version: latest docker image (1.5.3)
• Database Engine: SQLite
• Operating system: N/A
• CLI/Web interface/Web Extension: N/A

Describe the bug / actual behavior

If you run the Docker image with /shiori as mounted volume the shiori process cannot write the database. It crashes with "out of memory" (see stacktrace below).

Expected behavior

Shiori with SQLite can be used with a mounted volume (running without mounted volume doesn't make sense for production deployment).

To Reproduce

Create a compose.yaml

services:
  shiori:
    container_name: shiori
    image: "ghcr.io/go-shiori/shiori"
    restart: unless-stopped
    volumes:
      - ./data:/shiori  
      - /etc/localtime:/etc/localtime:ro
    ports:
      - 8080:8080

Running docker compose up results in:

shiori  | panic: unable to open database file: out of memory (14)
shiori  |
shiori  | goroutine 1 [running]:
shiori  | github.com/jmoiron/sqlx.MustConnect(...)
shiori  |       /home/runner/go/pkg/mod/github.com/jmoiron/sqlx@v1.3.5/sqlx.go:654
shiori  | github.com/go-shiori/shiori/internal/database.OpenSQLiteDatabase({0xc0000ca438, 0x15fc350})
shiori  |       /home/runner/work/shiori/shiori/internal/database/sqlite.go:27 +0xaa
shiori  | github.com/go-shiori/shiori/internal/cmd.openSQLiteDatabase()
shiori  |       /home/runner/work/shiori/shiori/internal/cmd/root.go:117 +0x6a
shiori  | github.com/go-shiori/shiori/internal/cmd.openDatabase()
shiori  |       /home/runner/work/shiori/shiori/internal/cmd/root.go:111 +0x79
shiori  | github.com/go-shiori/shiori/internal/cmd.preRunRootHandler(0xc00021cc80, {0xbbd9a8, 0x0, 0x0})
shiori  |       /home/runner/work/shiori/shiori/internal/cmd/root.go:64 +0x116
shiori  | github.com/spf13/cobra.(*Command).execute(0xc00021cc80, {0x15fbd98, 0x0, 0x0})
shiori  |       /home/runner/go/pkg/mod/github.com/spf13/cobra@v1.4.0/command.go:840 +0x50a
shiori  | github.com/spf13/cobra.(*Command).ExecuteC(0xc0001a3400)
shiori  |       /home/runner/go/pkg/mod/github.com/spf13/cobra@v1.4.0/command.go:974 +0x3bc
shiori  | github.com/spf13/cobra.(*Command).Execute(...)
shiori  |       /home/runner/go/pkg/mod/github.com/spf13/cobra@v1.4.0/command.go:902
shiori  | main.main()
shiori  |       /home/runner/work/shiori/shiori/main.go:19 +0x1e

Notes

This is probably a permission issue. I don't think you should create an extra user in a Docker container to run an application in default use cases (disclaimer: no Docker expert here, I'm just guessing). Just run mkdir /shiori instead of addgroup/adduser probably fixes this issue.

[schlamar]

Next issue on manual build, there is no alpine 3.16 on ghcr.io registry:

failed to solve: rpc error: code = Unknown desc = failed to solve with frontend dockerfile.v0: failed to create LLB definition: ghcr.io/ghcri/alpine:3.16: not found

Probably you should use the official Docker hub here.

[schlamar]

This Dockerfile worked for me

# build stage
FROM golang:1.19-alpine3.16 AS builder
WORKDIR /src
COPY . .
RUN go build -ldflags '-s -w'

# server image

FROM alpine:3.16
LABEL org.opencontainers.image.source https://github.com/go-shiori/shiori
COPY --from=builder /src/shiori /usr/bin/
RUN mkdir /shiori
WORKDIR /shiori
EXPOSE 8080
ENV SHIORI_DIR /shiori/
ENTRYPOINT ["/usr/bin/shiori"]
CMD ["serve"]

[schlamar]

A workaround is to create the "data" directory before running docker. It is only broken if the data directory is created by docker / the container.

The documented docker run from Usage.md works because there is no subdirectory involved.

[fmartingr]

Hey @schlamar, have you tried this with the dev tag? It works correctly on my machine (though is a Mac), but it automatically migrates the database and creates the appropriate folders:

docker run -v "$(pwd)/data:/shiori" -p 8080:8080 ghcr.io/go-shiori/shiori:dev

Though I tried with latest and it works as well in my case (and I have it setup on a server with a volume too, have been working for a while now). Which host operating system are you using?

[therysin]

Hey @schlamar, have you tried this with the dev tag? It works correctly on my machine (though is a Mac), but it automatically migrates the database and creates the appropriate folders:

docker run -v "$(pwd)/data:/shiori" -p 8080:8080 ghcr.io/go-shiori/shiori:dev

Though I tried with latest and it works as well in my case (and I have it setup on a server with a volume too, have been working for a while now). Which host operating system are you using?

Also having the same issue with both the latest and dev builds. I’m on Ubuntu 22.04.1 arm (Oracle Cloud).

Seemed like an awesome project but the container won’t start unfortunately (same out of memory error ).

I should also note that I’m using docker-compose(same file as OP to test), and it’s running behind traefik.

If the mounted volume lines are commented or removed from the compose file, shiori works fine (without data persistence).

[schlamar]

Dev fails, too:

$ sudo docker run -v "$(pwd)/data:/shiori" -p 8080:8080 ghcr.io/go-shiori/shiori:dev
Unable to find image 'ghcr.io/go-shiori/shiori:dev' locally
dev: Pulling from go-shiori/shiori
...
Status: Downloaded newer image for ghcr.io/go-shiori/shiori:dev
Failed to open database: unable to open database file: out of memory (14)

Running Arch Linux with current docker release:

$ docker --version
Docker version 20.10.22, build 3a2c30b63a

[Denow]

I have the same problem with Ubuntu 20.04.5 LTS (x86).

$ docker --version
Docker version 20.10.22, build 3a2c30b

[fmartingr]

I have the same problem with Ubuntu 20.04.5 LTS (x86).

$ docker --version
Docker version 20.10.22, build 3a2c30b

Are you running ubuntu over oracle cloud as well? Do you have the exact same error as @schlamar ?

[Denow]

No, I am running Ubuntu in a vm on an onprem esxi host. The error I am receiving is the same as @schlamar:

Failed to open database: unable to open database file: out of memory (14)

The container keeps restarting after this error message and the data dir is empty so it does look like a permission issue.

[Denow]

Nevermind I was mounting the wrong volume...

[therysin]

Nevermind I was mounting the wrong volume...

Can you expand further on what was wrong/ what volume you mounted to fix?

[Denow]

I was using the following tutorial for deploying shiori: https://noted.lol/tutorial-setting-up-shiori.
The docker-compose file in this tutorial changes the shiori data dir with an environment variable to SHIORI_DIR=/data and creates a volume for /data. Using the same docker-compose file gave me the Failed to open database: unable to open database file: out of memory (14) error.

version: "3"
services:
  shiori:
    image: ghcr.io/go-shiori/shiori
    container_name: shiori
    environment:
      - PUID=1000
      - PGID=1000
      - SHIORI_DIR=/data
    ports:
      - 8080:8080
    restart: unless-stopped
    volumes:
      - /data/shiori:/data

I removed the environment variable so shiori would write the db to the default location /shiori and also changed the volume path to the same location, now everything is working great.

[therysin]

I was using the following tutorial for deploying shiori: https://noted.lol/tutorial-setting-up-shiori. The docker-compose file in this tutorial changes the shiori data dir with an environment variable to SHIORI_DIR=/data and creates a volume for /data. Using the same docker-compose file gave me the Failed to open database: unable to open database file: out of memory (14) error.

version: "3"
services:
  shiori:
    image: ghcr.io/go-shiori/shiori
    container_name: shiori
    environment:
      - PUID=1000
      - PGID=1000
      - SHIORI_DIR=/data
    ports:
      - 8080:8080
    restart: unless-stopped
    volumes:
      - /data/shiori:/data

I removed the environment variable so shiori would write the db to the default location /shiori and also changed the volume path to the same location, now everything is working great.

Thanks. Was hoping it would be something I overlooked, but I’m already using the default shiori location 🥲. I’m using the exact same compose file as the OP.

[therysin]

I ended up using a regular docker volume for persistence and not a bind mount.

Probably a temp fix, but works out in the short term.

[fmartingr]

I have been trying to run and connect to an Oracle Cloud ubuntu instance for 15 minutes now. I wanted to try over there just in case, but this works on my mac/linux machines. You should be able to create a mounted volume if you have enough permissions over the folder being mounted. I'll leave this open for now, in case someone else can chip in here.

[danielnbalasoiu]

I had the same error and I fixed it by creating the local directory and then set the correct permissions expected by shiori.

mkdir ./shiori-data
chown -R 1000:1000 ./shiori-data

[milosimpson]

Also running into this issue trying to get shiori running in docker on a Synology NAS.

If I don't specify a data volume, I can get v.1.5.5 to boot because shiori will make a sqlite db in the/shiori dir inside the docker container, which works but isn't great as there is no upgrade path as the db is "trapped" inside the container.

When I try to give it an external volume/mount, it fails to be able to make the sqllite db file because user 1000/shiori doesn't have write permissions to the mounted directory.

I am guessing you want to use a non-root user for security reasons since this service exposes an http service.

I think the fix might be to: while booting, as root chown the SHIORI_DIR to the uid 1000/shiori user, then switch to the uid 1000/shiori user.

I tried to test this by pre-populating a shiori.db file in the mount volumn/directory, to get past the boot failure so that I could shell into the container and try to run sudo chown -R 1000:1000 on the mounted dir to see if that would temporarily make things work, but the image does not contain sudo.

Shelled into the container (v1.5.5) to look at who owns /shiori/shiori.db on a setup where there is no external volume mount.
User shiori owns the file.

Shelled into the container with an external volume mount with pre-existing shiori.db file (v1.5.3 that it doesn't try to migrate db on boot). User root owns that file/directory which means it isn't usable.

[GeekyNinja2003]

Hi , Thank you for reaching out regarding the Shiori Docker issue on Synology NAS. I appreciate your detailed explanation of the problem. It seems like you’re encountering a permissions issue when trying to use an external volume for Shiori in Docker. You’re correct in assuming that we aim to use a non-root user for security reasons, especially for services that expose HTTP. To address this issue, your suggested approach could be a viable solution. Here’s a step-by-step breakdown: 1. While booting as root: • Change ownership of the SHIORI_DIR to the UID 1000/shiori user. • Switch to the UID 1000/shiori user. This should resolve the permission problem and allow Shiori to create the SQLite DB file on the external volume. Unfortunately, the image does not contain sudo, as you’ve mentioned. In this case, you might need to use other methods to change ownership, such as using the chown command with the --no-dereference option. Feel free to test this approach and let us know how it works for you. If you have any further questions or encounter any issues during this process, please don’t hesitate to reach out for additional assistance. Best regards, Jerry;-)

…

On Mon, 23 Oct 2023 at 02:18 Milo Simpson ***@***.***> wrote: Also running into this issue trying to get shiori running in docker on a Synology NAS. If I don't specify a data volume, I can get v.1.5.5 to boot because shiori will make a sqlite db in the/shiori dir inside the docker container, which works but isn't great as there is no upgrade path as the db is "trapped" inside the container. When I try to give it an external volume/mount, it fails to be able to make the sqllite db file because user 1000/shiori doesn't have permissions to the mount. I am guessing you want to use a non-root user for security reasons / since this service exposes an http service. I think the fix might be to: while booting as root, chown the SHIORI_DIR to the uid 1000/shiori user, then switch to the uid 1000/shiori user. I tried to test this by pre-populating a shiori.db file in the mount dir, to at least get past the boot failure on v1.5.5 so that I could shell into the container and try to run sudo chown -R 1000:1000 on the mounted dir to see if that would temporarily make things work, but the image does not contain sudo. — Reply to this email directly, view it on GitHub <#546 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A7OXXSAQUYIE3QUV7HG6PU3YAXOZ3AVCNFSM6AAAAAATNPG3L6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONZUGM3DQMRUGU> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[fmartingr]

I need to unify both Dockerfiles, since we use one for the CI (production builds) and one for development, and that makes no sense right now. That said, Shiori should work using a mounted volume without much issue as I have been running it like that for some time now.

Maybe allowing to customize the user and group IDs on the image via environment variables would work well, but that something I need to investigate and make backwards compatible, so it will be after we release 1.6.0, since that already contains enough changes as it is.

[milosimpson]

Figured out my problem. It is a Synology NAS specific issue. The fix was to create a specific non-root user with access to the mount directory ala https://drfrankenstein.co.uk/step-2-setting-up-a-restricted-docker-user-and-obtaining-ids/ and then use that user with Docker Compose. The Synology "Container Manager" app is basically Synology's Portainer, and it can run docker compose yaml files.

version: '3.3'
services:
  shiori:
    image: ghcr.io/go-shiori/shiori
    container_name: shiori
    user: UID:GID   // this is the important part; set the user and group that have access to  /volume2/docker/shiori
    ports:
      - 8080:8080
    restart: unless-stopped
    volumes:
      - /volume2/docker/shiori:/shiori           
      // note you have to specify the the "real" path on host not the nice Synology share name

It seemed like most suggestions on the internet were to make user and group that the docker image wants on the host OS, which is easy to do on Linux systems, but not on the Synology.

The catch is that it seems like only compose and the yaml file approach can set the user and group id, not the "docker" command line or simple docker run UIs in the Synology.

The one nice thing that Shiroi could do is take UID and PID to use as environment variables, that way it could be run w/out compose.

[danielyrovas]

I ran into this issue using podman running in rootless mode. I manged to solve it by building a container as below:

FROM ghcr.io/go-shiori/shiori:latest AS source

FROM docker.io/alpine:3.19
LABEL org.opencontainers.image.source https://github.com/go-shiori/shiori
COPY --from=source /usr/bin/shiori /usr/bin/

EXPOSE 8080
WORKDIR /shiori
ENV SHIORI_DIR /shiori/
ENTRYPOINT ["/usr/bin/shiori"]
CMD ["server"]

[fmartingr]

I just tested the image on PR #907 on a linux laptop and it required no further modifications in order to mount a local path:

$ cd $(mktemp -d)
$ docker run -v ./data:/shiori -p 8080:8080 ghcr.io/go-shiori/shiori:pr-907
... server running, add bookmarks
^C
$ ls data
shiori.db
$ sqlite3 data/shiori.db "SELECT COUNT(*) FROM bookmark;
1