Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Docker container fails on first startup due to inability to connect to Sqlite database #19156

Closed
pierrebuckley opened this issue Jul 13, 2023 · 25 comments
Assignees

Comments

@pierrebuckley
Copy link

pierrebuckley commented Jul 13, 2023

Describe the Bug

I am attempting to setup Directus using the docker compose example provided in the Directus documentation. When the container is starting up, it immediately fails. I have included the container logs below. I am not sure how to proceed and would appreciate your feedback. Thank you.

Container Logs

[14:49:36.784] INFO: Initializing bootstrap...
[14:50:01.811] ERROR: Can't connect to the database.
[14:50:01.811] ERROR: SQLITE_CANTOPEN: unable to open database file
    err: {
      "type": "Error",
      "message": "SQLITE_CANTOPEN: unable to open database file",
      "stack":
          Error: SQLITE_CANTOPEN: unable to open database file
      "errno": 14,
      "code": "SQLITE_CANTOPEN"
    }

To Reproduce

  1. Create docker-compose.yaml file and paste content below and save file.
  2. Execute command docker compose up -d.

Docker Compose

version: '3'
services:
  directus:
    image: directus/directus:latest
    ports:
      - 8055:8055
    volumes:
      - /opt/directus/database:/directus/database
      - /opt/directus/uploads:/directus/uploads
    environment:
      KEY: 'b30562eb-6264-470a-9a5b-3c90ae971a0f'
      SECRET: '9c157795-1079-417a-bb18-0e950e10171a'
      ADMIN_EMAIL: 'my@email.com'
      ADMIN_PASSWORD: 'Pa$$w0rd'
      WEBSOCKETS_ENABLED: true

Directus Version

v10.4.3

Hosting Strategy

Self-Hosted (Docker Image)

@br41nslug
Copy link
Member

br41nslug commented Jul 13, 2023

Looks like you're either missing or forgot to include the database specification in your docker-compose snippet.

environment:
  DB_CLIENT: 'sqlite3'
  DB_FILENAME: '/directus/database/data.db'

Also double check the folder permissions of /opt/directus/database and /opt/directus/uploads the docker should have read and write access to those.

@0xTux
Copy link

0xTux commented Jul 13, 2023

Looks like you're either missing or forgot to include the database specification in your docker-compose snippet.

environment:
  DB_CLIENT: 'sqlite3'
  DB_FILENAME: '/directus/database/data.db'

Facing the same issue and this does not work either

@br41nslug
Copy link
Member

Facing the same issue and this does not work either

Thats not a lot to go on? Are you using the same config with missing required database settings? Did you check the permissions?

@br41nslug br41nslug closed this as not planned Won't fix, can't repro, duplicate, stale Jul 14, 2023
@jakebailey
Copy link

I had this exact problem, and the problem turned out to be the permissions on the two folders; just using:

version: '3'
services:
  directus:
    image: directus/directus:latest
    ports:
      - 8055:8055
    volumes:
      - ./database:/directus/database
      - ./uploads:/directus/uploads
    environment:
      KEY: '2b93bbfa-6650-459f-b6e1-1016d96e1b25'
      SECRET: '8053fec6-1e3d-4a6c-8f65-4e7291308f5a'
      ADMIN_EMAIL: 'admin@example.com'
      ADMIN_PASSWORD: 'd1r3ctu5'
      DB_CLIENT: 'sqlite3'
      DB_FILENAME: '/directus/database/data.db'
      WEBSOCKETS_ENABLED: true

Caused an error on first request:

directus-directus-1  | [20:21:09.837] INFO: Initializing bootstrap...
directus-directus-1  | [20:21:34.880] ERROR: Can't connect to the database.
directus-directus-1  | [20:21:34.880] ERROR: SQLITE_CANTOPEN: unable to open database file
directus-directus-1  |     err: {
directus-directus-1  |       "type": "Error",
directus-directus-1  |       "message": "SQLITE_CANTOPEN: unable to open database file",
directus-directus-1  |       "stack":
directus-directus-1  |           Error: SQLITE_CANTOPEN: unable to open database file
directus-directus-1  |       "errno": 14,
directus-directus-1  |       "code": "SQLITE_CANTOPEN"
directus-directus-1  |     }
directus-directus-1 exited with code 1

The two directories had been created by the root user inside the container:

total 12K
drwxr-xr-x 2 root    root    4.0K Jul 16 13:21 database/
-rw-r--r-- 1 jabaile jabaile  485 Jul 16 13:27 docker-compose.yml
drwxr-xr-x 2 root    root    4.0K Jul 16 13:21 uploads/

Removing them and recreating them as my current user fixed the problem and allowed startup to proceed; I haven't had this issue with other folder-using containers before, so maybe this comes down to which users the container runs as internally.

@robclancy
Copy link

This is the config the docs give us. I have the same issue. The docs where the config is shown (https://docs.directus.io/self-hosted/quickstart.html) should probably suggest you create the 2 volume folders yourself.

@rijkvanzanten
Copy link
Member

This is the config the docs give us. I have the same issue. The docs where the config is shown (https://docs.directus.io/self-hosted/quickstart.html) should probably suggest you create the 2 volume folders yourself.

What environment are you running this on @robclancy / @jakebailey? It shouldn't matter (that's kinda the point of Docker), but just to be sure. On my end (on macOS Ventura 13.4.1), things seem to work exactly as expected:

Created an empty folder, copy pasted the linked docker-compose.yml file

version: '3'
services:
  directus:
    image: directus/directus:latest
    ports:
      - 8055:8055
    volumes:
      - ./database:/directus/database
      - ./uploads:/directus/uploads
    environment:
      KEY: 'replace-with-random-value'
      SECRET: 'replace-with-random-value'
      ADMIN_EMAIL: 'admin@example.com'
      ADMIN_PASSWORD: 'd1r3ctu5'
      DB_CLIENT: 'sqlite3'
      DB_FILENAME: '/directus/database/data.db'
      WEBSOCKETS_ENABLED: true

ran docker compose up, and things kicked to life apparently without issue:

docker compose output
💲  docker compose up
[+] Running 7/7
 ✔ directus 6 layers [⣿⣿⣿⣿⣿⣿]      0B/0B      Pulled                                                                                  11.8s
   ✔ 8c6d1654570f Already exists                                                                                                       0.0s
   ✔ 137137f25bb5 Already exists                                                                                                       0.0s
   ✔ 1f1c601deed0 Already exists                                                                                                       0.0s
   ✔ 1dd397d4235d Already exists                                                                                                       0.0s
   ✔ 8e518b072b5c Pull complete                                                                                                        0.2s
   ✔ 7a593460609b Pull complete                                                                                                       10.7s
[+] Building 0.0s (0/0)
[+] Running 2/2
 ✔ Network directus-docker-test_default       Created                                                                                  0.1s
 ✔ Container directus-docker-test-directus-1  Created                                                                                  0.3s
Attaching to directus-docker-test-directus-1
directus-docker-test-directus-1  | [16:40:56.069] INFO: Initializing bootstrap...
directus-docker-test-directus-1  | [16:40:56.107] INFO: Installing Directus system tables...
directus-docker-test-directus-1  | [16:40:56.322] INFO: Running migrations...
directus-docker-test-directus-1  | [16:40:56.327] INFO: Applying Remove Collection Foreign Keys...
directus-docker-test-directus-1  | [16:40:56.482] INFO: Applying Remove System Relations...
directus-docker-test-directus-1  | [16:40:56.495] INFO: Applying Remove System Collections...
directus-docker-test-directus-1  | [16:40:56.512] INFO: Applying Remove System Fields...
directus-docker-test-directus-1  | [16:40:56.522] INFO: Applying Add Cascade System Relations...
directus-docker-test-directus-1  | [16:40:57.112] INFO: Applying Change Webhook URL Type...
directus-docker-test-directus-1  | [16:40:57.146] INFO: Applying Add Relations Sort Field...
directus-docker-test-directus-1  | [16:40:57.162] INFO: Applying Remove Locked Fields...
directus-docker-test-directus-1  | [16:40:57.190] INFO: Applying Webhooks Collections Text...
directus-docker-test-directus-1  | [16:40:57.211] INFO: Applying Add Refresh Interval...
directus-docker-test-directus-1  | [16:40:57.226] INFO: Applying Make Filesize Nullable...
directus-docker-test-directus-1  | [16:40:57.250] INFO: Applying Add Collections Accountability...
directus-docker-test-directus-1  | [16:40:57.265] INFO: Applying Remove Files Interface...
directus-docker-test-directus-1  | [16:40:57.274] INFO: Applying Rename Interfaces...
directus-docker-test-directus-1  | [16:40:57.312] INFO: Applying Restructure Relations...
directus-docker-test-directus-1  | [16:40:57.372] INFO: Applying Add Foreign Key Constraints...
directus-docker-test-directus-1  | [16:40:57.389] INFO: Applying Add System Fk Triggers...
directus-docker-test-directus-1  | [16:40:57.634] INFO: Applying Add Collections Icon Color...
directus-docker-test-directus-1  | [16:40:57.648] INFO: Applying Add Insights...
directus-docker-test-directus-1  | [16:40:57.670] INFO: Applying Add Deep Clone Config...
directus-docker-test-directus-1  | [16:40:57.685] INFO: Applying Change Filesize Bigint...
directus-docker-test-directus-1  | [16:40:57.709] INFO: Applying Add Conditions to Fields...
directus-docker-test-directus-1  | [16:40:57.727] INFO: Applying Add Default Folder...
directus-docker-test-directus-1  | [16:40:57.759] INFO: Applying Replace Groups...
directus-docker-test-directus-1  | [16:40:57.767] INFO: Applying Add Required to Fields...
directus-docker-test-directus-1  | [16:40:57.782] INFO: Applying Update Groups...
directus-docker-test-directus-1  | [16:40:57.801] INFO: Applying Change Image Metadata Structure...
directus-docker-test-directus-1  | [16:40:57.810] INFO: Applying Add Geometry Config...
directus-docker-test-directus-1  | [16:40:57.831] INFO: Applying Remove Limit Column...
directus-docker-test-directus-1  | [16:40:57.860] INFO: Applying Add Auth Provider...
directus-docker-test-directus-1  | [16:40:57.930] INFO: Applying Webhooks Collections Not Null...
directus-docker-test-directus-1  | [16:40:57.957] INFO: Applying Move Module Setup...
directus-docker-test-directus-1  | [16:40:58.022] INFO: Applying Webhooks URL Not Null...
directus-docker-test-directus-1  | [16:40:58.055] INFO: Applying Add Collection Organization...
directus-docker-test-directus-1  | [16:40:58.121] INFO: Applying Replace Fields Group...
directus-docker-test-directus-1  | [16:40:58.173] INFO: Applying Replace M2M Interface...
directus-docker-test-directus-1  | [16:40:58.179] INFO: Applying Rename Login Action...
directus-docker-test-directus-1  | [16:40:58.189] INFO: Applying Update Presets...
directus-docker-test-directus-1  | [16:40:58.242] INFO: Applying Add Auth Data...
directus-docker-test-directus-1  | [16:40:58.256] INFO: Applying Add Webhook Headers...
directus-docker-test-directus-1  | [16:40:58.275] INFO: Applying Set Unique to User Token...
directus-docker-test-directus-1  | [16:40:58.306] INFO: Applying Update Special Geometry...
directus-docker-test-directus-1  | [16:40:58.317] INFO: Applying Remove Collections Listing...
directus-docker-test-directus-1  | [16:40:58.349] INFO: Applying Add Notifications...
directus-docker-test-directus-1  | [16:40:58.372] INFO: Applying Add Shares...
directus-docker-test-directus-1  | [16:40:58.440] INFO: Applying Add Project Descriptor...
directus-docker-test-directus-1  | [16:40:58.461] INFO: Applying Remove Default Project Color...
directus-docker-test-directus-1  | [16:40:58.491] INFO: Applying Add Bookmark Icon and Color...
directus-docker-test-directus-1  | [16:40:58.576] INFO: Applying Add Translation Strings...
directus-docker-test-directus-1  | [16:40:58.603] INFO: Applying Rename Field Typecast Flags...
directus-docker-test-directus-1  | [16:40:58.613] INFO: Applying Add Field Validation...
directus-docker-test-directus-1  | [16:40:58.658] INFO: Applying Fix Typecast Flags...
directus-docker-test-directus-1  | [16:40:58.679] INFO: Applying Add Default Language...
directus-docker-test-directus-1  | [16:40:58.751] INFO: Applying Remove Default Value Panel Icon...
directus-docker-test-directus-1  | [16:40:58.817] INFO: Applying Add Flows...
directus-docker-test-directus-1  | [16:40:58.941] INFO: Applying Add Color to Insights Icon...
directus-docker-test-directus-1  | [16:40:59.000] INFO: Applying Drop Non Null From IP of Activity...
directus-docker-test-directus-1  | [16:40:59.069] INFO: Applying Drop Non Null From Sender of Notifications...
directus-docker-test-directus-1  | [16:40:59.150] INFO: Applying Rename Hook Trigger to Event...
directus-docker-test-directus-1  | [16:40:59.185] INFO: Applying Update Notifications Timestamp Column...
directus-docker-test-directus-1  | [16:40:59.247] INFO: Applying Add Custom Aspect Ratios...
directus-docker-test-directus-1  | [16:40:59.304] INFO: Applying Add Origin to Accountability...
directus-docker-test-directus-1  | [16:40:59.396] INFO: Applying Update Material Icons...
directus-docker-test-directus-1  | [16:40:59.508] INFO: Applying Add Preview Settings...
directus-docker-test-directus-1  | [16:40:59.553] INFO: Applying Migrate Translation Strings...
directus-docker-test-directus-1  | [16:40:59.739] INFO: Setting up first admin role...
directus-docker-test-directus-1  | [16:40:59.767] INFO: Adding first admin user...
directus-docker-test-directus-1  | [16:41:00.215] INFO: Done
directus-docker-test-directus-1  | [16:41:04.520] WARN: PUBLIC_URL should be a full URL
directus-docker-test-directus-1  | [16:41:04.551] WARN: Spatialite isn't installed. Geometry type support will be limited.
directus-docker-test-directus-1  | [16:41:04.662] INFO: GraphQL Subscriptions started at ws://0.0.0.0:8055/graphql
directus-docker-test-directus-1  | [16:41:04.662] INFO: WebSocket Server started at ws://0.0.0.0:8055/websocket
directus-docker-test-directus-1  | [16:41:04.685] INFO: Server started at http://0.0.0.0:8055

I was able to browse/use the app as normal, and based on the file system, it looks like the volumes are syncing / behaving as intended as well 🤔

CleanShot 2023-07-24 at 12 45 21@2x

@jakebailey
Copy link

I was originally using Docker within WSL (which is effectively a VM), but I also just tested on my native Arch Linux machine; doing exactly what you wrote above fails with the sqlite error.

If you've only been testing on macOS and not on Linux, it's possible that there's some weirdness coming from the mixed-OS setup (maybe due to the way FS accesses are proxied, or some differing mapping of UIDs between the two).

@robclancy
Copy link

Arch btw.

@rijkvanzanten
Copy link
Member

If you've only been testing on macOS and not on Linux, it's possible that there's some weirdness coming from the mixed-OS setup (maybe due to the way FS accesses are proxied, or some differing mapping of UIDs between the two).

Yeah that's what I'm starting to think too 🤔 @br41nslug can you give it another shot on your end with this exact setup under WSL?

@jakebailey
Copy link

Just note that if you're on Windows using Docker Desktop, IIRC that's a totally different thing than running the docker daemon within WSL itself; the former has its own special daemon that can run Linux and Windows containers, while the latter is just like installing docker straight onto an Ubuntu VM and running there. (I think most people seem to use the former, but I prefer the latter as I'm much more used to using docker on actual Linux machines.)

@rijkvanzanten
Copy link
Member

Great great point, which could also explain the difference as one would be running with special windows sauce, whereas the latter would be "pure" linux (which would be more similar to Arch)

@br41nslug
Copy link
Member

br41nslug commented Jul 24, 2023

This is reproducable on most linux os's using a default docker installation. The problem seems to be that Docker by default installs the deamon as root which will then create the non-existing database folder using root permissions then the container is unable to write the sqlite file using the node:node/1000:1000 internal user.

Coincedentally i reproduced this a couple days ago (but had lost track of this ticket 😅)

screengrab.mp4

@br41nslug br41nslug reopened this Jul 24, 2023
@jakebailey
Copy link

One thing I see commonly is container images which accept UID/GID via environment (defaulting to 1000:1000), and then access the filesystem with that user, which may help. e.g. https://docs.linuxserver.io/general/understanding-puid-and-pgid via https://github.com/just-containers/s6-overlay or something.

I am not an expert, though. Honestly, I'm a little confused from the perspective that I regularly use the postgres image and point it to a directory, and don't have this problem. What is the difference between the two setups such that directus' image fails?

@paescuj
Copy link
Member

paescuj commented Jul 24, 2023

I am not an expert, though. Honestly, I'm a little confused from the perspective that I regularly use the postgres image and point it to a directory, and don't have this problem. What is the difference between the two setups such that directus' image fails?

For security reasons, the directus image runs with the unprivileged user node instead of root.


I'm not sure whether there's much we can do, except for adding something like this to the documentation: #6480 (comment)

Another idea: We could test if the problem can be solved on the affected platforms by adding the explicit VOLUME declaration back in, see
https://github.com/directus/directus/pull/17965/files#diff-dd2c0eb6ea5cfc6c4bd4eac30934e2d5746747af48fef6da689e85b752f39557L49-L51.

@rijkvanzanten
Copy link
Member

@paescuj Another thing that might be worth trying is creating the database folder within the dockerfile and set the permissions appropriately before Directus runs?

&& mkdir -p database extensions uploads \

this is creating the folder as root, but it's user node that uses the nested directory:

directus/Dockerfile

Lines 35 to 37 in 3012348

USER node
WORKDIR /directus

It actually seems like there's a long thread on the docker-node repo on a similar topic: nodejs/docker-node#740

(it's still beyond me why this just works out of the box on some environments, but not others, but 🤷🏻 )

@rijkvanzanten
Copy link
Member

Cc @wrynegade who might have some ideas

@br41nslug
Copy link
Member

jakebailey does have a point i've never had this issue either with a postgres docker which goes through effectively the same process of creating a folder and populating it 🤔 That does make me think this is something we should be able to resolve with our image

@DieselTech
Copy link

What help is needed moving this forward? Fresh pulls of the container are still failing.

@DanielBiegler
Copy link
Contributor

DanielBiegler commented Aug 25, 2023

Just took a look at the postgres Dockerfile and they have a section explicitly for setting user/group IDs, see:

https://github.com/docker-library/postgres/blob/ee530cc079f232f9b1045db43d8c501ee2057d6d/16/bookworm/Dockerfile#L9-L17

Need to take a look 👀

@DanielBiegler
Copy link
Contributor

@vikyw89
Copy link

vikyw89 commented Oct 20, 2023

same problem on Fedora38 on empty dir

[vikyw@fedora directus-playground]$ docker compose up
[+] Running 1/0
 ✔ Container directus-playground-directus-1  Created                                                                                                                          0.0s 
Attaching to directus-playground-directus-1
directus-playground-directus-1  | [04:31:40.686] INFO: Initializing bootstrap...
directus-playground-directus-1  | [04:32:05.738] ERROR: Can't connect to the database.
directus-playground-directus-1  | [04:32:05.738] ERROR: SQLITE_CANTOPEN: unable to open database file
directus-playground-directus-1  |     err: {
directus-playground-directus-1  |       "type": "Error",
directus-playground-directus-1  |       "message": "SQLITE_CANTOPEN: unable to open database file",
directus-playground-directus-1  |       "stack":
directus-playground-directus-1  |           Error: SQLITE_CANTOPEN: unable to open database file
directus-playground-directus-1  |       "errno": 14,
directus-playground-directus-1  |       "code": "SQLITE_CANTOPEN"
directus-playground-directus-1  |     }
directus-playground-directus-1 exited with code 1

@vikyw89
Copy link

vikyw89 commented Oct 20, 2023

hacky fix by creating /uploads and /database manually before running docker compose up

root
    /uploads
    /database
    docker-compose.yml

@rmill2016
Copy link

hacky fix by creating /uploads and /database manually before running docker compose up

root
    /uploads
    /database
    docker-compose.yml

doesn't work for me, same issue. I'm on Ubuntu 23.10 x64

@rmill2016
Copy link

hacky fix by creating /uploads and /database manually before running docker compose up

root
    /uploads
    /database
    docker-compose.yml

doesn't work for me, same issue. I'm on Ubuntu 23.10 x64

WHAT WORKED FOR ME!
I had to explicitly set the permissions for both database and uploads on my volumes for my current user. Even recreating the volumes as the current user didn't work. What did in this case was doing:

chmod g+rwx database uploads
AND
chmod o+rwx database uploads

@connorwinston
Copy link
Member

Heyo! We have done research into this and this is primarily a docker and docker-compose issue rather than a Directus issue so we will not be continuing investigating/fixing at this time. After diving in this appears to be docker/docker-compose not properly handling permissions for volumes, there are workarounds but there is no straight forward fix. If you are experiencing this issue, you should verify and fix the permissions on the directory you are using with your volume to make sure it aligns. If the scope of this issue grows and this becomes a widespread issue we will of course reopen it and dig back into it 👍 !

@connorwinston connorwinston closed this as not planned Won't fix, can't repro, duplicate, stale Nov 30, 2023
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Jan 31, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
Archived in project
Development

No branches or pull requests