chown error when restarting containers

[tsmith-rv]

Similar to #852, when using images mysql:5.7 and mysql:8.0 I receive chown errors when trying to restart the container. The error specifically is: chown: changing ownership of '/var/lib/mysql/mysql.sock': No such file or directory.

Here is my docker-compose file:

version: '3.3'

services:
  db:
    platform: linux/x86_64
    image: mysql:8.0
    restart: always
    environment:
      MYSQL_DATABASE: 'database1'
      MYSQL_ROOT_PASSWORD: 'pass'
    ports:
      - '3306:3306'
    expose:
      - '3306'
    command: --sql-mode=''
    volumes:
      - ./_mysql-docker:/var/lib/mysql

Whether I stop the containers with Ctrl+C and run docker-compose up, or run docker stop then docker start I see the same issue. The first time I start the container it works fine, then subsequent runs produce the error in the output below.

My container logs when trying to start the container again:

db_1  | 2023-01-18 04:38:06+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.0.32-1.el8 started.
db_1  | chown: changing ownership of '/var/lib/mysql/mysql.sock': No such file or directory
db_1  | chown: changing ownership of '/var/lib/mysql/mysql.sock': No such file or directory

My host directory _mysql_docker contains the mysql.sock symbolic link and as a sanity test I checked inside the container and that symbolic link is exactly where it should be at /var/lib/mysql/mysql.sock.

My host machine is ARM running macOS Monterey.

[dbagwell]

I am getting the same error. If I remove the mysql.sock symlink from my host directory before starting the container, the container starts up just fine (recreating the symlink in the host directory).

[tsmith-rv]

I am getting the same error. If I remove the mysql.sock symlink from my host directory before starting the container, the container starts up just fine (recreating the symlink in the host directory).

Same @dbagwell! Any solution that involves deleting mysql.sock before starting the container allows the container to start with no problem. Seems like one solution might involve cleaning up at the symbolic link at some point in the process, but really I just want to know why chown says it can't find the file. That part just has me stumped.

[tianon]

That's weird -- even if the file is a dangling symlink, this should work fine because we use --no-dereference (so it should change the ownership of the symlink, which should work fine):

mysql/template/docker-entrypoint.sh

Line 231 in b3dc453

find "${!dirs[@]}" \! -user mysql -exec chown --no-dereference mysql '{}' +

[tsmith-rv]

That's weird -- even if the file is a dangling symlink, this should work fine because we use --no-dereference (so it should change the ownership of the symlink, which should work fine):

mysql/template/docker-entrypoint.sh

Line 231 in b3dc453

find "${!dirs[@]}" \! -user mysql -exec chown --no-dereference mysql '{}' +

Oh agreed for sure. Seems like some subtle nuance of how chown works, maybe? My guess is that "No such file or directory" is actually just a bit of a red herring since the symlink is obviously there. Possibly something negating the --no-dereference switch?

[tianon]

Do you still have a way to reliably reproduce? Could you maybe try something smaller like docker run --rm --mount ... mysql:image-you-use chown --no-dereference mysql /var/lib/mysql/mysql.sock directly?

(If you can reproduce reliably, steps to help others reproduce would be useful 😅)

[tsmith-rv]

@tianon sure! Here is an explicit list of steps I'm following:

1. From your terminal, run docker-compose -f ./docker-compose.local.yml up (docker-compose.local.yml contains the contents shown above in my original comment)
2. In that same terminal, once you receive the "all good" from your container (something like db_1 | Version: '5.7.41' socket: '/var/run/mysqld/mysqld.sock' port: 3306 MySQL Community Server (GPL)) , press Ctrl+C at which point you'll see something like Gracefully stopping... (press Ctrl+C again to force). Wait until you get Stopping service_db_1 ... done.
3. Continuing in that same terminal, run docker-compose -f ./docker-compose.local.yml up again. You should then see:

db_1  | 2023-01-19 00:28:27+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 5.7.41-1.el7 started.
db_1  | chown: changing ownership of '/var/lib/mysql/mysql.sock': No such file or directory
db_1  | chown: changing ownership of '/var/lib/mysql/mysql.sock': No such file or directory

then the container stops.

Note: I have also tried this same methodology, but instead replacing step 2 and step 3 with docker stop and docker start commands respectively and achieved the same result.

This is reliable as hell for me. Never fails (or, you know, succeeds?)

[tianon]

Doh, I guess this is probably something about either macOS not supporting chown on a symlink or the Docker Desktop file sharing not supporting chown on a symlink? Not sure, but I think the interaction between those is probably the culprit (to be honest, I'm surprised you're not seeing more issues with MySQL itself not liking its database files existing on that shared filesystem too).

[tianon]

To put that another way, if you switch from a bind-mount of /var/lib/mysql to your "host" system (outside the Docker Desktop VM) to a named volume (which then lives inside the VM), this problem should go away.

[tsmith-rv]

So, the good news...

@tianon heeeeeey switching to the named volume did work.

Funny enough, on my host machine, I tried running chown on symbolic links and that didn't return an error (even forced a "No such file or directory" error by trying it on a non-existent file to make sure it produces an error on macOS).

Here is what my file looks like now, which is working (and has actually seriously reduced database startup times as well as reduced latency on database inserts):

version: '3.3'

services:
  db:
    platform: linux/x86_64
    image: mysql:8.0
    restart: always
    environment:
      MYSQL_DATABASE: 'database1'
      MYSQL_ROOT_PASSWORD: 'pass'
    ports:
      - '3306:3306'
    expose:
      - '3306'
    command: --sql-mode=''
    volumes:
      - db:/var/lib/mysql

volumes:
  db:

Also:

I'm surprised you're not seeing more issues with MySQL itself not liking its database files existing on that shared filesystem

Funny you should mention it: we have. It is has been a huge headache, actually. We actually started seeing issues all of a sudden because of case sensitivity issues in the database files, but we resolved the issue using the top answer here: https://stackoverflow.com/questions/64146845/mysql-not-starting-in-a-docker-container-on-macos-after-docker-update

Of course what we didn't realize is that we were fixing a symptom rather than the cause. After switching to the named volume, I tested running the containers and our tests with the "grpc fuse for file sharing" option enabled again (and restarting docker, using new images, containers, volumes, etc.) and everything is still working, which means we no longer need this "workaround".

But still...

While I am now going to use the named volume instead (thank you very much @tianon), doesn't the problem I initially reported still seem like a bug? It seems like regardless of whether someone is running the oracle images on mac or not (arm or not? still not sure what the actual problem with chown is here) they should reasonably expect container restarts to be successful.

[tsmith-rv]

Heeeeeyyyyyyy making some progress here (maybe).

If I add a group to the chown invocation, it works. So, I updated the entrypoint.sh script with find "${!dirs[@]}" \! -user mysql -exec chown --no-dereference mysql:root '{}' + and no longer receive the error.

[rabbitlhf]

here is my way:

1. find the user and group id on host machine with command id in macos
2. add to docker-compose file with 'user' label ：user: 501:20

[amushate]

this worked for me by adding user and rw as below. Note user has to be mysql not your root user

version: '3.3'

services:
  db:
    platform: linux/x86_64
    image: mysql:8.0
    restart: always
    environment:
      MYSQL_DATABASE: 'database1'
      MYSQL_ROOT_PASSWORD: 'pass'
    ports:
      - '3306:3306'
    expose:
      - '3306'
    command: --sql-mode=''
    volumes:
      - db:/var/lib/mysql:rw
    user: mysql

volumes:
  db:

[JoseSecmotic]

In my case, I removed the mysql.sock file and it did work again

[tianon]

Running this image with the database shared over Docker Desktop's (or other) file sharing which is effectively remote file sharing is not likely to be a good time, and not likely to be well supported by MySQL itself. It's unfortunate that it doesn't work out-of-the-box, but I don't think it's going to be super fruitful for us to be chasing workarounds due to esoteric filesystems that MySQL upstream doesn't officially support either. 🙇