Cannot persist the data by doing docker commit

[skwokmag]

Hi,

Please help! This is urgent to me.

I have tried to start the mysql image (5.6.22) in a couple of ways:

docker run -it -h mysql --name mysqlt1 -e MYSQL_ROOT_PASSWORD=test -e MYSQL_USER=mmx -e MYSQL_PASSWORD=mmx -e MYSQL_DATABASE=mmx mysql:5.6.22
docker run -it -h mysql --name mysqlt1 -v $PWD/var/lib/mysql:/var/lib/mysql -v $PWD/etc/mysql:/etc/mysql -e MYSQL_ROOT_PASSWORD=test mysql:5.6.22

Then, I commit the image, stop the old one, and start the new image by executing the same docker run command with the tag replaced with the new image's.

Result:

Every time when the new image starts, it will execute /tmp/mysql-first-time.sql and so the data is removed.

What I'd like to achieve is to persist the data inside the container. In the mounted volume is fine as well.

Startup Logs:

$ docker run -it -h mysql --name mysqlt2 -e MYSQL_ROOT_PASSWORD=test -e MYSQL_USER=mmx -e MYSQL_PASSWORD=mmx -e MYSQL_DATABASE=mmx mysql:t2
Running mysql_install_db ...
Installing MySQL system tables...2015-01-28 05:29:26 12 [Note] InnoDB: Using atomics to ref count buffer pool pages
2015-01-28 05:29:26 12 [Note] InnoDB: The InnoDB memory heap is disabled
2015-01-28 05:29:26 12 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2015-01-28 05:29:26 12 [Note] InnoDB: Memory barrier is not used
2015-01-28 05:29:26 12 [Note] InnoDB: Compressed tables use zlib 1.2.7
2015-01-28 05:29:26 12 [Note] InnoDB: Using Linux native AIO
2015-01-28 05:29:26 12 [Note] InnoDB: Not using CPU crc32 instructions
2015-01-28 05:29:26 12 [Note] InnoDB: Initializing buffer pool, size = 128.0M
2015-01-28 05:29:26 12 [Note] InnoDB: Completed initialization of buffer pool
2015-01-28 05:29:26 12 [Note] InnoDB: The first specified data file ./ibdata1 did not exist: a new database to be created!
2015-01-28 05:29:26 12 [Note] InnoDB: Setting file ./ibdata1 size to 12 MB
2015-01-28 05:29:26 12 [Note] InnoDB: Database physically writes the file full: wait...
2015-01-28 05:29:26 12 [Note] InnoDB: Setting log file ./ib_logfile101 size to 48 MB
2015-01-28 05:29:26 12 [Note] InnoDB: Setting log file ./ib_logfile1 size to 48 MB
2015-01-28 05:29:27 12 [Note] InnoDB: Renaming log file ./ib_logfile101 to ./ib_logfile0
2015-01-28 05:29:27 12 [Warning] InnoDB: New log files created, LSN=45781
2015-01-28 05:29:27 12 [Note] InnoDB: Doublewrite buffer not found: creating new
2015-01-28 05:29:27 12 [Note] InnoDB: Doublewrite buffer created
2015-01-28 05:29:27 12 [Note] InnoDB: 128 rollback segment(s) are active.
2015-01-28 05:29:27 12 [Warning] InnoDB: Creating foreign key constraint system tables.
2015-01-28 05:29:27 12 [Note] InnoDB: Foreign key constraint system tables created
2015-01-28 05:29:27 12 [Note] InnoDB: Creating tablespace and datafile system tables.
2015-01-28 05:29:27 12 [Note] InnoDB: Tablespace and datafile system tables created.
2015-01-28 05:29:27 12 [Note] InnoDB: Waiting for purge to start
2015-01-28 05:29:27 12 [Note] InnoDB: 5.6.22 started; log sequence number 0
2015-01-28 05:29:27 12 [Note] Binlog end
2015-01-28 05:29:27 12 [Note] InnoDB: FTS optimize thread exiting.
2015-01-28 05:29:27 12 [Note] InnoDB: Starting shutdown...
2015-01-28 05:29:28 12 [Note] InnoDB: Shutdown completed; log sequence number 1625977
OK

Filling help tables...2015-01-28 05:29:29 35 [Note] InnoDB: Using atomics to ref count buffer pool pages
2015-01-28 05:29:29 35 [Note] InnoDB: The InnoDB memory heap is disabled
2015-01-28 05:29:29 35 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2015-01-28 05:29:29 35 [Note] InnoDB: Memory barrier is not used
2015-01-28 05:29:29 35 [Note] InnoDB: Compressed tables use zlib 1.2.7
2015-01-28 05:29:29 35 [Note] InnoDB: Using Linux native AIO
2015-01-28 05:29:29 35 [Note] InnoDB: Not using CPU crc32 instructions
2015-01-28 05:29:29 35 [Note] InnoDB: Initializing buffer pool, size = 128.0M
2015-01-28 05:29:29 35 [Note] InnoDB: Completed initialization of buffer pool
2015-01-28 05:29:29 35 [Note] InnoDB: Highest supported file format is Barracuda.
2015-01-28 05:29:29 35 [Note] InnoDB: 128 rollback segment(s) are active.
2015-01-28 05:29:29 35 [Note] InnoDB: Waiting for purge to start
2015-01-28 05:29:29 35 [Note] InnoDB: 5.6.22 started; log sequence number 1625977
2015-01-28 05:29:29 35 [Note] Binlog end
2015-01-28 05:29:29 35 [Note] InnoDB: FTS optimize thread exiting.
2015-01-28 05:29:29 35 [Note] InnoDB: Starting shutdown...
2015-01-28 05:29:31 35 [Note] InnoDB: Shutdown completed; log sequence number 1625987
OK

To start mysqld at boot time you have to copy
support-files/mysql.server to the right place for your system

PLEASE REMEMBER TO SET A PASSWORD FOR THE MySQL root USER !
To do so, start the server, then issue the following commands:

  /usr/bin/mysqladmin -u root password 'new-password'
  /usr/bin/mysqladmin -u root -h mysql password 'new-password'

Alternatively you can run:

  /usr/bin/mysql_secure_installation

which will also give you the option of removing the test
databases and anonymous user created by default.  This is
strongly recommended for production servers.

See the manual for more instructions.

You can start the MySQL daemon with:

  cd /usr ; /usr/bin/mysqld_safe &

You can test the MySQL daemon with mysql-test-run.pl

  cd mysql-test ; perl mysql-test-run.pl

Please report any problems at http://bugs.mysql.com/

The latest information about MySQL is available on the web at

  http://www.mysql.com

Support MySQL by buying support/licenses at http://shop.mysql.com

WARNING: Found existing config file /usr/my.cnf on the system.
Because this file might be in use, it was not replaced,
but was used in bootstrap (unless you used --defaults-file)
and when you later start the server.
The new default config file was created as /usr/my-new.cnf,
please compare it with your file and take the changes you need.

WARNING: Default config file /etc/mysql/my.cnf exists on the system
This file will be read by default by the MySQL server
If you do not want to use this, either remove it, or use the
--defaults-file argument to mysqld_safe when starting the server

Finished mysql_install_db
2015-01-28 05:29:32 1 [Note] Plugin 'FEDERATED' is disabled.
2015-01-28 05:29:32 1 [Note] InnoDB: Using atomics to ref count buffer pool pages
2015-01-28 05:29:32 1 [Note] InnoDB: The InnoDB memory heap is disabled
2015-01-28 05:29:32 1 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2015-01-28 05:29:32 1 [Note] InnoDB: Memory barrier is not used
2015-01-28 05:29:32 1 [Note] InnoDB: Compressed tables use zlib 1.2.7
2015-01-28 05:29:32 1 [Note] InnoDB: Using Linux native AIO
2015-01-28 05:29:32 1 [Note] InnoDB: Not using CPU crc32 instructions
2015-01-28 05:29:32 1 [Note] InnoDB: Initializing buffer pool, size = 128.0M
2015-01-28 05:29:32 1 [Note] InnoDB: Completed initialization of buffer pool
2015-01-28 05:29:32 1 [Note] InnoDB: Highest supported file format is Barracuda.
2015-01-28 05:29:32 1 [Note] InnoDB: 128 rollback segment(s) are active.
2015-01-28 05:29:32 1 [Note] InnoDB: Waiting for purge to start
2015-01-28 05:29:32 1 [Note] InnoDB: 5.6.22 started; log sequence number 1625987
2015-01-28 05:29:32 1 [Warning] No existing UUID has been found, so we assume that this is the first time that this server has been started. Generating a new UUID: a3ae727e-a6ae-11e4-8719-0242ac110032.
2015-01-28 05:29:32 1 [Note] Server hostname (bind-address): '*'; port: 3306
2015-01-28 05:29:32 1 [Note] IPv6 is available.
2015-01-28 05:29:32 1 [Note]   - '::' resolves to '::';
2015-01-28 05:29:32 1 [Note] Server socket created on IP: '::'.
2015-01-28 05:29:32 1 [Note] Event Scheduler: Loaded 0 events
2015-01-28 05:29:32 1 [Note] Execution of init_file '/tmp/mysql-first-time.sql' started.
2015-01-28 05:29:32 1 [Note] Execution of init_file '/tmp/mysql-first-time.sql' ended.
2015-01-28 05:29:32 1 [Note] mysqld: ready for connections.
Version: '5.6.22'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  MySQL Community Server (GPL)
2015-01-28 05:36:47 1 [Note] mysqld: Normal shutdown

Env:

Os: OSX 10.10.1
Docker: Docker version 1.3.1, build 4e9bbfa
Boot2Docker-cli version: v1.3.1
Git commit: 57ccdb8

Thanks so much!

[thaJeztah]

In your example, you're not using a bind-mount, because of this, docker will create a new (empty) volume and the mysql container will initialise a new database.

The Dockerfile of the mysql image contains a volume definition for the mysql-data (https://github.com/docker-library/mysql/blob/master/5.7/Dockerfile#L35), so the data itself will never be included in the container itself, but also won't be deleted if you delete the container (unless doing docker rm -v container)

Possible solutions;

• use a bind-mounted volume to store the data and use the same location for the new container
• Use a data-only container (built from the mysql image) and use --volumes-from to use that data in your mysql container.

Let me know if you need more help with this.

Also wondering, what was your reason to commit the container? Were you trying to commit the database so that you could "clone" the container and have a second container with the same data in it?

[skwokmag]

Thanks for your quick reply!
Your answer makes sense to me but the problem is, I am mounting '/var/lib/mysql' and '/etc/mysql' already in my second command. Did I do something wrong?
Saving the data inside the container is for the ready-to-use out-of-the-box experience. It is fine if it is not as I don't mind executing a sql script afterwards.

[thaJeztah]

Yes, I noticed that you used bind-mounted volumes in your first part. You weren't using them in the "long" example.

I also just noticed you're starting the container with -it (interactive), I take it this is just for debugging here? Because once you CTRL+C, MySQL (and the container) will stop.

I think something like this should work;

docker run -d -h mysql --name mysqlt1 -v $PWD/var/lib/mysql:/var/lib/mysql -v $PWD/etc/mysql:/etc/mysql -e MYSQL_ROOT_PASSWORD=test mysql:5.6.22

#once mysql is ready, stop and remove the container
docker stop mysqlt1
docker rm mysqlt1

# and start a new container using the same bind-mounts
docker run -d -h mysql --name mysqlt2 -v $PWD/var/lib/mysql:/var/lib/mysql -v $PWD/etc/mysql:/etc/mysql -e MYSQL_ROOT_PASSWORD=test mysql:5.6.22

Untested, just "from the top of my head".

To export/import your database, here's some (unfinished) notes I made a while ago that may be useful;
https://gist.github.com/thaJeztah/8d0e901bd21329d80cf2

[skwokmag]

Thanks again for your quick reply. I just tried the following. Would you know why I got the error regarding running mysqld as root?

$ docker run -it -h mysql --name mysqltmount_var_etc -v $PWD/mysql/var/lib/mysql:/var/lib/mysql -v $PWD/mysql/etc/mysql:/etc/mysql -e MYSQL_ROOT_PASSWORD=test -e MYSQL_USER=mmx -e MYSQL_PASSWORD=mmx -e MYSQL_DATABASE=mmx mysql:5.6.22
2015-01-28 06:47:26 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2015-01-28 06:47:26 1 [Warning] Setting lower_case_table_names=2 because file system for /var/lib/mysql/ is case insensitive
2015-01-28 06:47:26 1 [ERROR] Fatal error: Please read "Security" section of the manual to find out how to run mysqld as root!

2015-01-28 06:47:26 1 [ERROR] Aborting

2015-01-28 06:47:26 1 [Note] Binlog end
2015-01-28 06:47:26 1 [Note] mysqld: Shutdown complete

(yes, -it is only for debugging)

[thaJeztah]

Would you know why I got the error regarding running mysqld as root?

Interesting, I got the same error with mysql:5.6.22, however, running mysql:5.6(which looks to be mysql:5.6.21 mysql:5.6.21 worked fine;

update; mysql:5.6 is mysql:5.6.22, I hadn't pulled the latest version of that image. I'll update my example below.

docker run -d -h mysql --name mysqltmount_all_dir -v $PWD/mysql/var/lib/mysql:/var/lib/mysql -v $PWD/mysql/etc/mysql:/etc/mysql -e MYSQL_ROOT_PASSWORD=test -e MYSQL_USER=mmx -e MYSQL_PASSWORD=mmx -e MYSQL_DATABASE=mmx mysql:5.6.21

Checking logs of the first start;

docker logs mysqltmount_all_dir

...
2015-01-28 20:56:40 1 [Warning] No existing UUID has been found, so we assume that this is the first time that this server has been started. Generating a new UUID: 2888b34e-a730-11e4-8a66-0242ac110037.
2
....
2015-01-28 20:56:40 1 [Note] Execution of init_file '/tmp/mysql-first-time.sql' started.
2015-01-28 20:56:40 1 [Note] Execution of init_file '/tmp/mysql-first-time.sql' ended.
2015-01-28 20:56:40 1 [Note] mysqld: ready for connections.
Version: '5.6.21'  socket: '/tmp/mysql.sock'  port: 3306  MySQL Community Server (GPL)

Remove the first container and start a new one

docker stop mysqltmount_all_dir
docker rm mysqltmount_all_dir

docker run -d -h mysql --name mysqltmount_all_dir -v $PWD/mysql/var/lib/mysql:/var/lib/mysql -v $PWD/mysql/etc/mysql:/etc/mysql -e MYSQL_ROOT_PASSWORD=test -e MYSQL_USER=mmx -e MYSQL_PASSWORD=mmx -e MYSQL_DATABASE=mmx mysql:5.6.21

And check the logs of the second/new container

docker logs mysqltmount_all_dir

...

2015-01-28 20:57:37 1 [Note] Event Scheduler: Loaded 0 events
2015-01-28 20:57:37 1 [Note] mysqld: ready for connections.
Version: '5.6.21'  socket: '/tmp/mysql.sock'  port: 3306  MySQL Community Server (GPL)

So, yes, there seems to be an issue with mysql:5.6.22

[thaJeztah]

Created a separate issue for the "root" problem here: #45

[skwokmag]

Hi, thanks so much for your support. I think there may be something subtle happening here. I copied and pasted your command and it still failed for me:

$ docker run -d -h mysql --name mysqltmount_all_dir -v $PWD/mysql/var/lib/mysql:/var/lib/mysql -v $PWD/mysql/etc/mysql:/etc/mysql -e MYSQL_ROOT_PASSWORD=test -e MYSQL_USER=mmx -e MYSQL_PASSWORD=mmx -e MYSQL_DATABASE=mmx mysql:5.6
7dc5db3f8a0a34ef489765ac465e298d550b5120f81c35f68d018cca0f06602a
$ docker logs mysqltmount_all_dir
2015-01-28 07:37:04 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2015-01-28 07:37:04 1 [Warning] Setting lower_case_table_names=2 because file system for /var/lib/mysql/ is case insensitive
2015-01-28 07:37:04 1 [ERROR] Fatal error: Please read "Security" section of the manual to find out how to run mysqld as root!

2015-01-28 07:37:04 1 [ERROR] Aborting

2015-01-28 07:37:04 1 [Note] Binlog end
2015-01-28 07:37:04 1 [Note] mysqld: Shutdown complete

[yosifkit]

I am quite sure this is related to bind mounting over the config dir: 12ffb2f#commitcomment-9444466

[thaJeztah]

@skwokmag My mysql:5.6 image was outdated (updated my comment above), doing mysql:5.6.21 did work.

However, the "root" problem is because of a recent change in the MySQL image as @yosifkit mentioned. I forgot about that.

The problem occurs if you bind-mount the whole mysql configuration directory. Not doing that, will make it work also with the mysql:5.6.22 image;

 docker run -d -h mysql --name mysqltmount_all_dir -v $PWD/mysql/var/lib/mysql:/var/lib/mysql -e MYSQL_ROOT_PASSWORD=test -e MYSQL_USER=mmx -e MYSQL_PASSWORD=mmx -e MYSQL_DATABASE=mmx mysql:5.6.22

Also see the link that @yosifkit added (12ffb2f#commitcomment-9444466), that explains how to mount the configuration if you have to.

I hope you find your way around with these examples. I don't think there's an issue with the MySQL image and I suggest to close this issue (the GitHub issue tracker is not really meant as a support forum 😄)

[skwokmag]

Hi @thaJeztah , I am not sure why but it still failed for me:

skwok-wpc:docker_demo skwok$ docker rmi mysql:5.6.22
Untagged: mysql:5.6.22
skwok-wpc:docker_demo skwok$ 
skwok-wpc:docker_demo skwok$ 
skwok-wpc:docker_demo skwok$ 
skwok-wpc:docker_demo skwok$ docker rm mysqltmount_all_dir
mysqltmount_all_dir
skwok-wpc:docker_demo skwok$ docker run -it -h mysql --name mysqltmount_all_dir -v $PWD/mysql/var/lib/mysql:/var/lib/mysql -e MYSQL_ROOT_PASSWORD=test -e MYSQL_USER=mmx -e MYSQL_PASSWORD=mmx -e MYSQL_DATABASE=mmx mysql:5.6.22
Unable to find image 'mysql:5.6.22' locally
Pulling repository mysql
335228ceb173: Download complete 
511136ea3c5a: Download complete 
d0a18d3b84de: Download complete 
4d6ce913b130: Download complete 
628e63ee44af: Download complete 
ad01a54357ee: Download complete 
119481e10155: Download complete 
fb88d88b1c80: Download complete 
f53e99de0738: Download complete 
4320535b4010: Download complete 
40403c37203c: Download complete 
c9495b31f987: Download complete 
6d5b269aa9e0: Download complete 
66100f5f15c6: Download complete 
169fc3e22468: Download complete 
526f51d1feaa: Download complete 
Status: Image is up to date for mysql:5.6.22
2015-01-28 07:57:43 1 [Warning] Setting lower_case_table_names=2 because file system for /var/lib/mysql/ is case insensitive
2015-01-28 07:57:43 1 [Note] Plugin 'FEDERATED' is disabled.
mysqld: Table 'mysql.plugin' doesn't exist
2015-01-28 07:57:43 1 [ERROR] Can't open the mysql.plugin table. Please run mysql_upgrade to create it.
2015-01-28 07:57:43 1 [Note] InnoDB: Using atomics to ref count buffer pool pages
2015-01-28 07:57:43 1 [Note] InnoDB: The InnoDB memory heap is disabled
2015-01-28 07:57:43 1 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2015-01-28 07:57:43 1 [Note] InnoDB: Memory barrier is not used
2015-01-28 07:57:43 1 [Note] InnoDB: Compressed tables use zlib 1.2.7
2015-01-28 07:57:43 1 [Note] InnoDB: Using Linux native AIO
2015-01-28 07:57:43 1 [Note] InnoDB: Not using CPU crc32 instructions
2015-01-28 07:57:43 1 [Note] InnoDB: Initializing buffer pool, size = 128.0M
2015-01-28 07:57:43 1 [Note] InnoDB: Completed initialization of buffer pool
2015-01-28 07:57:43 7f3a1b77d720  InnoDB: Operating system error number 13 in a file operation.
InnoDB: The error means mysqld does not have the access rights to
InnoDB: the directory.
2015-01-28 07:57:43 7f3a1b77d720  InnoDB: Operating system error number 13 in a file operation.
InnoDB: The error means mysqld does not have the access rights to
InnoDB: the directory.
2015-01-28 07:57:43 1 [ERROR] InnoDB: Creating or opening ./ibdata1 failed!
2015-01-28 07:57:43 1 [ERROR] InnoDB: Could not open or create the system tablespace. If you tried to add new data files to the system tablespace, and it failed here, you should now edit innodb_data_file_path in my.cnf back to what it was, and remove the new ibdata files InnoDB created in this failed attempt. InnoDB only wrote those files full of zeros, but did not yet use them in any way. But be careful: do not remove old data files which contain your precious data!
2015-01-28 07:57:43 1 [ERROR] Plugin 'InnoDB' init function returned error.
2015-01-28 07:57:43 1 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
2015-01-28 07:57:43 1 [ERROR] Unknown/unsupported storage engine: InnoDB
2015-01-28 07:57:43 1 [ERROR] Aborting

2015-01-28 07:57:43 1 [Note] Binlog end
2015-01-28 07:57:43 1 [Note] Shutting down plugin 'partition'
2015-01-28 07:57:43 1 [Note] Shutting down plugin 'BLACKHOLE'
2015-01-28 07:57:43 1 [Note] Shutting down plugin 'ARCHIVE'
2015-01-28 07:57:43 1 [Note] Shutting down plugin 'PERFORMANCE_SCHEMA'
2015-01-28 07:57:43 1 [Note] Shutting down plugin 'INNODB_SYS_DATAFILES'
2015-01-28 07:57:43 1 [Note] Shutting down plugin 'INNODB_SYS_TABLESPACES'
2015-01-28 07:57:43 1 [Note] Shutting down plugin 'INNODB_SYS_FOREIGN_COLS'
2015-01-28 07:57:43 1 [Note] Shutting down plugin 'INNODB_SYS_FOREIGN'
2015-01-28 07:57:43 1 [Note] Shutting down plugin 'INNODB_SYS_FIELDS'
2015-01-28 07:57:43 1 [Note] Shutting down plugin 'INNODB_SYS_COLUMNS'
2015-01-28 07:57:43 1 [Note] Shutting down plugin 'INNODB_SYS_INDEXES'
2015-01-28 07:57:43 1 [Note] Shutting down plugin 'INNODB_SYS_TABLESTATS'
2015-01-28 07:57:43 1 [Note] Shutting down plugin 'INNODB_SYS_TABLES'
2015-01-28 07:57:43 1 [Note] Shutting down plugin 'INNODB_FT_INDEX_TABLE'
2015-01-28 07:57:43 1 [Note] Shutting down plugin 'INNODB_FT_INDEX_CACHE'
2015-01-28 07:57:43 1 [Note] Shutting down plugin 'INNODB_FT_CONFIG'
2015-01-28 07:57:43 1 [Note] Shutting down plugin 'INNODB_FT_BEING_DELETED'
2015-01-28 07:57:43 1 [Note] Shutting down plugin 'INNODB_FT_DELETED'
2015-01-28 07:57:43 1 [Note] Shutting down plugin 'INNODB_FT_DEFAULT_STOPWORD'
2015-01-28 07:57:43 1 [Note] Shutting down plugin 'INNODB_METRICS'
2015-01-28 07:57:43 1 [Note] Shutting down plugin 'INNODB_BUFFER_POOL_STATS'
2015-01-28 07:57:43 1 [Note] Shutting down plugin 'INNODB_BUFFER_PAGE_LRU'
2015-01-28 07:57:43 1 [Note] Shutting down plugin 'INNODB_BUFFER_PAGE'
2015-01-28 07:57:43 1 [Note] Shutting down plugin 'INNODB_CMP_PER_INDEX_RESET'
2015-01-28 07:57:43 1 [Note] Shutting down plugin 'INNODB_CMP_PER_INDEX'
2015-01-28 07:57:43 1 [Note] Shutting down plugin 'INNODB_CMPMEM_RESET'
2015-01-28 07:57:43 1 [Note] Shutting down plugin 'INNODB_CMPMEM'
2015-01-28 07:57:43 1 [Note] Shutting down plugin 'INNODB_CMP_RESET'
2015-01-28 07:57:43 1 [Note] Shutting down plugin 'INNODB_CMP'
2015-01-28 07:57:43 1 [Note] Shutting down plugin 'INNODB_LOCK_WAITS'
2015-01-28 07:57:43 1 [Note] Shutting down plugin 'INNODB_LOCKS'
2015-01-28 07:57:43 1 [Note] Shutting down plugin 'INNODB_TRX'
2015-01-28 07:57:43 1 [Note] Shutting down plugin 'MEMORY'
2015-01-28 07:57:43 1 [Note] Shutting down plugin 'CSV'
2015-01-28 07:57:43 1 [Note] Shutting down plugin 'MRG_MYISAM'
2015-01-28 07:57:43 1 [Note] Shutting down plugin 'MyISAM'
2015-01-28 07:57:43 1 [Note] Shutting down plugin 'sha256_password'
2015-01-28 07:57:43 1 [Note] Shutting down plugin 'mysql_old_password'
2015-01-28 07:57:43 1 [Note] Shutting down plugin 'mysql_native_password'
2015-01-28 07:57:43 1 [Note] Shutting down plugin 'binlog'
2015-01-28 07:57:43 1 [Note] mysqld: Shutdown complete

skwok-wpc:docker_demo skwok$

[thaJeztah]

I think your problem there is running the container interactively and with a TTY (-it), try doing the same, but "detached"; docker run -d -h mysql ......

(at least, I got the same result when running with -it, but it ran fine detached (-d))

[skwokmag]

Hi @thaJeztah ,

This is what I got with using '-d' instead of '-it':

skwok-wpc:docker_demo skwok$ docker run -d -h mysql --name mysqltmount_all_dir -v $PWD/mysql/var/lib/mysql:/var/lib/mysql -e MYSQL_ROOT_PASSWORD=test -e MYSQL_USER=mmx -e MYSQL_PASSWORD=mmx -e MYSQL_DATABASE=mmx mysql:5.6.22
e931bdff3b0105ae4cecdd8aa061c7825232e106e903f390db53ca560228867f
skwok-wpc:docker_demo skwok$ docker logs mysql
2015/01/28 14:01:12 Error response from daemon: No such container: mysql
skwok-wpc:docker_demo skwok$ docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
skwok-wpc:docker_demo skwok$ 

[thaJeztah]

because you named your container mysqltmount_all_dir?

docker logs mysqltmount_all_dir should give you the logs.

If you're running on a host with limited memory, and the container stops during the startup, you may have to enable swap. See this ticket for more information on that; #3

[skwokmag]

Hi @thaJeztah , yes you are right, I used a wrong container name there. I have corrected that and here is the new screen output:

skwok-wpc:docker_demo skwok$ docker run -d -h mysql --name mysqltmount_all_dir -v $PWD/mysql/var/lib/mysql:/var/lib/mysql -e MYSQL_ROOT_PASSWORD=test -e MYSQL_USER=mmx -e MYSQL_PASSWORD=mmx -e MYSQL_DATABASE=mmx mysql:5.6.22
7035f582d6ebb55dd0bf0f824e86bf42aea99053b769a44e400bda7dd03ee80c
skwok-wpc:docker_demo skwok$ docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
skwok-wpc:docker_demo skwok$ docker logs mysqltmount_all_dir
2015-01-29 01:13:10 1 [Warning] Setting lower_case_table_names=2 because file system for /var/lib/mysql/ is case insensitive
2015-01-29 01:13:10 1 [Note] Plugin 'FEDERATED' is disabled.
mysqld: Table 'mysql.plugin' doesn't exist
2015-01-29 01:13:10 1 [ERROR] Can't open the mysql.plugin table. Please run mysql_upgrade to create it.
2015-01-29 01:13:10 1 [Note] InnoDB: Using atomics to ref count buffer pool pages
2015-01-29 01:13:10 1 [Note] InnoDB: The InnoDB memory heap is disabled
2015-01-29 01:13:10 1 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2015-01-29 01:13:10 1 [Note] InnoDB: Memory barrier is not used
2015-01-29 01:13:10 1 [Note] InnoDB: Compressed tables use zlib 1.2.7
2015-01-29 01:13:10 1 [Note] InnoDB: Using Linux native AIO
2015-01-29 01:13:10 1 [Note] InnoDB: Not using CPU crc32 instructions
2015-01-29 01:13:10 1 [Note] InnoDB: Initializing buffer pool, size = 128.0M
2015-01-29 01:13:10 1 [Note] InnoDB: Completed initialization of buffer pool
2015-01-29 01:13:10 7ffd9b922720  InnoDB: Operating system error number 13 in a file operation.
InnoDB: The error means mysqld does not have the access rights to
InnoDB: the directory.
2015-01-29 01:13:10 7ffd9b922720  InnoDB: Operating system error number 13 in a file operation.
InnoDB: The error means mysqld does not have the access rights to
InnoDB: the directory.
2015-01-29 01:13:10 1 [ERROR] InnoDB: Creating or opening ./ibdata1 failed!
2015-01-29 01:13:10 1 [ERROR] InnoDB: Could not open or create the system tablespace. If you tried to add new data files to the system tablespace, and it failed here, you should now edit innodb_data_file_path in my.cnf back to what it was, and remove the new ibdata files InnoDB created in this failed attempt. InnoDB only wrote those files full of zeros, but did not yet use them in any way. But be careful: do not remove old data files which contain your precious data!
2015-01-29 01:13:10 1 [ERROR] Plugin 'InnoDB' init function returned error.
2015-01-29 01:13:10 1 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
2015-01-29 01:13:10 1 [ERROR] Unknown/unsupported storage engine: InnoDB
2015-01-29 01:13:10 1 [ERROR] Aborting

2015-01-29 01:13:10 1 [Note] Binlog end
2015-01-29 01:13:10 1 [Note] Shutting down plugin 'partition'
2015-01-29 01:13:10 1 [Note] Shutting down plugin 'BLACKHOLE'
2015-01-29 01:13:10 1 [Note] Shutting down plugin 'ARCHIVE'
2015-01-29 01:13:10 1 [Note] Shutting down plugin 'PERFORMANCE_SCHEMA'
2015-01-29 01:13:10 1 [Note] Shutting down plugin 'INNODB_SYS_DATAFILES'
2015-01-29 01:13:10 1 [Note] Shutting down plugin 'INNODB_SYS_TABLESPACES'
2015-01-29 01:13:10 1 [Note] Shutting down plugin 'INNODB_SYS_FOREIGN_COLS'
2015-01-29 01:13:10 1 [Note] Shutting down plugin 'INNODB_SYS_FOREIGN'
2015-01-29 01:13:10 1 [Note] Shutting down plugin 'INNODB_SYS_FIELDS'
2015-01-29 01:13:10 1 [Note] Shutting down plugin 'INNODB_SYS_COLUMNS'
2015-01-29 01:13:10 1 [Note] Shutting down plugin 'INNODB_SYS_INDEXES'
2015-01-29 01:13:10 1 [Note] Shutting down plugin 'INNODB_SYS_TABLESTATS'
2015-01-29 01:13:10 1 [Note] Shutting down plugin 'INNODB_SYS_TABLES'
2015-01-29 01:13:10 1 [Note] Shutting down plugin 'INNODB_FT_INDEX_TABLE'
2015-01-29 01:13:10 1 [Note] Shutting down plugin 'INNODB_FT_INDEX_CACHE'
2015-01-29 01:13:10 1 [Note] Shutting down plugin 'INNODB_FT_CONFIG'
2015-01-29 01:13:10 1 [Note] Shutting down plugin 'INNODB_FT_BEING_DELETED'
2015-01-29 01:13:10 1 [Note] Shutting down plugin 'INNODB_FT_DELETED'
2015-01-29 01:13:10 1 [Note] Shutting down plugin 'INNODB_FT_DEFAULT_STOPWORD'
2015-01-29 01:13:10 1 [Note] Shutting down plugin 'INNODB_METRICS'
2015-01-29 01:13:10 1 [Note] Shutting down plugin 'INNODB_BUFFER_POOL_STATS'
2015-01-29 01:13:10 1 [Note] Shutting down plugin 'INNODB_BUFFER_PAGE_LRU'
2015-01-29 01:13:10 1 [Note] Shutting down plugin 'INNODB_BUFFER_PAGE'
2015-01-29 01:13:10 1 [Note] Shutting down plugin 'INNODB_CMP_PER_INDEX_RESET'
2015-01-29 01:13:10 1 [Note] Shutting down plugin 'INNODB_CMP_PER_INDEX'
2015-01-29 01:13:10 1 [Note] Shutting down plugin 'INNODB_CMPMEM_RESET'
2015-01-29 01:13:10 1 [Note] Shutting down plugin 'INNODB_CMPMEM'
2015-01-29 01:13:10 1 [Note] Shutting down plugin 'INNODB_CMP_RESET'
2015-01-29 01:13:10 1 [Note] Shutting down plugin 'INNODB_CMP'
2015-01-29 01:13:10 1 [Note] Shutting down plugin 'INNODB_LOCK_WAITS'
2015-01-29 01:13:10 1 [Note] Shutting down plugin 'INNODB_LOCKS'
2015-01-29 01:13:10 1 [Note] Shutting down plugin 'INNODB_TRX'
2015-01-29 01:13:10 1 [Note] Shutting down plugin 'MEMORY'
2015-01-29 01:13:10 1 [Note] Shutting down plugin 'CSV'
2015-01-29 01:13:10 1 [Note] Shutting down plugin 'MRG_MYISAM'
2015-01-29 01:13:10 1 [Note] Shutting down plugin 'MyISAM'
2015-01-29 01:13:10 1 [Note] Shutting down plugin 'sha256_password'
2015-01-29 01:13:10 1 [Note] Shutting down plugin 'mysql_old_password'
2015-01-29 01:13:10 1 [Note] Shutting down plugin 'mysql_native_password'
2015-01-29 01:13:10 1 [Note] Shutting down plugin 'binlog'
2015-01-29 01:13:10 1 [Note] mysqld: Shutdown complete

skwok-wpc:docker_demo skwok$ 

Please note that I have already shared the dir in Virtual box by executing:
VBoxManage sharedfolder add boot2docker-vm -name mysqlshare -hostpath /Users/skwok/Work/sources/mmx1/docker_demo/mysql/var/lib/mysql

Would it be possible that it is related to:
http://bugs.mysql.com/bug.php?id=72658

[skwokmag]

FYI, I ran bash with the container and saw the followings (please note the group:user of the mysql dir):

root@mysql1:/# ls -la /var/lib/
total 44
drwxr-xr-x 16 root root 4096 Jan 22 01:31 .
drwxr-xr-x 18 root root 4096 Jan 22 01:31 ..
drwxr-xr-x 7 root root 4096 Jan 22 01:31 apt
drwxr-xr-x 14 root root 4096 Jan 22 01:31 dpkg
drwxr-xr-x 2 root root 4096 Jul 14 2013 initscripts
drwxr-xr-x 2 root root 4096 Jul 14 2013 insserv
drwxrwsr-x 2 libuuid libuuid 4096 Dec 11 2012 libuuid
drwxr-xr-x 2 root root 4096 Dec 24 13:41 misc
drwxr-xr-x 1 1000 staff 204 Jan 29 01:39 mysql
drwxr-xr-x 2 root root 4096 Jan 16 21:15 pam
drwxr-xr-x 2 root root 4096 Nov 23 2012 update-rc.d
drwxr-xr-x 2 root root 4096 Jul 14 2013 urandom

[thaJeztah]

The problem you're having there is most likely caused by the fact you're using the VirtualBox shared folder. Docker will always preserve the uid/gid of the host when mounting a volume. You can try to chown those before using, but even then the performance will be really bad (VirtualBox shared folders are really slow)

Anyway, these are not issues that can be solved by the MySQL image; I suggest to use a regular mounted host directory, not a VBox shared folder and browse the boot2docker issue tracker for tips on working with the VBox shares, because there are a lot of issues with those.

[mikeys]

What's the best approach for solving this? I'm simply trying to mount the data directory of the container's to the host one

[thaJeztah]

I'm simply trying to mount the data directory of the container's to the host one

Mounting the data-directory of a container to the "host" should work. But if you're using Boot2Docker and the directory on the "host" (host being the VirtualBox VM) is a VirtualBox shared folder, permissions can become tricky. At least, that's what I've heard in various places (I don't use Boot2Docker myself).

[mikeys]

I'm using Boot2Docker on OSX and from what I understand, one of the main benefits of Boot2docker is to abstract the mounting of volumes from your host machine into the docker containers through some sort of mapping between the VBox folders and your host folders. I'm not sure about the internals of such abstraction.

[caleb]

@mikeys Boot2Docker creates a VirtualBox share of /Users on your Mac to /Users on the Linux host. This is what lets you share any folder under your Mac's /Users directory with a docker container, because the paths are the same.

Really what it does is automatically mount a VirtualBox share named Users to /Users (under Linux), and it so happens that it creates a share called Users from your Mac's /Users.

The problem is that these shares are owned by the docker user (uid 1000) on in the Linux host. This means that any container that needs to write to a shared folder on your Mac will need to run its process as UID 1000.

This is only a problem with Boot2Docker because it uses the virtual box sharing, which changes the uid of the share to the docker user and you can't change this easily from Linux.

In Vagrant, I usually solved this by changing my Vagrant file to use different permissions for the virtual box share:

config.vm.synced_folder ".", "/vagrant", :mount_options => ['dmode=777,fmode=666']

Maybe you could do something like that inside of your Boot2Docker image, I haven't looked into it. I think you would make these changes to /etc/rc.d/automount-shares in the Linux VM

One way I solved this was to just set my containers up so that I could configure them with the UID I wanted them to run their processes as, so I could just set it to 1000. The downside to this is that you basically need to create your own images for everything since most containers I've found don't let you do this.

I hope this helps.

[thaJeztah]

Thanks for helping out, @caleb! Didn't have the right info at hand to give a helpful answer myself.

[caleb]

I built my own Boot2Docker iso that mounts filesystems with more liberal permissions, it was very easy:

Clone https://github.com/boot2docker/boot2docker

Edit rootfs/rootfs/etc/rc.d/automount-shares and change the line that says:

mountOptions='defaults'

to

mountOptions='defaults,dmode=777,fmode=666'

Then build the image with your existing boot2docker installation and write the iso to the current directory:

docker build -t boot2docker . && docker run --rm boot2docker > boot2docker.iso

Then you can destroy your current boot2docker vm: boot2docker stop && boot2docker delete

Then copy the new iso to your ~/.boot2docker directory: cp boot2docker.iso ~/.boot2docker

Then re-init your boot2docker VM: boot2docker init && boot2docker up

There you go, when you ssh into your boot2docker vm now, all the directories will be 777 and files will be 666.

[thaJeztah]

@caleb interesting. Given that Boot2Docker + VBox will (probably) only be used for development setups, this might actually be a nice (although duct-tape-ish) workaround that saves many people headaches.

Have you discussed that solution in the Boot2Docker repository? I think you should; to see what the maintainers think of it?

[md5]

@thaJeztah I played around with those same mountOptions a bit. The first drawback that I noticed of setting fmode=0666 was that executable files were not executable via the mount. You could use fmode=0777, but then every file was exectuable.

[thaJeztah]

The first drawback that I noticed of setting fmode=0666 was that executable files were not executable via the mount.

Ouch! Yup that's not handy.

Bummer, because that would have reduced a lot of support questions wrt b2d. Otoh, is the situation worse than the current case?

[md5]

Bummer, because that would have reduced a lot of support questions wrt b2d. Otoh, is the situation worse than the current case?

@thaJeztah if you're implying that the situation is in fact better with fmode=0777 than without it, I'd tend to agree. Then again, it seems like enough of a hack that it could have unpleasant repercussions.

[caleb]

I knew that would be a problem, but I just store uploaded files, databases, etc inside of shared volumes, so the executable bit is not important to me (maybe except for shell scripts, but you can always run them through the interpreter).

Maybe this shouldn't be a default, but I believe that for me it's better that the mysql container can write to a mounted directory.

In production you have to worry about permissions as well, but you can just chown/chmod the directories to fit your needs, whereas boot2docker's shared filesystems don't allow you do this.

And of course, the other solution is to just use data containers (that live inside the b2d VM) and samba containers to peek into them when you need to. This is something that I haven't looked into yet.

[thaJeztah]

@md5 yeah. I'm really not sure what's best here. In the end, VBox sharing in itself was added as a stop-gap solution until something better was implemented. I see numerous questions come by from people running into both performance and permission problems. It's hard to explain, especially to people not fully understanding the relation to their workstation (OS X) and the boot2docker VM, let alone volumes and containers.

So anything that would make it work better "out of the box" would be welcome. Not sure if this is the way to go either.

[arnlen]

Thanks @caleb, your solution worded for me on box ubuntu/trusty64
Added to my Vagrantfile:

config.vm.synced_folder ENV["MYSQL_PATH"], "/mysql_datadir", mount_options: ['dmode=777,fmode=666']

[michaeljs1990]

None of these are great solution to me because it requires everyone to modify the base box every time the install a new version or like just recently with docker toolbox you would have to do the above again to configure file permissions.

I just copy /usr/local/var/mysql/* into the volume i am going to mount and then chmod 664 the whole folder.

[brlo]

@skwokmag, just add --user 1000 (1000 - docker UID) like this:

docker run --name mysql -v $PWD/var/lib/mysql:/var/lib/mysql -v $PWD/etc/mysql:/etc/mysql -e MYSQL_ALLOW_EMPTY_PASSWORD=yes --user 1000 mysql:5.7.13

an you'll get this:

...
2016-07-28T10:40:31.770773Z 0 [Note] mysqld: ready for connections.
Version: '5.7.13'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  MySQL Community Server (GPL)

[thaJeztah]

Also note that the permissions issue should be solved by Docker for Mac

[sebastiannm]

Hi,
I'm using docker for Mac and still same issue: Fatal error: Please read "Security" section of the manual to find out how to run mysqld as root!

docker run --name mysqlt1 -h mysql --rm -it -v ~/mysql:/var/lib/mysql -e MYSQL_ROOT_PASSWORD=pass -e MYSQL_USER=pass -e MYSQL_PASSWORD=pass -e MYSQL_DATABASE=pass_dev mysql:5.7

[thiagoh]

@brlo that worked for me! Thanks