New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow creating multiple databases on startup. #18

Closed
wants to merge 1 commit into
base: master
from

Conversation

Projects
None yet
@tonio-ramirez
Copy link

tonio-ramirez commented Oct 21, 2014

This can be useful in some cases. For example, if you're auditing entities, and want to store the audit information on a separate database.

@yosifkit

This comment has been minimized.

Copy link
Member

yosifkit commented Oct 21, 2014

I like it (as long as we go document how to do it with the proper syntax in docker run).

@yosifkit

This comment has been minimized.

Copy link
Member

yosifkit commented Oct 21, 2014

Ping @tianon

@tianon

This comment has been minimized.

Copy link
Member

tianon commented Oct 23, 2014

Seems reasonable, since database names can't have spaces in them.

@thaJeztah

This comment has been minimized.

Copy link

thaJeztah commented Oct 23, 2014

since database names can't have spaces in them.

Well, in fact, they can: docs. TL;DR MySQL supports any Unicode character when quoted using back ticks. Not saying it's good practice, but it is supported.

@tianon

This comment has been minimized.

Copy link
Member

tianon commented Oct 23, 2014

I'm not sure I'm comfortable with our script supporting such an obscene
practice! :)

(but really only because I don't have a better idea for making this work
cleanly with just environment variables :P)

@thaJeztah

This comment has been minimized.

Copy link

thaJeztah commented Oct 23, 2014

@tianon yup agree, maybe document that it isn't supported is sufficient? Eg "known issues" or just "not supported".

Just to prevent support questions 😄

@md5

This comment has been minimized.

Copy link

md5 commented Oct 24, 2014

Perhaps an approach like the /docker-entrypoint-initdb.d/ thing we did in the Postgres image would be appropriate: docker-library/postgres#23

I was just looking at this entrypoint and comparing it to https://github.com/cpuguy83/docker-mysql while thinking about how to make a simple, replication capable MySQL container for testing purposes.

My first thought was that it would be nice to be able to add to TEMP_FILE; seeing this PR, it seems that the entrypoint.d approach would allow additional DBs to be created just as easily.

@md5

This comment has been minimized.

Copy link

md5 commented Oct 25, 2014

Actually, it looks like mysqld has a --bootstrap option that might be a better approach than the current --init-file approach.

I got hip to --bootstrap in the GoogleCloudPlatform/compute-docker-mysql repo, which shows how to use it: https://github.com/GoogleCloudPlatform/compute-docker-mysql/blob/master/run-mysqld#L22-L35

@thaJeztah

This comment has been minimized.

Copy link

thaJeztah commented Oct 25, 2014

Nice find on the --bootstrap, didn't know that one

@md5

This comment has been minimized.

Copy link

md5 commented Oct 25, 2014

Unfortunately, after playing with it last night, I found that --bootstrap implies --skip-grant-tables, which makes the GRANT ALL statements break.

It's possible to work around this by manipulating the mysql.user table directly in the case of the GRANT ALL *.* statement and by manipulating mysql.db in the case of the GRANT ALL ${MYSQL_DATABASE}.* statement, but that sounds like a recipe for future breakage. I also discovered that GRANT ALL statements work correctly with --bootstrap/--skip-grant-tables in MariaDB, but that doesn't help with MySQL.

@thaJeztah

This comment has been minimized.

Copy link

thaJeztah commented Oct 25, 2014

Maybe we should just deprecate MySQL and go for MariaDB 😄

@md5

This comment has been minimized.

Copy link

md5 commented Oct 25, 2014

As long as the image is still called "mysql".

@yosifkit

This comment has been minimized.

Copy link
Member

yosifkit commented Oct 27, 2014

Oh, that sounds fun, though I think the MySQL people might get a bit grumpy if we just drop it. We could do mysql:maria or mysql:mariadb and then change latest to be MariaDB, but that may create inconsistency. (We should probably open a separate issue about adding/migrating to Maria, since that is not the purpose of this PR)

@md5

This comment has been minimized.

Copy link

md5 commented Oct 27, 2014

I thought we were joking!

I see no problem having a maria image though.

@thaJeztah

This comment has been minimized.

Copy link

thaJeztah commented Oct 27, 2014

Yeah, thought we were joking as well, but adding an official MariaDB image to the repository might be a good thing (haven't checked if an official one exists to be honest)

@md5

This comment has been minimized.

Copy link

md5 commented Oct 27, 2014

Looks like @tianon has an unofficial image here: https://github.com/tianon/dockerfiles/tree/master/mariadb

@thaJeztah

This comment has been minimized.

Copy link

thaJeztah commented Oct 27, 2014

There are others as well probably, but no (semi) official?

@yosifkit

This comment has been minimized.

Copy link
Member

yosifkit commented Oct 27, 2014

Nothing in the Official Images (previously know as "stackbrew").

@tianon

This comment has been minimized.

Copy link
Member

tianon commented Nov 3, 2014

Totally off-topic, but I agree that MariaDB would need to be separate.
It's a fork of MySQL, and MySQL is definitely still a thing, so it would be
dishonest IMO to call "MariaDB" pure "MySQL".

@thaJeztah

This comment has been minimized.

Copy link

thaJeztah commented Nov 3, 2014

@tianon reading back my comment; I meant to say;
"Adding MariaDB to the registry", didn't mean this repository;
they're clearly a different thing, different maintainers so shouldn't
be mixed up

@yosifkit

This comment has been minimized.

Copy link
Member

yosifkit commented Nov 3, 2014

Back on topic, I like the "create multiple databases", though we should now add it to 5.6 as well, and then add docs to tell users how to use it (and not allow spaces in db names).

@tianon?

@yosifkit yosifkit referenced this pull request Nov 3, 2014

Closed

Make a MariaDB image #294

@yosifkit

This comment has been minimized.

Copy link
Member

yosifkit commented Nov 3, 2014

Off topic: we need MariaDB. docker-library/official-images#294

@tianon

This comment has been minimized.

Copy link
Member

tianon commented Nov 3, 2014

Yeah, SGTM

@thaJeztah

This comment has been minimized.

Copy link

thaJeztah commented Nov 3, 2014

on topic; I like the "create multiple databases" as well. Is this all working correctly (looking back at the --bootstrap sidesteps?)

@yosifkit

This comment has been minimized.

Copy link
Member

yosifkit commented Nov 4, 2014

I think it should work fine in --init-file, as long and the format with newlines is correct. This way we still get to use GRANT rather than mucking with the mysql.user table directly.

williamsbdev added a commit to williamsbdev/mysql that referenced this pull request Dec 3, 2014

Feature: adding hook to execute additional scripts on startup
This will close both docker-library#18 and docker-library#24.

The scripts are located in the /docker-entrypoint-initdb.d directory.

Paired on this with @benkiefer
@tfwright

This comment has been minimized.

Copy link

tfwright commented Aug 19, 2015

Hi all, I'm new to docker and trying out this image and I ran into a need to multiple databases. I've been sifting through this and other linked issues and for the life of me I can't parse what was settled on for best practice here. Do I need a new dockerfile that extends this one or can I specify multiple databases (or initdb scripts) in docker-compose.yml? Or something else entirely? Thanks for any guidance you can offer!

@yosifkit

This comment has been minimized.

Copy link
Member

yosifkit commented Aug 19, 2015

@tfwright, you can have any number of initdb sql scripts, so that should be the easiest way to create multiple databases.

@tfwright

This comment has been minimized.

Copy link

tfwright commented Aug 19, 2015

@yosifkit Thanks. To confirm, right now I'm just using the image in my compose config, so I'll need to change that to point to a custom dockerfile, right?

@yosifkit

This comment has been minimized.

Copy link
Member

yosifkit commented Aug 19, 2015

@tfwright, not necessarily, you could just have the sql file with the database creation in the directory with the compose file and add a volume directive to the yaml with ./create-databases.sql:/docker-entrypoint-initdb.d/. I think even ./*.sql:/docker-entrypoint-initdb.d/ should work if you have multiple scripts or even just put them in a folder: ./sql-setup/:/docker-entrypoint-initdb.d/.

@tfwright

This comment has been minimized.

Copy link

tfwright commented Aug 19, 2015

@yosifkit I tried adding a simple script and volume directive

#   new file:   src/sql-setup.sql
@@ -15,6 +15,7 @@ mysql:
   volumes:
      - "./src/my.cnf:/etc/mysql/conf.d/z_my.cnf"
      - /var/lib/mysql
+     - "./src/sql-setup.sql:/docker-entrypoint-initdb.d/"

but docker complains when trying to start the container

Recreating 1632cce382_spectre_mysql_1...
Cannot start container 45b56877b97225ccd74c06bd3bfe09c7c06161b099922889ba0d4ba09bb0ad77: [8] System error: not a directory

What did I miss?

@tfwright

This comment has been minimized.

Copy link

tfwright commented Aug 19, 2015

I was able to fix the above by changing the volume directive to "./src/sql-setup.sql:/docker-entrypoint-initdb.d/sql-setup.sql" which doesn't jive with what I understand about docker's syntax...but who cares? It works!

Thanks again!

@paulredmond

This comment has been minimized.

Copy link

paulredmond commented Oct 8, 2015

@tfwright not sure if there is a better way, but your suggestion worked for me ❤️

@solars

This comment has been minimized.

Copy link

solars commented Dec 18, 2015

Came across this ticket, but saw that it is still open. I'm using a mysql container for rails development like this:

docker run --name cpc-mysql -p 127.0.0.1:3307:3306 -e MYSQL_RANDOM_ROOT_PASSWORD=true -e MYSQL_USER=google_report -e MYSQL_PASSWORD=asdf -e MYSQL_DATABASE=app-dev -d mysql

I would need a second database app-test for testing, what is currently the cleanest way to do this?

@thaJeztah

This comment has been minimized.

Copy link

thaJeztah commented Dec 18, 2015

@solars have you tried adding .sql or .sh files to the docker-entrypoint-initdb.d directory (or mounting a volume in that location)? #90

@solars

This comment has been minimized.

Copy link

solars commented Dec 18, 2015

Thanks for the pointer, I didn't see this option yet! In this case I will create a clean Dockerfile and add sql as needed. Thanks a lot!

@solars

This comment has been minimized.

Copy link

solars commented Dec 18, 2015

@thaJeztah one more thing, I read here (first answer): http://stackoverflow.com/questions/32482780/how-to-create-populated-mysql-docker-image-on-build-time

that in the Dockerfile (build) it doesn't work - is this still true? If so, how should I do it?

@thaJeztah

This comment has been minimized.

Copy link

thaJeztah commented Dec 18, 2015

@solars correct; it's possible, but the VOLUME declaration in the Dockerfile complicates things. (also see moby/moby#8177, moby/moby#18286, moby/moby#18287)

@solars

This comment has been minimized.

Copy link

solars commented Dec 18, 2015

@thaJeztah so I need to specify it in the run command above? or what is the suggested way to do it?

@yosifkit

This comment has been minimized.

Copy link
Member

yosifkit commented Feb 24, 2016

Agreeing with #86 (comment), I think this is solved. Comment if you feel there is more to discuss.

@yosifkit yosifkit closed this Feb 24, 2016

@Glideh

This comment has been minimized.

Copy link

Glideh commented Jul 13, 2016

So how do we do multiple databases after all ?
Separate container ? Seems to need quite a lot of config.
MYSQL_DATABASE space/coma separated ? Would be handy.

@yosifkit

This comment has been minimized.

Copy link
Member

yosifkit commented Jul 13, 2016

@Glideh, either a new container or add a script to /docker-entrypoint-initdb.d/. An .sql file would be the simplest, I think.

rlucioni added a commit to edx/devstack that referenced this pull request Dec 9, 2016

Add SQL for creating users and databases
The included script is idempotent, executed when the mysql container is initialized. This approach is borrowed from docker-library/mysql#18.

ECOM-6563

rlucioni added a commit to edx/devstack that referenced this pull request Dec 9, 2016

Add SQL for creating users and databases
The included script is idempotent, executed when the mysql container is initialized. This approach is borrowed from docker-library/mysql#18.

ECOM-6563
@skytreader

This comment has been minimized.

Copy link

skytreader commented Feb 19, 2017

I'm trying to follow the discussion here but it seems that the script in docker-entrypoint-initdb.d is not being run at all.

My Dockerfile is as follows:

FROM ubuntu:16.04
ADD . ./app
WORKDIR ./app

RUN apt-get update
RUN apt-get install -y python python-pip
RUN pip install -r requirements.txt

Mounted in docker-compose.yml via

version: '2'
services:
  db:
    image: "mysql:5.7"
    environment:
      - MYSQL_ALLOW_EMPTY_PASSWORD=yes
    ports:
      - "3308:3306"
    volumes:
      - "./docker-setup/init.sql:/docker-entrypoint-initdb.d/init.sql"
  web:
    build: .
    command: python run.py
    ports:
      - "80:7070"
    volumes:
      - .:/app
    depends_on:
      - db

With init.sql containing

CREATE DATABASE IF NOT EXISTS app DEFAULT CHARACTER SET = utf-8;
CREATE DATABASE IF NOT EXISTS app_test DEFAULT CHARACTER SET = utf-8;

However, when I run docker-compose up, I ultimately end up with the error:

web_1  | OperationalError: (_mysql_exceptions.OperationalError) (1049, "Unknown database 'app'")

To be clear, using MYSQL_DATABASE env var allows my web and db to run harmoniously. It's just that, if possible, I want to run app and app_test on the same MySQL service.

Anything I'm doing wrong?

@mrts

This comment has been minimized.

Copy link

mrts commented Jul 30, 2017

Anything I'm doing wrong?

Yes, you are attempting to mount files as volumes.

Here's a working example for PosgreSQL that has similar structure:

myapp-postgresql:
    image: postgres:9.6.2
    volumes:
        - ../docker-posgresql-multi-db:/docker-entrypoint-initdb.d
    environment:
        - POSTGRES_MULTIPLE_DATABASES=db1,db2
        - POSTGRES_USER=myapp
        - POSTGRES_PASSWORD=

The docker-posgresql-multi-db directory contains a script for creating multiple databases, see https://github.com/mrts/docker-posgresql-multiple-databases for the script that you can reuse.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment