docker-compose only runs first time

[rduivenvoorde]

Tried to run 'docker-compose up' and 'docker-compose up -d'.

First time all runs (except with an css exception that the page does not look right), but after stopping the container, removing and start and instance again there is an error with the database not starting.

without -d option you have the following in terminal:

 # second run
me@server:~/Rocket.Chat$ docker-compose up
Creating rocketchat_db_1
Creating rocketchat_rocketchat_1
Attaching to rocketchat_db_1, rocketchat_rocketchat_1
db_1         | 2015-11-10T19:13:31.039+0000 I JOURNAL  [initandlisten] journal dir=/data/db/journal   
db_1         | 2015-11-10T19:13:31.039+0000 I JOURNAL  [initandlisten] recover : no journal files present, no recovery needed
rocketchat_1 | 
rocketchat_1 | /app/bundle/programs/server/node_modules/fibers/future.js:278
rocketchat_1 |                                          throw(ex);
rocketchat_1 |                                                ^
rocketchat_1 | Error: failed to connect to [db:27017]
rocketchat_1 |     at Object.Future.wait (/app/bundle/programs/server/node_modules/fibers/future.js:398:15)
rocketchat_1 |     at new MongoConnection (packages/mongo/mongo_driver.js:213:1)
rocketchat_1 |     at new MongoInternals.RemoteCollectionDriver (packages/mongo/remote_collection_driver.js:4:1)
rocketchat_1 |     at Object. (packages/mongo/remote_collection_driver.js:38:1)
rocketchat_1 |     at Object.defaultRemoteCollectionDriver (packages/underscore/underscore.js:750:1)  
rocketchat_1 |     at new Mongo.Collection (packages/mongo/collection.js:102:1)
rocketchat_1 |     at AccountsServer.AccountsCommon (accounts_common.js:23:18)
rocketchat_1 |     at new AccountsServer (accounts_server.js:16:5)
rocketchat_1 |     at Package (globals_server.js:5:12)
rocketchat_1 |     at /app/bundle/programs/server/packages/accounts-base.js:1814:4
rocketchat_1 |     - - - - -
rocketchat_1 |     at [object Object]. (/app/bundle/programs/server/npm/npm-mongo/node_modules/mongodb/lib/mongodb/connection/server.js:556:74)
rocketchat_1 |     at [object Object].emit (events.js:106:17)
rocketchat_1 |     at [object Object]. (/app/bundle/programs/server/npm/npm-mongo/node_modules/mongodb/lib/mongodb/connection/connection_pool.js:156:15)
rocketchat_1 |     at [object Object].emit (events.js:98:17)
rocketchat_1 |     at Socket. (/app/bundle/programs/server/npm/npm-mongo/node_modules/mongodb/lib/mongodb/connection/connection.js:534:10)
rocketchat_1 |     at Socket.emit (events.js:95:17)
rocketchat_1 |     at net.js:441:14
rocketchat_1 |     at process._tickCallback (node.js:448:13)
rocketchat_rocketchat_1 exited with code 8
db_1         | 2015-11-10T19:13:34.034+0000 I JOURNAL  [initandlisten] preallocateIsFaster=true 39.64 

If i stop it again, and run it with -d option it also fails, showing:

me@server$ docker-compose ps
         Name                        Command               State      Ports   
-----------------------------------------------------------------------------
rocketchat_db_1           /entrypoint.sh mongod --sm ...   Up       27017/tcp 
rocketchat_rocketchat_1   node main.js                     Exit 8        

Running on Debian stable server behind apache reverse proxy. DNS-record made for chat.ourdomain.org so in docker-compose.yml it says:

ROOT_URL=http://chat.qgis.org

Any help appreciated, thanks.

[timlinux]

I think it is just a sequencing issue:

docker-compose up -d db
Creating rocketchat_db_1
timlinux@qgis2:~/rocket.chat$ docker-compose logs db
Attaching to rocketchat_db_1
db_1 | 2015-11-10T20:52:47.421+0000 I JOURNAL  [initandlisten] journal dir=/data/db/journal
db_1 | 2015-11-10T20:52:47.421+0000 I JOURNAL  [initandlisten] recover : no journal files present, no recovery needed
db_1 | 2015-11-10T20:52:50.106+0000 I JOURNAL  [initandlisten] preallocateIsFaster=true 30.86
db_1 | 2015-11-10T20:52:52.622+0000 I JOURNAL  [initandlisten] preallocateIsFaster=true 30.02
db_1 | 2015-11-10T20:52:56.230+0000 I JOURNAL  [initandlisten] preallocateIsFaster=true 31.84
db_1 | 2015-11-10T20:52:56.231+0000 I JOURNAL  [initandlisten] preallocateIsFaster check took 8.809 secs
db_1 | 2015-11-10T20:52:56.231+0000 I JOURNAL  [initandlisten] preallocating a journal file /data/db/journal/prealloc.0
db_1 | 2015-11-10T20:52:57.267+0000 I JOURNAL  [initandlisten] preallocating a journal file /data/db/journal/prealloc.1
db_1 | 2015-11-10T20:52:58.844+0000 I JOURNAL  [initandlisten] preallocating a journal file /data/db/journal/prealloc.2
db_1 | 2015-11-10T20:52:59.975+0000 I JOURNAL  [durability] Durability thread started
db_1 | 2015-11-10T20:52:59.975+0000 I JOURNAL  [journal writer] Journal writer thread started
db_1 | 2015-11-10T20:53:00.027+0000 I CONTROL  [initandlisten] MongoDB starting : pid=1 port=27017 dbpath=/data/db 64-bit host=c6d5fb35a0ba
db_1 | 2015-11-10T20:53:00.027+0000 I CONTROL  [initandlisten] 
db_1 | 2015-11-10T20:53:00.027+0000 I CONTROL  [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/defrag is 'always'.
db_1 | 2015-11-10T20:53:00.027+0000 I CONTROL  [initandlisten] **        We suggest setting it to 'never'
db_1 | 2015-11-10T20:53:00.027+0000 I CONTROL  [initandlisten] 
db_1 | 2015-11-10T20:53:00.027+0000 I CONTROL  [initandlisten] db version v3.0.7
db_1 | 2015-11-10T20:53:00.027+0000 I CONTROL  [initandlisten] git version: 6ce7cbe8c6b899552dadd907604559806aa2e9bd
db_1 | 2015-11-10T20:53:00.027+0000 I CONTROL  [initandlisten] build info: Linux ip-10-183-78-195 3.2.0-4-amd64 #1 SMP Debian 3.2.46-1 x86_64 BOOST_LIB_VERSION=1_49
db_1 | 2015-11-10T20:53:00.027+0000 I CONTROL  [initandlisten] allocator: tcmalloc
db_1 | 2015-11-10T20:53:00.027+0000 I CONTROL  [initandlisten] options: { storage: { mmapv1: { smallFiles: true } } }
db_1 | 2015-11-10T20:53:00.028+0000 I INDEX    [initandlisten] allocating new ns file /data/db/local.ns, filling with zeroes...
db_1 | 2015-11-10T20:53:00.257+0000 I STORAGE  [FileAllocator] allocating new datafile /data/db/local.0, filling with zeroes...
db_1 | 2015-11-10T20:53:00.258+0000 I STORAGE  [FileAllocator] creating directory /data/db/_tmp
db_1 | 2015-11-10T20:53:00.321+0000 I STORAGE  [FileAllocator] done allocating datafile /data/db/local.0, size: 16MB,  took 0.026 secs
db_1 | 2015-11-10T20:53:00.336+0000 I NETWORK  [initandlisten] waiting for connections on port 27017
^CERROR: 
Aborting.

Now spin up rocket.chat:

$ docker-compose up -d rocketchat
rocketchat_db_1 is up-to-date
Creating rocketchat_rocketchat_1

[engelgabriel]

Can we add a timeout between getting the DB and the Rocket.Chat up, so the DB is ready for connections?

[timlinux]

@engelgabriel I haven't poked around in the startup script in the rocket container but you could put something like this at the start of it:

# wait for mongodb  to come up
until `nc -z db 27017`; do
    echo "$(date) - waiting for mongodb.."
    sleep 1
done
echo "mongodb ready"

Just make sure that you have netcat installed in the rocket container's Dockerfile....

[engelgabriel]

Thanks @timlinux !!!

I don't have the tools to test it here. Can anyone try this out and submit a PR please?

[geekgonecrazy]

@timlinux that's a pretty nice way of handling it.

Problem with this is, we can't assume that the container linked is going to be named db. They might name it anything.

Also what if i'm not linking but actually using a mongo instance outside of docker?

@engelgabriel seems like this should be handled in rocket.chat its self. It just attempt to connect and a retry with a backoff.

[timlinux]

@geekgonecrazy true but you already make the assumption here: https://github.com/RocketChat/Rocket.Chat/blob/master/Dockerfile#L24 - I guess that can be overridden with a -e directive when running docker. You could tweak my snippet to use and env var for the nc host probe too instead of hard coding the host.

If you can do it in rocket.chat itself so much the better...

[geekgonecrazy]

@timlinux right mongo_url can be overrode.

Ideally in rocket.chat its self. Otherwise we're going to keep adding things to the dockerfile and make it unmanagable.

[timlinux]

@geekgonecrazy agreed!

[marceloschmidt]

@geekgonecrazy I don't think that's an option, unless Meteor does it, as we have no control over the server connection code: https://github.com/meteor/meteor/blob/devel/packages%2Fmongo%2Fmongo_driver.js#L179

[Sing-Li]

docker/compose#374 😬

[Sing-Li]

Do we still want to continue to support dev-ployment, might be the real question.

Since we started using proper versioned binary builds, and stop running meteor to deploy, we've essentially stopped supporting dev-ployment. A dev is supposed to know what he/she is doing,

In production deployment (non-dev) scenarios using docker-compose.yml - this seldom happens. Mongo runs 24 x 7 and supports live backups. The operations or DevOps staff will not arbitrarily stop or kill mongo instances.

[rduivenvoorde]

Hi, personally I did want to test this for production use. But (ok, maybe just me :-) ) do some tries before it is running as you want. Eg I first tried to run it without reverse proxy in front, then with one etc. I even had to restart docker on our prod server because of the fact that the mongodb container was unstop/killable.
In short: I think even as non dev scenario, the docker one could be more robust then is currently.
Also speaking from my own experience: I was asked to test a tool as Rocket.Chat, but knowing nothing(!) about mongodb and almost nothing about node, I went for the docker way, hoping to be able to just start stop kill it a couple of times etc.
I'm in favour or tim's timer/polling scenario, while the docker project develops it's 'dependency-chain' framework.

[geekgonecrazy]

@Sing-Li awesome I hadn't seen this. Hopefully that is finished and lands in a build soon.

As far as supporting dev version. I agree that when it comes to non-versioned deployments that's up to them. We do need to update all of our documentation to steer non-devs down that path.

But in this case the image isn't a dev image its a versioned image produced from our build artifacts.

I've personally never experienced this issue. I've used docker-compose on multiple occasions and the db always seems to start in time. When it comes to deploying new versions of rocket.chat I didn't have to stop anything.

You can do:

docker pull rocketchat/rocket.chat
docker-compose up -d

And it'll automatically kill the old rocket.chat image and start with the new one. At least that has been my experience.

[Sing-Li]

Updated our documentation as per @timlinux's sequencing steps:

https://github.com/RocketChat/Rocket.Chat/wiki/Deploy-with-Docker

@rduivenvoorde - if you still have problems following the new instructions, please re-open this issue.