Could not fetch initial state with docker-compose

[matteosuppo]

I'm trying to use hydra with docker-compose.

version: '2'

services:
  consent:
    build: .
    links:
      - hydra
      - rethinkdb:database
    ports:
      - "3000:3000"
    depends_on:
      - rethinkdb
    restart: unless-stopped

  hydra:
    image: oryam/hydra
    volumes:
      - hydravolume:/root
    links:
      - rethinkdb:database
    depends_on:
      - rethinkdb
    ports:
      - "4444:4444"
    environment:
      - SYSTEM_SECRET=${SYSTEM_SECRET}
      - CONSENT_URL=http://${DOCKER_IP}:3000
      - DATABASE_URL=rethinkdb://database:28015/hydra
    restart: unless-stopped

  rethinkdb:
      image: rethinkdb
      ports:
          - "8080:8080"
          - "28015:28015"
          - "29015:29015"

volumes:
  hydravolume:
    driver: local

When I start it with docker-compose-up the first time everything is fine but if I launch it again:

hydra_1      | time="2016-06-15T14:00:58Z" level=info msg="DATABASE_URL set, connecting to RethinkDB." 
hydra_1      | time="2016-06-15T14:00:58Z" level=info msg="Connecting with RethinkDB: rethinkdb://database:28015/hydra (database:28015) (hydra)" 
hydra_1      | time="2016-06-15T14:00:58Z" level=info msg="Connected to RethinkDB!" 
hydra_1      | time="2016-06-15T14:00:58Z" level=fatal msg="Could not fetch initial state: gorethink: Cannot perform read: primary replica for shard [\"\", +inf) not available in: \nr.Table(\"hydra_policies\")" 
hydra_1      | Pointing cluster at https://localhost:4444

Because I put restart: unless-stopped it stops and starts a few times before:

hydra_1      | time="2016-06-15T14:02:25Z" level=info msg="DATABASE_URL set, connecting to RethinkDB." 
hydra_1      | time="2016-06-15T14:02:25Z" level=info msg="Connecting with RethinkDB: rethinkdb://database:28015/hydra (database:28015) (hydra)" 
hydra_1      | time="2016-06-15T14:02:25Z" level=info msg="Connected to RethinkDB!" 
hydra_1      | time="2016-06-15T14:02:25Z" level=fatal msg="Could not fetch initial state: cipher: message authentication failed" 

[aeneasr]

Hey thanks for the report. I believe that this is caused by not enough wait time. I will investigate and let you know once I found the issue.

[aeneasr]

try replacing https://github.com/ory-am/hydra/blob/master/Dockerfile-dangerous#L10 with ENTRYPOINT sleep 10; /go/bin/hydra host --dangerous-auto-logon and rerun docker-compose build and docker-compose up

which system are you on? windows? linux? mac?

[aeneasr]

i cannot reproduce the issue on my environment

[aeneasr]

Ok the problem is that you are not using the right dockerfile. Dockerfile does not have any delays, which is why docker-compose.yml uses Dockerfile-dangerous instead:

  hydra:
    build:
      context: .
      dockerfile: Dockerfile-dangerous

This causes hydra to try and connect immediately to rethinkdb but rethinkdb hasn't started up yet properly (but is listening on the port). This causes the shard failure

[aeneasr]

On second thought, this should be handled better by hydra. Instead of dying there should be retry attempts.

[matteosuppo]

If I set it to wait for 10 seconds it avoids the error gorethink: Cannot perform read: primary replica for shard but I still get cipher: message authentication failed

Now I'll try cleaning everything up

[aeneasr]

Ok I just removed the wait time from Dockerfile-dangerous and introduced restart: unless-stopped in the hydra section of docker-compose.yml. Everything worked fine then.

The cipher: message authentication failed might have to do something with a missing or wrong system secret. Have you set the system secret using SYSTEM_SECRET=<some-supersecret-password>? For example:

SYSTEM_SECRET=passwordtutorial DOCKER_IP=$(docker-machine ip default) docker-compose up

If you have troubles try running docker-compose rm -f and docker-compose up. Don't forget to set the environment variables as mentioned above

[aeneasr]

the message auth failed error is now tracked as #104

[matteosuppo]

Ah now I get it. The first time I set the SYSTEM_SECRET but after that I didn't, so it generated a different secret and it didn't work. Thank you very much.

[aeneasr]

Perfect, glad I could help! The system secret is not stored by hydra, it is up to the environment to provide it :) this reduces risks on database breaches and similar issues.