Docker in TravisCI causes linking db's to fail

[inooid]

After spending two days now trying to get my docker images to run on TravisCI, I am asking for your help. After eliminating the problems that were issued here: #4778 about the iptables that failed I ran into a database connection problem that I cannot reproduce on other machines.

The repo it's about:
https://travis-ci.org/inooid/BoydDamesAPI-for-portfolio/builds/81777509

I am trying to run my specs, but it keeps failing to connect to the database. I have tried using with and without environment variables inside of my database.yml, but neither seems to work. I also tried running it on different docker-machine's, but everywhere it seems to run. I've checked the hosts file and the db hostname is properly set by docker.

Is the database container being shutdown before the docker-compose run app rake task is being called? I tried running docker-compose up -d db in a before_install hook, so I can make sure that the postgres database is running, but still no results.

[BanzaiMan]

Some users report good results with adding --privileged when are faced with some Docker problems. Perhaps that may be the case here as well.

[inooid]

@BanzaiMan I tried your solution on a different branch that I used to try docker without docker-compose on, but unfortunately still the same result.

https://travis-ci.org/inooid/BoydDamesAPI-for-portfolio/builds/82217186

Could it be a firewall issue? Does Travis automatically enforce some ruling around ports or 'external' containers (so to speak)? The linking is done properly, because the etc/hosts has records of db and the ENV variables are all properly set, because of the container linking (I checked that in previous builds).

It seems like either the db container gets shut down as soon as the tests start or the connection is blocked by some sort of firewall/external IP that can't be accessed. Thoughts?

[plukevdh]

Running into this as well. Attempting to add the privileged: true directive does not appear to solve this. Running the same command set locally works so its specific to the Travis setup. I've tried bouncing the docker-compose setup actions into before_install, install and script block with no luck.

[chrismeyersfsu]

I'm also experiencing this issue. Can not connect to ports on the docker container.

[inooid]

I wrote a script that waits for the postgres container to be fully started and ready to use, but the instance still can't connect to the Docker db container.

I got my tests working on Codeship, but not on Travis, so it has to be something Travis related.
https://travis-ci.org/inooid/BoydDamesAPI-for-portfolio/builds/85133689

[pando85]

Same problem here, I can't get any connection to my postgres container in travis, in local test its working perfectly.
EDIT: That's not true for me, I tested that I only need to wait for postgress to start correctly.

• My result is here: https://travis-ci.org/pando85/cherrymusic
• Travis script: https://github.com/pando85/cherrymusic/blob/devel-django/.travis.yml
• Test script: https://github.com/pando85/cherrymusic/blob/devel-django/test_scripts/run_tests.sh
• Docker links: https://github.com/pando85/cherrymusic/blob/devel-django/test.yml

[carhartl]

@pando85 👍

[carhartl]

Related to the solution by @pando85 introducing a sleep to wait for the postgres container to start, I found that at times even 20 seconds weren't sufficient in my case, causing the build to error - thus I'm now using this to detect the linked container's availability:

  - |
    while ! psql --host=localhost --username=postgres -c 'SELECT 1'> /dev/null 2>&1; do
      echo 'Waiting for postgres...'
      sleep 1;
    done;

[huygn]

Ran into same issue with docker-compose: https://travis-ci.org/gnhuy91/django-tutorial-docker/builds/119759198
Database containers are up and ports got exposed correctly but my app cannot access to db's port.

[erikdstock]

same issue here: https://travis-ci.org/erikdstock/tipjar/builds/145257098
in this case i've mapped my postgres container's port to host 5433 since 5432 always says it is taken.
i've looked far and wide for an answer on this. runs fine locally.

[denvaar]

I am also running into this same problem.

[ghost]

I am too having this issue. I have two services defined in my docker-compose.yml file (an application and a database service) and my application is unable to connect to the database. docker ps shows the database service running.

Any ideas here? I'd hate to not use Travis CI because of this. Thanks!

[lhanson]

Similar trouble here, my build fails with:

docker: Error response from daemon: driver failed programming external connectivity on endpoint clever_meitner (070beca60a1b21075de9bf691cbca06993467b8bd819f07d488d94e8fe9e5d4f): Error starting userland proxy: listen tcp 0.0.0.0:5432: bind: address already in use.

Remapping the postgres port to something different seems to allow it to start up, but is not ideal.

[briankung]

I haven't tried privileged: true or the timeout yet, but I, too, am experiencing this. It's quite frustrating.

[arroyo]

I'm having this same issue as well, I've spent many hours trying to debug and tried various options I saw in other travis-ci issues. Nothing seems to work, it does look like some sort of port restrictions are being imposed within travis since the same docker scripts work just fine in other environments.

Key issue: I have a mysql container at port 3301 and an app container. I need the app container to be able to hit the mysql container. Really pulling my hair out on this one.

Adding mysql to the host script could be problematic as well since mysql would be running at 127.0.0.1 and the container would not be able to hit it.

Is my only option to create a completely self contained container that run all of the services in the same container? That should work just fine but it really goes against how to best use docker and means I have to manage a separate container just for travis.

[GFite]

Patiently waiting on a solution! Would really like to run django test (manage.py test) in Travis without changing docker-compose.yml or the config that works locally and up on aws.

Django test attempts to create test database for alias 'default', but

_mysql_exceptions.OperationalError: (2003, "Can't connect to MySQL server on 'db' (111)")

config:

{
  "language": "python",
  "python": "3.6",
  "sudo": [
    "required"
  ],
  "services": [
    "docker"
  ],
  "before_install": [
    "sudo service mysql stop",
    "sudo /etc/init.d/postgresql stop",
    "sudo docker-compose up -d --build",
    "sudo docker exec -it hydros1_web_1 bash -c \"python manage.py test\""
  ],
  "before_script": null,
  "script": [
    "coverage run source='.' python manage.py test"
  ],
  "after_success": [
    "coveralls"
  ],
  "group": "stable",
  "dist": "trusty",
  "os": "linux"
}

Error Log:

Successfully built 609a39c65874
Creating hydros1_db_1
Creating hydros1_rabbit_1
Creating hydros1_web_1
Creating hydros1_nginx_1
Creating hydros1_worker_1
Creating hydros1_flower_1
travis_time:end:16301ef8:start=1489805820241928276,finish=1489806001947557958,duration=181705629682
[0Ktravis_fold:end:before_install.3
[0Ktravis_fold:start:before_install.4
[0Ktravis_time:start:005ddd7a
[0K$ sudo docker exec -it hydros1_web_1 bash -c "python manage.py test"
Creating test database for alias 'default'...
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/django/db/backends/base/base.py", line 199, in ensure_connection
    self.connect()
  File "/usr/local/lib/python3.6/site-packages/django/db/backends/base/base.py", line 171, in     connect
    self.connection = self.get_new_connection(conn_params)
  File "/usr/local/lib/python3.6/site-packages/django/db/backends/mysql/base.py", line 263, in get_new_connection
    conn = Database.connect(**conn_params)
  File "/usr/local/lib/python3.6/site-packages/MySQLdb/__init__.py", line 81, in Connect
    return Connection(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/MySQLdb/connections.py", line 191, in __init__
    super(Connection, self).__init__(*args, **kwargs2)
_mysql_exceptions.OperationalError: (2003, "Can't connect to MySQL server on 'db' (111)")

[danielmacho72]

Resolved as proposed by olalonde user under https://discuss.circleci.com/t/how-to-permanently-shutdown-circleci-default-postgres-mongodb-etc/7329/4

- sudo service postgresql stop
# wait for postgresql to shutdown
- while sudo lsof -Pi :5432 -sTCP:LISTEN -t; do sleep 1; done

See my working travis.yml under

https://github.com/danielmacho72/wakuas

[stale]

Thanks for contributing to this issue. As it has been 90 days since the last activity, we are automatically closing the issue. This is often because the request was already solved in some way and it just wasn't updated or it's no longer applicable. If that's not the case, please do feel free to either reopen this issue or open a new one. We'll gladly take a look again! You can read more here: https://blog.travis-ci.com/2018-03-09-closing-old-issues

[fabriziocucci]

Unfortunately, this issue is not stale!

I've applied @danielmacho72 's workaround but it should really be handled by the Travis team.

[stale]

Thanks for contributing to this issue. As it has been 90 days since the last activity, we are automatically closing the issue in 7 days. This is often because the request was already solved in some way and it just wasn't updated or it's no longer applicable. If that's not the case, please respond before the issue is closed, or open a new one after. We'll gladly take a look again! You can read more here: https://blog.travis-ci.com/2018-03-09-closing-old-issues

[KillWolfVlad]

Unfortunately, this issue is not stale!

[cklll]

sharing my solution
i am developing with django and postgres

originally i hard coded the postgres host with 192.168.99.100 and it works
but suddenly, it fails to connect to postgres only in travis, getting connection timeout

then i change the host to postgres
something like

services:
  postgres:
    environment:
      POSTGRES_USER: user
      POSTGRES_PASSWORD: password
      POSTGRES_DB: db
    image: postgres:10.5
    restart: always
    ports:
      - 5432:5432

  django:
    depends_on:
      - postgres
    environment:
      POSTGRES_USER: user
      POSTGRES_PASSWORD: password
      POSTGRES_DB: db
      POSTGRES_PORT: 5432
      POSTGRES_HOST: postgres

[stale]

Thanks for contributing to this issue. As it has been 90 days since the last activity, we are automatically closing the issue in 7 days. This is often because the request was already solved in some way and it just wasn't updated or it's no longer applicable. If that's not the case, please respond before the issue is closed, or open a new one after. We'll gladly take a look again! You can read more here: https://blog.travis-ci.com/2018-03-09-closing-old-issues

[ChuaCheowHuan]

The error output from Travis job log:

psycopg2.OperationalError: could not connect to server: Connection refused
539	Is the server running on host "db" (172.18.0.2) and accepting
540	TCP/IP connections on port 5432?

.
.
.

django.db.utils.OperationalError: could not connect to server: Connection
refused 587	Is the server running on host "db" (172.18.0.2) and accepting
588	TCP/IP connections on port 5432?

The command "docker-compose run web python manage.py test" exited with 1.

For anyone trying to test Dockerized Django with Postgres web apps in Travis & still stuck with this Postgress connection issue, introducing a delay in the docker-compose.yml file managed to resolve the problem for me.

command: bash -c 'while !</dev/tcp/db/5432; do sleep 1; done; python3 manage.py migrate'

command: bash -c 'while !</dev/tcp/db/5432; do sleep 1; done; python3 manage.py runserver 0.0.0.0:8000'

See this post in stackoverflow.

If you want an example of docker-compose.yml or .travis.yml files for testing Dockerized Django web apps with Postgres in Travis, see my blog post.