Tests fail in container

[michaelbukachi]

Describe the bug
I'm trying to run tests within a container but they keep failing

Environment

• Host OS - Ubuntu
• Docker image if applicable - python:3.6.8-slim
• Python Version - python:3.6.8
• Virtualenv/Pyenv etc.. if applicable

To Reproduce
Steps to reproduce the behavior:

1. Create simple test && setup fixtures
2. Build test container
3. Run test container with docker run -v /var/run/docker.sock:/var/run/docker.sock image
   Here's the Dockerfile

FROM python:3.6.8-slim

RUN apt-get update

RUN apt-get install -y apt-file

RUN apt-file update

RUN apt-get install -y libpq-dev gcc

RUN mkdir /project

ADD requirements.txt /project

WORKDIR /project

RUN pip install --upgrade pip

RUN pip install -r requirements.txt

RUN apt-get install -y wget

RUN wget -O get-docker.sh https://get.docker.com

RUN chmod +x get-docker.sh && ./get-docker.sh

COPY tests /project/tests
COPY pytest.ini /project
COPY .env.test /project

CMD ["pytest", "-x", "tests"]

Expected behavior
Tests to run successfully.

Actual Behavior

    def gevent_wait_callback(conn, timeout=None):
        """A wait callback useful to allow gevent to work with Psycopg."""
        while 1:
>           state = conn.poll()
E           sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) could not connect to server: Connection refused
E               Is the server running on host "localhost" (127.0.0.1) and accepting
E               TCP/IP connections on port 5532?
E           could not connect to server: Cannot assign requested address
E               Is the server running on host "localhost" (::1) and accepting
E               TCP/IP connections on port 5532?
E           
E           (Background on this error at: http://sqlalche.me/e/e3q8)

/usr/local/lib/python3.6/site-packages/psycogreen/gevent.py:32: OperationalError

.....
# more errors
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/usr/local/lib/python3.6/site-packages/pytest_mock_resources/container/__init__.py:37: in retriable_check_fn
    check_fn()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

    def check_postgres_fn():
        try:
            get_sqlalchemy_engine(config["root_database"])
        except sqlalchemy.exc.OperationalError:
            raise ContainerCheckFailed(
                "Unable to connect to a presumed Postgres test container via given config: {}".format(
>                   config
                )
            )
E           pytest_mock_resources.container.ContainerCheckFailed: Unable to connect to a presumed Postgres test container via given config: {'username': 'user', 'password': 'password', 'port': 5532, 'root_database': 'dev', 'image': 'postgres:9.6.10-alpine'}

/usr/local/lib/python3.6/site-packages/pytest_mock_resources/container/postgres.py:56: ContainerCheckFailed

Additional context
When I run tests outside the container they run successfully.

[DanCardin]

I just copied your Dockerfile (with some adjustments, as we dont have a requirements.txt, etc) and ran this repo's tests inside a container (locally) and it appears to have all ran fine.

I could definitely imagine our use of localhost being an issue, though it's not obvious to me why I can't reproduce. I could also imagine gevent being the problem, but i dont know enough about gevent to produce a minimal example which was known to fail.

I'd be interested to know:

• if you for-sure don't have something bound to port 5532?
• if you monitor docker ps and do you see the container spinning up?
• is there any way you can access the spun-up postgres from inside the container's commandline, though timing might be somewhat difficult.

[michaelbukachi]

Interesting. What command are you running in terminal? Maybe I'm mixing up commands. The goal is to set it up on GitLab using dind, Also, planning to do a PR for the CI section of the documentation once I have a working setup.

[DanCardin]

Also, planning to do a PR for the CI section of the documentation once I have a working setup.
Very cool, thanks!

For running pytest-mock-resources's own tests:

FROM python:3.6.8-slim

RUN apt-get update
RUN apt-get install -y apt-file
RUN apt-file update
RUN apt-get install -y libpq-dev gcc
RUN mkdir /project

WORKDIR /project

RUN pip install --upgrade pip
RUN apt-get install -y wget
RUN wget -O get-docker.sh https://get.docker.com
RUN chmod +x get-docker.sh && ./get-docker.sh

ENV PATH="/root/.poetry/bin:${PATH}"
RUN curl -sSL https://raw.githubusercontent.com/sdispater/poetry/1.0.0/get-poetry.py | python \
    && poetry config virtualenvs.create false

COPY . /project
RUN poetry install -E postgres -E mongo -E redshift -E redis

CMD ["pytest", "-x", "tests"]

docker build -f Dockerfile -t foo .
docker run -it -v /var/run/docker.sock:/var/run/docker.sock foo

If that works for you, then it kind of makes me think it's related to gevent (given that it was oddly in your stacktrace?)?

[michaelbukachi]

Hmm. I still can't get it to work. Here's a minimal example:
example.py

import sqlalchemy as sa
from sqlalchemy.ext.declarative import declarative_base

Base = declarative_base()


class User(Base):
    __tablename__ = 'users'
    id = sa.Column(sa.Integer, primary_key=True)
    name = sa.Column(sa.String)

tests/conftest.py

from pytest_mock_resources import create_postgres_fixture

pg = create_postgres_fixture(scope='module')

tests/test_example.py

from example import User


def test_user_is_created_successfully(pg):
    pg.add(User(name='test'))
    pg.flush()
    assert pg.query(User).count()

Dockerfile

FROM python:3.6.8-slim

RUN apt-get update
RUN apt-get install -y apt-file
RUN apt-file update
RUN apt-get install -y libpq-dev gcc
RUN mkdir /project

ADD requirements.txt /project

WORKDIR /project

RUN pip install --upgrade pip
RUN pip install -r requirements.txt
RUN apt-get install -y wget
RUN wget -O get-docker.sh https://get.docker.com
RUN chmod +x get-docker.sh && ./get-docker.sh

COPY . /project


CMD ["pytest", "-x", "tests"]

requirements.txt

psycopg2-binary==2.8.5
SQLAlchemy==1.3.16
pytest==5.2.0
pytest-mock-resources==1.2.2

The error I'm getting is:

E       sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) could not connect to server: Connection refused
E               Is the server running on host "localhost" (127.0.0.1) and accepting
E               TCP/IP connections on port 5532?
E       could not connect to server: Cannot assign requested address
E               Is the server running on host "localhost" (::1) and accepting
E               TCP/IP connections on port 5532?
E       
E       (Background on this error at: http://sqlalche.me/e/e3q8)

/usr/local/lib/python3.6/site-packages/psycopg2/__init__.py:127: OperationalError

... # More errors
    def check_postgres_fn():
        try:
            get_sqlalchemy_engine(config["root_database"])
        except sqlalchemy.exc.OperationalError:
            raise ContainerCheckFailed(
                "Unable to connect to a presumed Postgres test container via given config: {}".format(
>                   config
                )
            )
E           pytest_mock_resources.container.ContainerCheckFailed: Unable to connect to a presumed Postgres test container via given config: {'username': 'user', 'password': 'password', 'port': 5532, 'root_database': 'dev', 'image': 'postgres:9.6.10-alpine'}

/usr/local/lib/python3.6/site-packages/pytest_mock_resources/container/postgres.py:56: ContainerCheckFailed

[DanCardin]

I did have to change your conftest to

from pytest_mock_resources import create_postgres_fixture
from example import Base

pg = create_postgres_fixture(Base, scope="module", session=True)

(which make perfect sense), but otherwise this passes for me 😕. you definitely don't have a zombie container running on that port, or otherwise have something bound to that port on the host?

[michaelbukachi]

I've made the changes. But it's still not working. What platform are you using? I'm using linux.

[DanCardin]

I'm currently on osx, but i've run it on linux and our CI (circle CI) runs (in docker) on linux as well.

The changes I made would only apply after you've successfully instantiated the fixture (i.e. in the test body), whereas you're failing to start up the container before the tests start.

It's still not clear to me whether you're seeing the container spin up at all (and then be torn down). I.e. your error is caused by it not being able to connect for some reason, vs there being a service which we're trying to connect to with incorrect credentials.

In addition to clarity on the above, i'd be interested in seeing if you could run pmr postgres (which is largely a glorified docker run -v 5532:5432 postgres). If that's successful, then try running the tests.

[michaelbukachi]

It runs successfully, here's output:

foo    | 781999bfdc843c985d045ab817e637f7d230d14f89a596b8a55ffacb840670ea
foo    | CONTAINER ID        IMAGE                    COMMAND                  CREATED             STATUS                  PORTS                      NAMES
foo    | 781999bfdc84        postgres:9.6.10-alpine   "docker-entrypoint.s…"   1 second ago        Up Less than a second   0.0.0.0:5532->5432/tcp     ecstatic_burnell
foo    | 67a7d9d135d3        gitlab-pmr_foo           "./run.sh"               2 seconds ago       Up 1 second                                        foo

One thing I've noticed is that I can't access any service on localhost from within the container. I've verified this with a http server running on the host machine.

[DanCardin]

We have an environment variable which can be used to configure the host that it looks at export PYTEST_MOCK_RESOURCES_HOST=foo. Looking at https://stackoverflow.com/questions/31324981/how-to-access-host-port-from-docker-container/31328031#31328031, i wonder if it needs to be docker0, or 127.0.0.1, or one of the workarounds in docker/for-linux's linked github issue thread.

It does feel odd to me that we wouldn't have this problem in our other linux environments though.

[michaelbukachi]

Hmm. Let me look into it. I'm considering running a dind version instead of mounting the socket. Maybe, my internal setup is broken so that's why it's not working. What linux versions are you running, ubuntu?

[DanCardin]

the only linux machines i can confidently use as comparisons are CI (circleci), which would be Ubuntu, afaik

[oakhan3]

Hey @michaelbukachi

This problem makes sense - docker in *nix systems does not seem to support host.docker.internal - which is used by pytest-mock-resources to find the docker host when running tests from within a container on Mac or Windows OS.

See: https://github.com/schireson/pytest-mock-resources/blob/master/src/pytest_mock_resources/container/__init__.py#L14

Following this issue:

docker/for-linux#264 (comment)

Could you do the following and get back to us? -

1. Find the docker host on your ubuntu host machine - this is the ip listed with docker0 when you run ip config (usually 172.17.0.1 or 172.17.0.2) or listed in /etc/hosts
2. Start up your python container
3. Export the ip from step 1 to an env var called PYTEST_MOCK_RESOURCES_HOST
4. Try running your tests

If this works then it confirms the issue and means we could add something to that "get_docker_host" function to do the above automatcially.

You could also run your docker container with the --net=host option instead of the above and it should work, although I'm not sure if that could be implemented in gitlab.

[michaelbukachi]

Passing 172.17.0.1 to PYTEST_MOCK_RESOURCES_HOST causes the test to hang indefinitely.

[michaelbukachi]

Got it working using dind. Had to used a compose file so as not worry about network issues.

version: '3.4'

services:
  docker:
    image: docker:dind
    privileged: true
    environment:
      DOCKER_TLS_CERTDIR: ''
  foo:
    container_name: foo
    environment:
      DOCKER_HOST: tcp://docker:2375
      PYTEST_MOCK_RESOURCES_HOST: docker
    build:
      context: .
      dockerfile: Dockerfile

Next up, testing on gitlab!:smile:

[michaelbukachi]

Here's a working gitlab configuration:

services:
  - docker:dind

variables:
  DOCKER_TLS_CERTDIR: ''

stages:
  - testing

testing-job:
  image: python:3.6.8-slim
  stage: testing
  variables:
    DOCKER_HOST: tcp://docker:2375
    PYTEST_MOCK_RESOURCES_HOST: docker
  before_script:
    - apt-get update && apt-get install -y wget libpq-dev gcc
    - wget -O get-docker.sh https://get.docker.com
    - chmod +x get-docker.sh && ./get-docker.sh
  script:
    - pip install -r requirements.txt
    - pytest -x tests

Do I need to do a PR for the documentation?:smiley:

[oakhan3]

hah, fantastic, yes please 👍

[DanCardin]

Given the now merged doc update on working in dind/gitlab, closing this issue. Feel free to suggest I reopen if that's not true