Provided docker-compose errors indefinitely if postgres startup is slow

[echristoffersen]

Issue: Indefinite error loop when awx apps are ready before postgres.
Workaround: Start postgres container, check log that postgres is ready for connections, then start up awx containers.

I downloaded the provided docker-compose script and ran command:
docker-compose up -d

Tailing the awx_task container, I get the below error message.

django.db.utils.ProgrammingError:

 relation "main_schedule" does not exist
LINE 1: ...le"."next_run", "main_schedule"."extra_data" FROM "main_sche...
                                                             ^

2017-09-15 19:57:27,432 INFO exited: celery (exit status 1; not expected)
[2017-09-15 19:57:28,234: INFO/MainProcess] Scheduler: Sending due task task_manager (awx.main.scheduler.tasks.run_task_manager)
2017-09-15 19:57:28,237 INFO spawned: 'celery' with pid 3421
[2017-09-15 19:57:28,238: DEBUG/MainProcess] awx.main.scheduler.tasks.run_task_manager sent. id->a3ddb65d-21c8-49f9-ba6c-6298e2ce67b6
[2017-09-15 19:57:28,239: DEBUG/MainProcess] beat: Waking up in 19.90 seconds.
2017-09-15 19:57:29,241 INFO success: celery entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2017-09-15 19:57:30,706 WARNING  py.warnings /var/lib/awx/venv/awx/lib/python2.7/site-packages/django/core/management/base.py:260: RemovedInDjango19Warning: "requires_model_validation" is deprecated in favor of "requires_system_checks".
  RemovedInDjango19Warning)

2017-09-15 19:57:30,749 INFO     awx.main.tasks Syncing Schedules
2017-09-15 19:57:30,749 INFO     awx.main.tasks Syncing Schedules
Traceback (most recent call last):
  File "/usr/bin/awx-manage", line 9, in <module>
    load_entry_point('awx==1.0.0.312', 'console_scripts', 'awx-manage')()
  File "/usr/lib/python2.7/site-packages/awx/__init__.py", line 107, in manage
    execute_from_command_line(sys.argv)
...
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_schedule" does not exist
LINE 1: ...le"."next_run", "main_schedule"."extra_data" FROM "main_sche...
                                                             ^

I'm not sure if this should be considered a bug, but I'll report in case others run into the same issue.

[bartmeuris]

I had this issue too. Relying on container startup order is in my experience a bad idea - it's better if the applications would just retry the connection until it succeeds.

That said, many don't, and I usually solve this by waiting for the postgres instance in a wrapper script. There are multiple options:

• Using pg_isready (requires more packages, but is probably the safest)
• Using netcat in a loop, but netcat is notoriously implementation-specific, this is how it works with the netcat-openbsd package (default on ubuntu/debian I think), not sure if other implementations support the -z flag:

while ! nc -z ${DATABASE_HOST} ${DATABASE_PORT:-5432}; do
    sleep .3
done

• Use a a pure-shell approach:

while ! (echo >/dev/tcp/${DATABASE_HOST}/${DATABASE_PORT:-5432}) &>/dev/null; do
    sleep .3;
done

[geerlingguy]

Thanks for reporting this! I, luckily, haven't run into this yet... but I have heard from a few people who have seen it happen. If it becomes a regular issue, I might work to wrap things like @bartmeuris suggests, or find some other workaround.

[geerlingguy]

@echristoffersen - Hmm, is it actually this issue: #19 ?

[geerlingguy]

I'm closing all issues on this repository as it has been deprecated; please see: #49