Make test failing locally

[john-westcott-iv]

Please confirm the following

• I agree to follow this project's code of conduct.
• I have checked the current issues for duplicates.
• I understand that AWX is open source software provided for free and that I might not receive a timely response.

Bug Summary

When running make test on an AWX container locally we are seeing failing tests with stacks like:

Traceback (most recent call last):
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/utils.py", line 89, in _execute
    return self.cursor.execute(sql, params)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/psycopg/cursor.py", line 723, in execute
    raise ex.with_traceback(None)
psycopg.errors.UndefinedColumn: column main_unifiedjob.result_stdout_text does not exist
LINE 1: SELECT "main_unifiedjob"."id", "main_unifiedjob"."result_std...
                                       ^

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/awx_devel/awx/main/tests/functional/api/test_unified_jobs_view.py", line 48, in negative_test_cases
    e['job'].result_stdout_text = e['text']
  File "/awx_devel/awx/main/models/unified_jobs.py", line 1035, in result_stdout_text
    related = UnifiedJobDeprecatedStdout.objects.get(pk=self.pk)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/models/manager.py", line 87, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/models/query.py", line 633, in get
    num = len(clone)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/models/query.py", line 380, in __len__
    self._fetch_all()
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/models/query.py", line 1881, in _fetch_all
    self._result_cache = list(self._iterable_class(self))
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/models/query.py", line 91, in __iter__
    results = compiler.execute_sql(
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/models/sql/compiler.py", line 1560, in execute_sql
    cursor.execute(sql, params)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/utils.py", line 67, in execute
    return self._execute_with_wrappers(
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/utils.py", line 80, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/utils.py", line 89, in _execute
    return self.cursor.execute(sql, params)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/utils.py", line 91, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/utils.py", line 89, in _execute
    return self.cursor.execute(sql, params)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/psycopg/cursor.py", line 723, in execute
    raise ex.with_traceback(None)
django.db.utils.ProgrammingError: column main_unifiedjob.result_stdout_text does not exist
LINE 1: SELECT "main_unifiedjob"."id", "main_unifiedjob"."result_std...

However, the same tests do not fail when run through the github actions.
This was initially thought to be part of the django migration but its been confirmed that this is also the case pre-upgrade.

The error appears to be related to how sqlite/py.test is being used in conjunction with the UnifiedJobDeprecatedStdout class. In the migrations, you can see the result_stdout_text field being created in migration 0001 and then "removed" from the DB state against the UnifiedJob model and added to the UnifiedJobDeprecatedStdout model in migration 0017.

@AlanCoding also noted that the following patch allows the tests to run locally but causes a significant increase in time:

diff --git a/pytest.ini b/pytest.ini
index d4ffd2db9b..d27343b6f1 100644
--- a/pytest.ini
+++ b/pytest.ini
@@ -3,7 +3,7 @@ DJANGO_SETTINGS_MODULE = awx.settings.development
 python_paths = /var/lib/awx/venv/tower/lib/python3.8/site-packages
 site_dirs = /var/lib/awx/venv/tower/lib/python3.8/site-packages
 python_files = *.py
-addopts = --reuse-db --nomigrations --tb=native
+addopts = --tb=native
 markers =
     ac: access control test
     survey: tests related to survey feature

We should try to determine:

1. if there is a way around this error without incurring a time penalty
2. why the github actions are not failing with this error

AWX version

latest

Select the relevant components

• UI
• API
• Docs
• Collection
• CLI
• Other

Installation method

docker development environment

Modifications

no

Ansible version

No response

Operating system

No response

Web browser

No response

Steps to reproduce

make docker-compose-build

make test

Expected results

Tests should pass.

Actual results

Stack trances around the unified jobs.

Additional information

No response

[AlanCoding]

Outside of the container, I ran:

AWX_DOCKER_CMD=/start_tests.sh COMPOSE_TAG=devel make docker-runner

This completed without any errors. However, if I shell into the container with the server running and run:

/start_tests.sh

I get 167 test failures (it does not continue on to the CLI or migration checks). Those tests include this one about:

column main_unifiedjob.result_stdout_text does not exist

This is surprising, because the two should be running the same thing.

[AlanCoding]

This is because the conf.d/ folder include happens after the test database is defined

awx/awx/settings/development.py

Line 116 in 2ce9440

include(optional('/etc/tower/conf.d/*.py'), scope=locals())

The conf.d folders were used in the docker-compose environments since #9289, and this should have existed ever since.

I believe this will be addressed by #13895, and I will test that and try to move ahead with that as the preferred solution.

[AlanCoding]

Confirmed that #13895 gets make test passing locally.

[AlanCoding]

To @john-westcott-iv's question here - why does result_stdout_text get created using sqlite3 but not postgres? The behavior of postgres is arguably more correct, because if you're jumping to the current state, the present models should not specify that this field should be created.

I still don't know, but it is quickly obvious that the implementations for both are in a totally separate place, and basically rewrites of each other.

https://github.com/django/django/blob/c813fb327cb1b09542be89c5ceed367826236bc2/django/db/backends/sqlite3/creation.py#L25

vs.

https://github.com/django/django/blob/c813fb327cb1b09542be89c5ceed367826236bc2/django/db/backends/postgresql/creation.py#L40

These do not even overwrite the same method, which for squlite3 is _create_test_db and postgres is _execute_create_test_db. To understand the difference, you need to consult the base class behavior. There actually does seem to be a dedicated flag for "disable migrations for tests" which we, surprisingly, do not set. This would probably still be ran in either database case, so I'm still not at an answer, but may be close.

[AlanCoding]

Believe to be addressed by linked PR which is merged now.