Fixed the Customized Notification returning incorrect values for host_status_counts #9251

Saurabh-Thakre · 2021-02-06T14:52:27Z

Fixed the customized notification returning incorrect values for host_status_counts. Summarizing the issue in the following

SUMMARY

When running a Job Template with a customized notification associated with it, it is returning incorrect values for host_status_counts in SLACK channel. I could see that in the code that we are taking the values for host_status_counts from job_host_summaries for the first rack than taking it from the /api/v2/jobs/ID.

https://github.com/ansible/tower/blob/370440f63d51f424009665edfd35b10322ddbd06/awx/main/models/notifications.py#L387

The values shows in /api/v2/jobs/ID look perfectly correct to me than what shows in /api/v2/jobs/ID/job_host_summaries. In the job_host_summaries we calculate the host_status_counts values for dark,ok, processed, etc. individually for different hosts. And right now we are taking host_status_counts from the first rack of the job_host_summaries rather than just taking it from /api/v2/jobs/ID host_status_counts which is calculated correctly. Looking at the job_host_summaries API output I think we are taking the values from the first rack of job_host_summaries

ISSUE TYPE

Bugfix Pull Request

COMPONENT NAME

API

AWX VERSION

* Tower version: 3.8.1

ADDITIONAL INFORMATION

With the PR, we could see the correct values are getting received in SLACK notifications.

Following data is from /api/v2/jobs/185 for host_status_counts

softwarefactory-project-zuul · 2021-02-06T15:03:30Z

Build failed.

awx-api-lint : FAILURE in 1m 55s
awx-api : FAILURE in 6m 21s
awx-ui : SUCCESS in 10m 40s
awx-swagger : SUCCESS in 7m 37s
awx-detect-schema-change : SUCCESS in 8m 00s (non-voting)
awx-ansible-modules : SUCCESS in 2m 39s

softwarefactory-project-zuul · 2021-02-06T16:00:02Z

Build failed.

awx-api-lint : FAILURE in 1m 50s
awx-api : FAILURE in 6m 50s
awx-ui : SUCCESS in 10m 28s
awx-swagger : SUCCESS in 7m 26s
awx-detect-schema-change : SUCCESS in 8m 32s (non-voting)
awx-ansible-modules : SUCCESS in 2m 26s

softwarefactory-project-zuul · 2021-02-06T16:30:19Z

Build failed.

awx-api-lint : SUCCESS in 2m 16s
awx-api : FAILURE in 7m 51s
awx-ui : SUCCESS in 9m 57s
awx-swagger : SUCCESS in 8m 40s
awx-detect-schema-change : SUCCESS in 9m 38s (non-voting)
awx-ansible-modules : SUCCESS in 3m 21s

awx/main/models/notifications.py

chrismeyersfsu · 2021-02-08T16:42:33Z

Looks like some tests need to be updated

2021-02-06 16:27:41.237095 | static | FAILED awx/main/tests/functional/models/test_notifications.py::TestJobNotificationMixin::test_context[InventoryUpdate]
2021-02-06 16:27:41.237200 | static | FAILED awx/main/tests/functional/models/test_notifications.py::TestJobNotificationMixin::test_context[SystemJob]
2021-02-06 16:27:41.237300 | static | FAILED awx/main/tests/functional/models/test_notifications.py::TestJobNotificationMixin::test_context[WorkflowJob]

you can run those tests individually in the dev environment:

source /var/lib/awx/venv/awx/bin/activate
py.test /awx_devel/awx/main/tests/functional/models/test_notifications.py

Saurabh-Thakre · 2021-02-09T09:44:19Z

Looks like some tests need to be updated

2021-02-06 16:27:41.237095 | static | FAILED awx/main/tests/functional/models/test_notifications.py::TestJobNotificationMixin::test_context[InventoryUpdate]
2021-02-06 16:27:41.237200 | static | FAILED awx/main/tests/functional/models/test_notifications.py::TestJobNotificationMixin::test_context[SystemJob]
2021-02-06 16:27:41.237300 | static | FAILED awx/main/tests/functional/models/test_notifications.py::TestJobNotificationMixin::test_context[WorkflowJob]

you can run those tests individually in the dev environment:

source /var/lib/awx/venv/awx/bin/activate
py.test /awx_devel/awx/main/tests/functional/models/test_notifications.py

Sure, I'm on it.

softwarefactory-project-zuul · 2021-02-09T12:21:29Z

Build failed.

awx-api-lint : SUCCESS in 11m 23s
awx-api : FAILURE in 16m 04s
awx-ui : SUCCESS in 19m 20s
awx-swagger : SUCCESS in 17m 56s
awx-detect-schema-change : SUCCESS in 17m 25s (non-voting)
awx-ansible-modules : SUCCESS in 13m 18s

chrismeyersfsu · 2021-02-09T18:09:48Z

Looks like I was wrong about the if condition. Without it, an inventory update notification will fail on the line events = qs.only('event_data').filter(event='playbook_on_stats')

2021-02-09 12:16:47.269972 | static | ____________ TestJobNotificationMixin.test_context[InventoryUpdate] ____________
2021-02-09 12:16:47.270026 | static | [gw0] linux -- Python 3.6.8 /var/lib/awx/venv/awx/bin/python3
2021-02-09 12:16:47.270059 | static | Traceback (most recent call last):
2021-02-09 12:16:47.270141 | static |   File "/awx_devel/awx/main/tests/functional/models/test_notifications.py", line 123, in test_context
2021-02-09 12:16:47.270188 | static |     context = job.context(job_serialization)
2021-02-09 12:16:47.270249 | static |   File "/awx_devel/awx/main/models/notifications.py", line 387, in context
2021-02-09 12:16:47.270307 | static |     events = qs.only('event_data').filter(event='playbook_on_stats')
2021-02-09 12:16:47.270491 | static |   File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/models/query.py", line 892, in filter
2021-02-09 12:16:47.270555 | static |     return self._filter_or_exclude(False, *args, **kwargs)
2021-02-09 12:16:47.270649 | static |   File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/models/query.py", line 910, in _filter_or_exclude
2021-02-09 12:16:47.270692 | static |     clone.query.add_q(Q(*args, **kwargs))
2021-02-09 12:16:47.270779 | static |   File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/models/sql/query.py", line 1290, in add_q
2021-02-09 12:16:47.270836 | static |     clause, _ = self._add_q(q_object, self.used_aliases)
2021-02-09 12:16:47.270924 | static |   File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/models/sql/query.py", line 1318, in _add_q
2021-02-09 12:16:47.270968 | static |     split_subq=split_subq, simple_col=simple_col,
2021-02-09 12:16:47.271059 | static |   File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/models/sql/query.py", line 1190, in build_filter
2021-02-09 12:16:47.271116 | static |     lookups, parts, reffed_expression = self.solve_lookup_type(arg)
2021-02-09 12:16:47.271220 | static |   File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/models/sql/query.py", line 1049, in solve_lookup_type
2021-02-09 12:16:47.271291 | static |     _, field, _, lookup_parts = self.names_to_path(lookup_splitted, self.get_meta())
2021-02-09 12:16:47.271393 | static |   File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/models/sql/query.py", line 1420, in names_to_path
2021-02-09 12:16:47.271454 | static |     "Choices are: %s" % (name, ", ".join(available)))
2021-02-09 12:16:47.271646 | static | django.core.exceptions.FieldError: Cannot resolve keyword 'event' into field. Choices are: counter, created, end_line, event_data, id, inventory_update, inventory_update_id, modified, start_line, stdout, uuid, verbosity

Saurabh-Thakre · 2021-02-10T10:31:17Z

Looks like I was wrong about the if condition. Without it, an inventory update notification will fail on the line events = qs.only('event_data').filter(event='playbook_on_stats')

2021-02-09 12:16:47.269972 | static | ____________ TestJobNotificationMixin.test_context[InventoryUpdate] ____________
2021-02-09 12:16:47.270026 | static | [gw0] linux -- Python 3.6.8 /var/lib/awx/venv/awx/bin/python3
2021-02-09 12:16:47.270059 | static | Traceback (most recent call last):
2021-02-09 12:16:47.270141 | static |   File "/awx_devel/awx/main/tests/functional/models/test_notifications.py", line 123, in test_context
2021-02-09 12:16:47.270188 | static |     context = job.context(job_serialization)
2021-02-09 12:16:47.270249 | static |   File "/awx_devel/awx/main/models/notifications.py", line 387, in context
2021-02-09 12:16:47.270307 | static |     events = qs.only('event_data').filter(event='playbook_on_stats')
2021-02-09 12:16:47.270491 | static |   File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/models/query.py", line 892, in filter
2021-02-09 12:16:47.270555 | static |     return self._filter_or_exclude(False, *args, **kwargs)
2021-02-09 12:16:47.270649 | static |   File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/models/query.py", line 910, in _filter_or_exclude
2021-02-09 12:16:47.270692 | static |     clone.query.add_q(Q(*args, **kwargs))
2021-02-09 12:16:47.270779 | static |   File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/models/sql/query.py", line 1290, in add_q
2021-02-09 12:16:47.270836 | static |     clause, _ = self._add_q(q_object, self.used_aliases)
2021-02-09 12:16:47.270924 | static |   File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/models/sql/query.py", line 1318, in _add_q
2021-02-09 12:16:47.270968 | static |     split_subq=split_subq, simple_col=simple_col,
2021-02-09 12:16:47.271059 | static |   File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/models/sql/query.py", line 1190, in build_filter
2021-02-09 12:16:47.271116 | static |     lookups, parts, reffed_expression = self.solve_lookup_type(arg)
2021-02-09 12:16:47.271220 | static |   File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/models/sql/query.py", line 1049, in solve_lookup_type
2021-02-09 12:16:47.271291 | static |     _, field, _, lookup_parts = self.names_to_path(lookup_splitted, self.get_meta())
2021-02-09 12:16:47.271393 | static |   File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/models/sql/query.py", line 1420, in names_to_path
2021-02-09 12:16:47.271454 | static |     "Choices are: %s" % (name, ", ".join(available)))
2021-02-09 12:16:47.271646 | static | django.core.exceptions.FieldError: Cannot resolve keyword 'event' into field. Choices are: counter, created, end_line, event_data, id, inventory_update, inventory_update_id, modified, start_line, stdout, uuid, verbosity

I don't think it is failing because we removed the if condition. It because the InventoryUpdate is not aware of event . We may need to filter those out, the InventoryUpdate and WorkflowJob ??

chrismeyersfsu · 2021-02-10T15:01:05Z

try:
  has_event_property = any([f for f in self.event_class._meta.fields if f.name == 'event'])
except NotImplementedError:
  has_event_property = False

if has_event_property:
  ...

Below is how this would play out for the various Job types.

softwarefactory-project-zuul · 2021-02-11T11:33:40Z

Build succeeded.

awx-api-lint : SUCCESS in 9m 41s
awx-api : SUCCESS in 14m 50s
awx-ui : SUCCESS in 17m 02s
awx-swagger : SUCCESS in 15m 22s
awx-detect-schema-change : SUCCESS in 15m 04s (non-voting)
awx-ansible-modules : SUCCESS in 10m 46s

chrismeyersfsu · 2021-02-11T14:00:03Z

recheck

chrismeyersfsu · 2021-02-11T15:53:16Z

regate

awx/main/models/notifications.py

softwarefactory-project-zuul · 2021-02-16T17:08:27Z

Build succeeded.

awx-api-lint : SUCCESS in 2m 11s
awx-api : SUCCESS in 8m 00s
awx-ui : SUCCESS in 10m 53s
awx-swagger : SUCCESS in 9m 52s
awx-detect-schema-change : SUCCESS in 7m 33s (non-voting)
awx-ansible-modules : SUCCESS in 2m 48s

Fixed the customized notification returning incorrect values for host_status_counts Update notifications.py Removed if condition Added exception handling A nitpick

softwarefactory-project-zuul · 2021-02-18T12:49:14Z

Build succeeded.

awx-api-lint : SUCCESS in 12m 43s
awx-api : SUCCESS in 18m 40s
awx-ui : SUCCESS in 19m 41s
awx-swagger : SUCCESS in 18m 11s
awx-detect-schema-change : FAILURE in 18m 21s (non-voting)
awx-ansible-modules : SUCCESS in 13m 01s

chrismeyersfsu · 2021-02-19T14:47:04Z

(closed and reopened to get zuul working)

softwarefactory-project-zuul · 2021-02-19T15:07:54Z

Build succeeded.

awx-api-lint : SUCCESS in 14m 31s
awx-api : SUCCESS in 17m 03s
awx-ui : SUCCESS in 20m 57s
awx-swagger : SUCCESS in 17m 51s
awx-detect-schema-change : SUCCESS in 17m 38s (non-voting)
awx-ansible-modules : SUCCESS in 13m 13s

softwarefactory-project-zuul · 2021-02-19T15:29:39Z

Build failed (gate pipeline). For information on how to proceed, see
http://docs.openstack.org/infra/manual/developers.html#automated-testing

awx-api-lint : FAILURE in 4m 20s
awx-api : SUCCESS in 12m 30s
awx-ui : SUCCESS in 21m 30s
awx-swagger : SUCCESS in 8m 30s
awx-detect-schema-change : SUCCESS in 8m 10s (non-voting)
awx-ansible-modules : SUCCESS in 2m 36s
awx-push-new-schema : SKIPPED (non-voting)

chrismeyersfsu · 2021-02-19T15:37:40Z

regate

softwarefactory-project-zuul · 2021-02-19T16:07:29Z

Build succeeded (gate pipeline).

awx-api-lint : SUCCESS in 2m 52s
awx-api : SUCCESS in 20m 29s
awx-ui : SUCCESS in 11m 59s
awx-swagger : SUCCESS in 8m 58s
awx-detect-schema-change : SUCCESS in 9m 47s (non-voting)
awx-ansible-modules : SUCCESS in 2m 34s
awx-push-new-schema : SUCCESS in 8m 46s (non-voting)

awxbot added type:bug component:api labels Feb 6, 2021

chrismeyersfsu reviewed Feb 8, 2021

View reviewed changes

awx/main/models/notifications.py Outdated Show resolved Hide resolved

chrismeyersfsu reviewed Feb 8, 2021

View reviewed changes

awx/main/models/notifications.py Show resolved Hide resolved

chrismeyersfsu approved these changes Feb 11, 2021

View reviewed changes

chrismeyersfsu added the mergeit label Feb 11, 2021

chrismeyersfsu approved these changes Feb 11, 2021

View reviewed changes

chrismeyersfsu reviewed Feb 12, 2021

View reviewed changes

awx/main/models/notifications.py Outdated Show resolved Hide resolved

chrismeyersfsu removed the mergeit label Feb 12, 2021

chrismeyersfsu approved these changes Feb 17, 2021

View reviewed changes

Updated notifications.py

6aca9d8

Fixed the customized notification returning incorrect values for host_status_counts Update notifications.py Removed if condition Added exception handling A nitpick

Saurabh-Thakre force-pushed the Saurabh-Thakre-patch-1 branch from 805cbc8 to 6aca9d8 Compare February 18, 2021 12:29

chrismeyersfsu added mergeit and removed component:api labels Feb 19, 2021

chrismeyersfsu closed this Feb 19, 2021

chrismeyersfsu reopened this Feb 19, 2021

softwarefactory-project-zuul bot merged commit 182a7e8 into ansible:devel Feb 19, 2021

Saurabh-Thakre deleted the Saurabh-Thakre-patch-1 branch February 19, 2021 16:18

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fixed the Customized Notification returning incorrect values for host_status_counts #9251

Fixed the Customized Notification returning incorrect values for host_status_counts #9251

Saurabh-Thakre commented Feb 6, 2021 •

edited

softwarefactory-project-zuul bot commented Feb 6, 2021

softwarefactory-project-zuul bot commented Feb 6, 2021

softwarefactory-project-zuul bot commented Feb 6, 2021

chrismeyersfsu commented Feb 8, 2021

Saurabh-Thakre commented Feb 9, 2021

softwarefactory-project-zuul bot commented Feb 9, 2021

chrismeyersfsu commented Feb 9, 2021

Saurabh-Thakre commented Feb 10, 2021

chrismeyersfsu commented Feb 10, 2021 •

edited

softwarefactory-project-zuul bot commented Feb 11, 2021

chrismeyersfsu commented Feb 11, 2021

chrismeyersfsu commented Feb 11, 2021

softwarefactory-project-zuul bot commented Feb 16, 2021

softwarefactory-project-zuul bot commented Feb 18, 2021

chrismeyersfsu commented Feb 19, 2021

softwarefactory-project-zuul bot commented Feb 19, 2021

softwarefactory-project-zuul bot commented Feb 19, 2021

chrismeyersfsu commented Feb 19, 2021

softwarefactory-project-zuul bot commented Feb 19, 2021

Fixed the Customized Notification returning incorrect values for host_status_counts #9251

Fixed the Customized Notification returning incorrect values for host_status_counts #9251

Conversation

Saurabh-Thakre commented Feb 6, 2021 • edited

SUMMARY

ISSUE TYPE

COMPONENT NAME

AWX VERSION

ADDITIONAL INFORMATION

softwarefactory-project-zuul bot commented Feb 6, 2021

softwarefactory-project-zuul bot commented Feb 6, 2021

softwarefactory-project-zuul bot commented Feb 6, 2021

chrismeyersfsu commented Feb 8, 2021

Saurabh-Thakre commented Feb 9, 2021

softwarefactory-project-zuul bot commented Feb 9, 2021

chrismeyersfsu commented Feb 9, 2021

Saurabh-Thakre commented Feb 10, 2021

chrismeyersfsu commented Feb 10, 2021 • edited

softwarefactory-project-zuul bot commented Feb 11, 2021

chrismeyersfsu commented Feb 11, 2021

chrismeyersfsu commented Feb 11, 2021

softwarefactory-project-zuul bot commented Feb 16, 2021

softwarefactory-project-zuul bot commented Feb 18, 2021

chrismeyersfsu commented Feb 19, 2021

softwarefactory-project-zuul bot commented Feb 19, 2021

softwarefactory-project-zuul bot commented Feb 19, 2021

chrismeyersfsu commented Feb 19, 2021

softwarefactory-project-zuul bot commented Feb 19, 2021

Saurabh-Thakre commented Feb 6, 2021 •

edited

chrismeyersfsu commented Feb 10, 2021 •

edited