log job lifecycle #9130

fosterseth · 2021-01-21T17:50:54Z

feature issue #8749

SUMMARY

example output in /var/log/tower/job_lifecycle.log

{"type": "job", "task_id": 353, "state": "created", "template_name": "sleep", "time": "2021-01-28T04:25:46.661029+00:00"}
{"type": "job", "task_id": 353, "state": "acknowledged", "template_name": "sleep", "time": "2021-01-28T04:25:47.125391+00:00"}
{"type": "job", "task_id": 350, "state": "blocked", "template_name": "sleep", "blocked_by": "projectupdate-351", "time": "2021-01-28T04:25:47.166337+00:00"}
{"type": "job", "task_id": 352, "state": "blocked", "template_name": "sleep", "blocked_by": "projectupdate-351", "time": "2021-01-28T04:25:47.176640+00:00"}
{"type": "job", "task_id": 353, "state": "blocked", "template_name": "sleep", "blocked_by": "projectupdate-351", "time": "2021-01-28T04:25:47.177712+00:00"}

these events also show up in the stdout log
awx_1 | 2021-02-03 18:41:05,656 DEBUG awx.analytics.job_lifecycle job-740 created

Dependency Graph Changes

The DependencyGraph tells us what is blocked, but doesn't tell us what is doing the blocking. I rewrote it to expose that information.

e.g. here is the new representation in dependency_graph.data

{'inventory_source_ids': {},
 'inventory_source_updates': {},
 'inventory_updates': {},
 'job_template_jobs': {},
 'project_updates': {12: <ProjectUpdate: 2021-01-14 05:48:48.599418+00:00-2-pending>},
 'system_job': None,
 'workflow_job_template_jobs': {}}

now instead of is_job_blocked in the task manager, we have job_blocked_by which returns a UnifiedJob object, or None if it is not blocked.

Job Explanation Changes

We can also update job_explanation with some information about why a job is blocked.
e.g.
waiting until the project update finishes

To reduce performance loss with task.save(), we only update this job_explanation if it is different from the current job_explanation, and if more than 30 seconds have passed since job.created.

There is a PR open that will display job explanation in the UI, so this might be our mechanism of displaying information to the user.

#8726

ISSUE TYPE

Feature Pull Request

COMPONENT NAME

API

AWX VERSION

awx: 16.0.0

softwarefactory-project-zuul · 2021-01-21T18:10:50Z

Build failed.

awx-api-lint : FAILURE in 7m 45s
awx-api : FAILURE in 13m 20s
awx-ui : SUCCESS in 19m 15s
awx-swagger : SUCCESS in 15m 09s
awx-detect-schema-change : SUCCESS in 15m 09s (non-voting)
awx-ansible-modules : SUCCESS in 9m 14s

softwarefactory-project-zuul · 2021-01-22T21:12:47Z

Build failed.

awx-api-lint : FAILURE in 8m 30s
awx-api : FAILURE in 13m 10s
awx-ui : SUCCESS in 17m 16s
awx-swagger : SUCCESS in 14m 15s
awx-detect-schema-change : SUCCESS in 14m 20s (non-voting)
awx-ansible-modules : SUCCESS in 9m 24s

softwarefactory-project-zuul · 2021-01-22T23:04:03Z

Build failed.

awx-api-lint : FAILURE in 2m 50s
awx-api : FAILURE in 13m 17s
awx-ui : SUCCESS in 9m 41s
awx-swagger : SUCCESS in 7m 29s
awx-detect-schema-change : SUCCESS in 7m 45s (non-voting)
awx-ansible-modules : SUCCESS in 2m 36s

fosterseth · 2021-01-22T23:11:16Z

@gamuniz

What are you thoughts on the information output to /var/log/tower/job_lifecycle.log (see above)

if there is additional information helpful for debugging purposes, we could include them.

gamuniz · 2021-01-25T15:19:22Z

@fosterseth that looks really good and should be super helpful, this should be exposed to awx logger by default right?

ryanpetrello · 2021-01-25T16:16:44Z

awx/main/models/unified_jobs.py

@@ -397,6 +397,7 @@ def create_unified_job(self, **kwargs):
            # credentials and labels
            unified_job.save()

+        logger_job_lifecycle.info(f"{unified_job._meta.model_name}-{unified_job.id} created", extra={'type': unified_job._meta.model_name, 'template_name': unified_job.unified_job_template.name, 'job_id': unified_job.id, 'state': 'created'})


Could we make this a function on model.UnifiedJob and use it everywhere below e.g.,

def log_lifecycle(self, status, **extra): .... uj.log_lifecycle('created') uj.log_lifecycle('blocked', blocked_by='blah blah blah')

ryanpetrello · 2021-01-25T16:20:55Z

awx/main/scheduler/task_manager.py

                logger.debug("{} is blocked from running".format(task.log_format))
+                logger_job_lifecycle.info(f"{task._meta.model_name}-{task.id} blocked by {blocked_by._meta.model_name}-{blocked_by.id}", extra={'type': task._meta.model_name, 'template_name': task.unified_job_template.name, 'job_id': task.id, 'state': 'blocked', 'blocked_by': f"{blocked_by._meta.model_name}-{blocked_by.id}"})
+                job_explanation = str.format(f"waiting for {blocked_by._meta.model_name}-{blocked_by.id} to finish")
+                if task.job_explanation != job_explanation and task.created < (tz_now() - timedelta(seconds=30)):


Yea, I think this is a reasonable approach to avoid a stampede of updates.

ryanpetrello · 2021-01-25T16:21:32Z

awx/main/scheduler/task_manager.py

@@ -539,6 +552,10 @@ def process_pending_tasks(self, pending_tasks):
                    logger.debug("No instance available in group {} to run job {} w/ capacity requirement {}".format(
                                 rampart_group.name, task.log_format, task.task_impact))
            if not found_acceptable_queue:
+                job_explanation = "This job is not ready to start because there is not enough available capacity."
+                if task.job_explanation != job_explanation and task.created < (tz_now() - timedelta(seconds=30)):
+                    task.job_explanation = "This job is not ready to start because there is not enough available capacity."


nitpick, but task.job_explanation = job_explanation no need to duplicate this string.

good catch thanks. I also realize I need to unset/change this job_explanation when we start the task

Yes, good call. We may want to make sure that when a job goes into waiting, we unset this explicitly.

ryanpetrello · 2021-01-25T16:21:45Z

awx/main/scheduler/task_manager.py

@@ -539,6 +552,10 @@ def process_pending_tasks(self, pending_tasks):
                    logger.debug("No instance available in group {} to run job {} w/ capacity requirement {}".format(
                                 rampart_group.name, task.log_format, task.task_impact))
            if not found_acceptable_queue:
+                job_explanation = "This job is not ready to start because there is not enough available capacity."


This string should be wrapped in i18n _(...)

I am not sure if this is still relevant

awx/awx/api/serializers.py

Line 806 in dfa6522

# Because this string is saved in the db in the source language,

does this mean we shouldn't mark job_explanation for translation?

Hey, good catch! Yep, I think you're right about this.

I think in this case we might actually want to wrap this in ugettext_noop (not ugettext_lazy). This will mark the string for translation (so it shows up in our generated i18n files), but it won't actually translate it here.

https://docs.djangoproject.com/en/2.2/ref/utils/#django.utils.translation.ugettext_noop

ryanpetrello · 2021-01-25T16:22:28Z

awx/main/tasks.py

@@ -1186,16 +1186,19 @@ def pre_run_hook(self, instance, private_data_dir):
        '''
        Hook for any steps to run before the job/task starts
        '''
+        logger_job_lifecycle.info(f"{instance._meta.model_name}-{instance.id} pre run", extra={'type': instance._meta.model_name, 'template_name': instance.unified_job_template.name, 'job_id': instance.id, 'state': 'pre_run'})


I like the use of the hooks for this 👍

Instead of 'job_id' could we use self.event_data_key for accuracy for all types of "jobs"?

thinking about parsing this json -- would we want a fixed key here across all entries? maybe it could be named task_id to be more generic

Yea, that's a good point. If we want to stick with job_id, that probably makes sense considering I think there's additional context in here that tells you what type of job it is.

ryanpetrello · 2021-01-25T16:23:42Z

awx/main/tasks.py


    def post_run_hook(self, instance, status):
        '''
        Hook for any steps to run before job/task is marked as complete.
        '''
+        logger_job_lifecycle.info(f"{instance._meta.model_name}-{instance.id} post run", extra={'type': instance._meta.model_name, 'template_name': instance.unified_job_template.name, 'job_id': instance.id, 'state': 'post_run'})


Same comment as above here.

ryanpetrello · 2021-01-25T16:23:47Z

awx/main/tasks.py


    def final_run_hook(self, instance, status, private_data_dir, fact_modification_times, isolated_manager_instance=None):
        '''
        Hook for any steps to run after job/task is marked as complete.
        '''
+        logger_job_lifecycle.info(f"{instance._meta.model_name}-{instance.id} finalize run", extra={'type': instance._meta.model_name, 'template_name': instance.unified_job_template.name, 'job_id': instance.id, 'state': 'finalize_run'})


awx/main/scheduler/task_manager.py

ryanpetrello · 2021-01-25T16:26:19Z

awx/settings/defaults.py

@@ -940,6 +943,15 @@ def IS_TESTING(argv=None):
            'backupCount': 5,
            'formatter':'simple',
        },
+        'job_lifecycle': {
+            'level': 'INFO',
+            'class':'logging.handlers.RotatingFileHandler',


You might want to sync up w/ @chrismeyersfsu on his impending logrotate work, as he's been planning on removing these rotating file handlers.

This definitely needs an update now that @chrismeyersfsu has merged his log rotation PR, but I don't expect anything super complicated.

ryanpetrello

This is coming along nicely - I've got some minor feedback.

softwarefactory-project-zuul · 2021-01-26T20:00:10Z

Build failed.

awx-api-lint : SUCCESS in 2m 01s
awx-api : FAILURE in 13m 17s
awx-ui : SUCCESS in 11m 08s
awx-swagger : SUCCESS in 10m 08s
awx-detect-schema-change : SUCCESS in 8m 15s (non-voting)
awx-ansible-modules : SUCCESS in 2m 36s

softwarefactory-project-zuul · 2021-01-27T21:52:09Z

Build succeeded.

awx-api-lint : SUCCESS in 8m 48s
awx-api : SUCCESS in 13m 47s
awx-ui : SUCCESS in 16m 01s
awx-swagger : SUCCESS in 14m 17s
awx-detect-schema-change : SUCCESS in 14m 30s (non-voting)
awx-ansible-modules : SUCCESS in 8m 45s

softwarefactory-project-zuul · 2021-01-28T04:19:23Z

Build failed.

awx-api-lint : FAILURE in 3m 14s
awx-api : SUCCESS in 8m 19s
awx-ui : SUCCESS in 11m 45s
awx-swagger : SUCCESS in 13m 31s
awx-detect-schema-change : SUCCESS in 14m 47s (non-voting)
awx-ansible-modules : SUCCESS in 4m 13s

fosterseth · 2021-01-28T04:33:01Z

in development environment, job_lifecycle logs can be blue, if we want that

ryanpetrello · 2021-01-28T15:29:35Z

in development environment, job_lifecycle logs can be blue, if we want that

I like it 😄

ryanpetrello · 2021-01-28T15:37:51Z

awx/main/models/unified_jobs.py

+        if self.unified_job_template:
+            extra["template_name"] = self.unified_job_template.name
+        if state == "blocked" and blocked_by:
+            blocked_by_msg = f"{blocked_by._meta.model_name}-{blocked_by.id}"


We've got a property that's pretty similar to this that we use in other logging contexts:

UnifiedJob.log_format

It also includes the current status of the job. If that's too verbose, that's fine, but it might be an opportunity for reuse here.

In [1]: Job.objects.first().log_format Out[1]: 'job 1 (successful)'```

In [4]: [j.log_format for j in UnifiedJob.objects.all()] Out[4]: ['job 1 (successful)', 'project_update 2 (successful)', 'ad_hoc_command 3 (successful)', 'ad_hoc_command 4 (successful)', 'ad_hoc_command 5 (successful)', 'job 6 (successful)', 'project_update 7 (successful)']```

ryanpetrello

Really great work on this - I like how it turned out, and this is going to be really useful to figure out why jobs aren't running yet.

I'd say once we've tossed this through an integration run, I'm a 👍 on merging this.

ryanpetrello · 2021-01-28T15:46:38Z

@fosterseth,

@jakemcdermott had mentioned to me the possibility of folding his work on #8726 into this PR and verifying and landing them around the same time. Is this something you two have discussed as an option?

fosterseth · 2021-01-28T16:37:21Z

I've gone back and forth on whether we should provide a waiting_on field that would identify the job that is causing the block. We could then provide that job as a link in the ui, which I think would be really nice

    waiting_on = models.ForeignKey(
        'self',
        null=True,
        editable=False,
        related_name='+',
        on_delete=polymorphic.SET_NULL,
    )

e.g.

Currently, we can stuff that info into the job_explanation, but it won't be a link in the UI or anything.

Because waiting_on could be a projectupdate/job/inventorysource, we would also need another field in the json that identifies the type of resource, so that we can get the appropriate link.

@jakemcdermott and I brainstormed some ideas how to display this link. One thought was, if waiting_on is null, then just display job_explanation. If not null, then have a predefined string with the link to the waiting_on resource (e.g. "waiting on <project update 123>"). The downside of this is that there might be case we want job_explanation information to take a priority over waiting_on information, so having these two fields compete for the same space in the UI is not ideal. Having a separate "Waiting On" field in the UI solves this, but then again it's yet another field on the detail page, introducing more clutter.

Maybe we should punt the UI link idea until later, and settle with the generic text in the job_explanation for now?

ryanpetrello · 2021-01-29T16:26:41Z

@fosterseth before we merge this, can we squash a few commits (or at least, improve a few of the latter commit messages)?

fosterseth · 2021-01-29T18:26:53Z

@ryanpetrello Yep I plan to squash them all into one

softwarefactory-project-zuul · 2021-01-29T21:53:38Z

Build failed.

awx-api-lint : FAILURE in 8m 54s
awx-api : SUCCESS in 12m 50s
awx-ui : SUCCESS in 16m 59s
awx-swagger : SUCCESS in 16m 00s
awx-detect-schema-change : SUCCESS in 14m 48s (non-voting)
awx-ansible-modules : SUCCESS in 9m 34s

softwarefactory-project-zuul · 2021-01-29T22:24:20Z

Build succeeded.

awx-api-lint : SUCCESS in 9m 02s
awx-api : SUCCESS in 7m 43s
awx-ui : SUCCESS in 16m 28s
awx-swagger : SUCCESS in 15m 29s
awx-detect-schema-change : SUCCESS in 8m 46s (non-voting)
awx-ansible-modules : SUCCESS in 3m 51s

fosterseth · 2021-02-01T07:15:23Z

states for all jobs

created
acknowledged
waiting
pre run
preparing playbook
running playbook
post run
finalize run

special processing for some jobs

start job fact cache
finish job fact cache
event processing cache
notifications sent

stuck in pending

blocked by
needs capacity

elyezer · 2021-02-01T15:19:26Z

awx/main/scheduler/task_manager.py

+            blocked_by = self.job_blocked_by(task)
+            if blocked_by:
+                task.log_lifecycle("blocked", blocked_by=blocked_by)
+                job_explanation = gettext_noop(f"waiting for {blocked_by._meta.model_name}-{blocked_by.id} to finish")


Is it possible to make the blocked by job name to become a link in the UI so we can easily navigate to it and follow if we want to?

exactly, we plan to make this a link in the UI, which would be helpful to users

softwarefactory-project-zuul · 2021-02-01T16:30:40Z

Build succeeded.

awx-api-lint : SUCCESS in 9m 42s
awx-api : SUCCESS in 13m 11s
awx-ui : SUCCESS in 18m 12s
awx-swagger : SUCCESS in 15m 56s
awx-detect-schema-change : SUCCESS in 14m 55s (non-voting)
awx-ansible-modules : SUCCESS in 9m 42s

softwarefactory-project-zuul · 2021-02-02T18:04:14Z

Build succeeded.

awx-api-lint : SUCCESS in 8m 30s
awx-api : SUCCESS in 14m 28s
awx-ui : SUCCESS in 17m 47s
awx-swagger : SUCCESS in 13m 58s
awx-detect-schema-change : SUCCESS in 13m 53s (non-voting)
awx-ansible-modules : SUCCESS in 9m 37s

softwarefactory-project-zuul · 2021-02-02T20:51:23Z

Build succeeded.

awx-api-lint : SUCCESS in 2m 02s
awx-api : SUCCESS in 12m 41s
awx-ui : SUCCESS in 10m 03s
awx-swagger : SUCCESS in 14m 20s
awx-detect-schema-change : SUCCESS in 14m 18s (non-voting)
awx-ansible-modules : SUCCESS in 2m 33s

softwarefactory-project-zuul · 2021-02-03T01:29:44Z

Build succeeded.

awx-api-lint : SUCCESS in 1m 51s
awx-api : SUCCESS in 6m 53s
awx-ui : SUCCESS in 12m 15s
awx-swagger : SUCCESS in 7m 46s
awx-detect-schema-change : SUCCESS in 7m 58s (non-voting)
awx-ansible-modules : SUCCESS in 2m 25s

fosterseth · 2021-02-03T03:14:26Z

was curious how big this file gets -- here it is after running the yolo tests

[root@ip-10-0-4-116 tower]# ls -lh
total 1.7M
-rw-r-----. 1 awx  awx   52K Feb  3 02:58 callback_receiver.log
-rw-r-----. 1 awx  awx   99K Feb  3 02:58 dispatcher.log
-rw-r-----. 1 awx  awx     0 Feb  3 01:55 isolated_manager.log
-rw-r-----. 1 awx  awx  667K Feb  3 02:58 job_lifecycle.log
-rw-r-----. 1 awx  awx     0 Feb  3 01:55 management_playbooks.log
drwxr-xr-x. 2 awx  awx     6 Feb  3 02:14 playbook_profiling
-rw-------. 1 root root 658K Feb  3 02:11 setup-2021-02-03-01:50:34.log
-rw-r-----. 1 awx  awx   32K Feb  3 02:58 task_system.log
-rw-r-----. 1 awx  awx  212K Feb  3 03:05 tower.log
-rw-r-----. 1 awx  awx     0 Feb  3 01:55 tower_rbac_migrations.log
-rw-r-----. 1 awx  awx     0 Feb  3 01:55 tower_system_tracking_migrations.log
-rw-r-----. 1 awx  awx     0 Feb  3 01:55 wsbroadcast.log

softwarefactory-project-zuul · 2021-02-03T17:02:26Z

Build succeeded.

awx-api-lint : SUCCESS in 4m 46s
awx-api : SUCCESS in 9m 00s
awx-ui : SUCCESS in 13m 05s
awx-swagger : SUCCESS in 10m 07s
awx-detect-schema-change : SUCCESS in 16m 21s (non-voting)
awx-ansible-modules : SUCCESS in 10m 32s

softwarefactory-project-zuul · 2021-02-03T19:51:29Z

Build succeeded.

awx-api-lint : SUCCESS in 2m 08s
awx-api : SUCCESS in 6m 54s
awx-ui : SUCCESS in 16m 54s
awx-swagger : SUCCESS in 13m 30s
awx-detect-schema-change : SUCCESS in 7m 59s (non-voting)
awx-ansible-modules : SUCCESS in 9m 21s

softwarefactory-project-zuul · 2021-02-04T16:30:45Z

Build succeeded.

awx-api-lint : SUCCESS in 8m 38s
awx-api : SUCCESS in 13m 35s
awx-ui : SUCCESS in 17m 01s
awx-swagger : SUCCESS in 16m 43s
awx-detect-schema-change : SUCCESS in 13m 43s (non-voting)
awx-ansible-modules : SUCCESS in 9m 22s

Various points (e.g. created, running, processing events), are structured into json format and output to /var/log/tower/job_lifecycle.log As part of this work, the DependencyGraph is reworked to return which job object is doing the blocking, rather than a boolean.

softwarefactory-project-zuul · 2021-02-04T17:42:58Z

Build succeeded.

awx-api-lint : SUCCESS in 1m 56s
awx-api : SUCCESS in 6m 51s
awx-ui : SUCCESS in 16m 31s
awx-swagger : SUCCESS in 15m 57s
awx-detect-schema-change : SUCCESS in 16m 24s (non-voting)
awx-ansible-modules : SUCCESS in 10m 41s

softwarefactory-project-zuul · 2021-02-04T19:40:11Z

Build succeeded (gate pipeline).

awx-api-lint : SUCCESS in 1m 54s
awx-api : SUCCESS in 7m 05s
awx-ui : SUCCESS in 16m 20s
awx-swagger : SUCCESS in 13m 25s
awx-detect-schema-change : SUCCESS in 7m 43s (non-voting)
awx-ansible-modules : SUCCESS in 2m 29s
awx-push-new-schema : SUCCESS in 7m 34s (non-voting)

fosterseth changed the title ~~Feat wip job lifecycle~~ [wip] job lifecycle Jan 21, 2021

awxbot added type:enhancement component:api labels Jan 21, 2021

ryanpetrello reviewed Jan 25, 2021

View reviewed changes

awx/main/scheduler/task_manager.py Outdated Show resolved Hide resolved

ryanpetrello reviewed Jan 25, 2021

View reviewed changes

ryanpetrello suggested changes Jan 25, 2021

View reviewed changes

ryanpetrello requested a review from chrismeyersfsu January 25, 2021 16:32

fosterseth changed the title ~~[wip] job lifecycle~~ log job lifecycle Jan 28, 2021

ryanpetrello reviewed Jan 28, 2021

View reviewed changes

ryanpetrello approved these changes Jan 28, 2021

View reviewed changes

ryanpetrello requested a review from jakemcdermott January 28, 2021 15:45

jakemcdermott approved these changes Jan 28, 2021

View reviewed changes

fosterseth force-pushed the feat_wip_job_lifecycle branch from 158595f to 269facf Compare January 29, 2021 21:36

elyezer reviewed Feb 1, 2021

View reviewed changes

fosterseth force-pushed the feat_wip_job_lifecycle branch from f30baff to 17e2129 Compare February 2, 2021 17:46

fosterseth force-pushed the feat_wip_job_lifecycle branch from 6dfb396 to 82db4bd Compare February 2, 2021 20:36

fosterseth force-pushed the feat_wip_job_lifecycle branch 2 times, most recently from 0686062 to 227750a Compare February 3, 2021 16:45

fosterseth force-pushed the feat_wip_job_lifecycle branch from 227750a to 978b2b1 Compare February 3, 2021 19:34

ryanpetrello approved these changes Feb 3, 2021

View reviewed changes

fosterseth force-pushed the feat_wip_job_lifecycle branch from 978b2b1 to 78f4f1e Compare February 4, 2021 16:13

fosterseth force-pushed the feat_wip_job_lifecycle branch from 78f4f1e to 41d0a2f Compare February 4, 2021 17:26

fosterseth added the mergeit label Feb 4, 2021

softwarefactory-project-zuul bot merged commit ba318f8 into ansible:devel Feb 4, 2021

chrismeyersfsu mentioned this pull request Feb 12, 2021

support job lifecycle logs chrismeyersfsu/sosreport-elk#11

Open

log job lifecycle #9130

log job lifecycle #9130

Conversation

fosterseth commented Jan 21, 2021 • edited Loading

SUMMARY

Dependency Graph Changes

Job Explanation Changes

ISSUE TYPE

COMPONENT NAME

AWX VERSION

softwarefactory-project-zuul bot commented Jan 21, 2021

softwarefactory-project-zuul bot commented Jan 22, 2021

softwarefactory-project-zuul bot commented Jan 22, 2021

fosterseth commented Jan 22, 2021

gamuniz commented Jan 25, 2021

ryanpetrello Jan 25, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ryanpetrello Jan 25, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ryanpetrello Jan 26, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ryanpetrello Jan 25, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ryanpetrello Jan 25, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ryanpetrello left a comment

Choose a reason for hiding this comment

softwarefactory-project-zuul bot commented Jan 26, 2021

softwarefactory-project-zuul bot commented Jan 27, 2021

softwarefactory-project-zuul bot commented Jan 28, 2021

fosterseth commented Jan 28, 2021

ryanpetrello commented Jan 28, 2021

Choose a reason for hiding this comment

ryanpetrello Jan 28, 2021 • edited Loading

Choose a reason for hiding this comment

ryanpetrello Jan 28, 2021 • edited Loading

Choose a reason for hiding this comment

ryanpetrello left a comment

Choose a reason for hiding this comment

ryanpetrello commented Jan 28, 2021

fosterseth commented Jan 28, 2021 • edited Loading

ryanpetrello commented Jan 29, 2021 • edited Loading

fosterseth commented Jan 29, 2021

softwarefactory-project-zuul bot commented Jan 29, 2021

softwarefactory-project-zuul bot commented Jan 29, 2021

fosterseth commented Feb 1, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

softwarefactory-project-zuul bot commented Feb 1, 2021

softwarefactory-project-zuul bot commented Feb 2, 2021

softwarefactory-project-zuul bot commented Feb 2, 2021

softwarefactory-project-zuul bot commented Feb 3, 2021

fosterseth commented Feb 3, 2021

softwarefactory-project-zuul bot commented Feb 3, 2021

softwarefactory-project-zuul bot commented Feb 3, 2021

softwarefactory-project-zuul bot commented Feb 4, 2021

softwarefactory-project-zuul bot commented Feb 4, 2021

softwarefactory-project-zuul bot commented Feb 4, 2021

fosterseth commented Jan 21, 2021 •

edited

Loading

ryanpetrello Jan 25, 2021 •

edited

Loading

ryanpetrello Jan 25, 2021 •

edited

Loading

ryanpetrello Jan 26, 2021 •

edited

Loading

ryanpetrello Jan 25, 2021 •

edited

Loading

ryanpetrello Jan 25, 2021 •

edited

Loading

ryanpetrello Jan 28, 2021 •

edited

Loading

ryanpetrello Jan 28, 2021 •

edited

Loading

fosterseth commented Jan 28, 2021 •

edited

Loading

ryanpetrello commented Jan 29, 2021 •

edited

Loading