periodically check the status of a copr build #490

TomasTomecek · 2020-03-13T11:15:17Z

TODO:

write tests
merge deployment: process path_to_secrets actually + random commits #497

Fixes #334

softwarefactory-project-zuul · 2020-03-13T11:33:02Z

Build failed.

packit-service-pre-commit : SUCCESS in 2m 35s
packit-service-tests : FAILURE in 16m 49s
packit-service-tests-requre : SUCCESS in 15m 06s

lachmanfrantisek

Looks simple but really promising!

(It will solve also the branch builds for now..;)

lachmanfrantisek · 2020-03-13T14:17:16Z

packit_service/worker/tasks.py

+    bind=True,
+    name="task.babysit_copr_build",
+    # 120 minutes
+    default_retry_delay=120 * 60,


This is the time when we start this baby-sit task for the first time? (Any then retry each minute?) (i.e. 120, 1, 1, 1, ...)

I would try some exponential sequence as Dominika suggested. (e.g. 1, 2, 4, 8, 16, ... ) or, a constant + exponential sequence (e.g. 10, 1, 2, 4, 8, ... )

[examples are in minutes and mean time to the next check]

I'm not too sold on the exponential backoff

packit_service/worker/build/copr_build.py

packit_service/models.py

packit_service/worker/tasks.py

softwarefactory-project-zuul · 2020-03-13T17:41:06Z

Build failed.

packit-service-pre-commit : SUCCESS in 2m 54s
packit-service-tests : SUCCESS in 17m 44s
packit-service-tests-requre : FAILURE in 18m 57s

TomasTomecek · 2020-03-13T17:54:53Z

two requre tests broke

lachmanfrantisek · 2020-03-16T08:07:18Z

two requre tests broke

Let's check on other PRs if it fails everywhere.

It may be caused by an API change in pygithub or Github.

github.GithubException.GithubException: 401 {"message":"'Issuer' claim ('iss') must be an Integer","documentation_url":"https://developer.github.com/v3"}

sakalosj · 2020-03-16T09:18:31Z

LGTM 👍

TomasTomecek · 2020-03-16T19:51:53Z

I'm definitely cursed - "simple" PR turned into... touch and change everything

softwarefactory-project-zuul · 2020-03-16T20:15:43Z

Build failed.

packit-service-pre-commit : SUCCESS in 2m 39s
packit-service-tests : FAILURE in 22m 54s
packit-service-tests-requre : FAILURE in 19m 52s

lachmanfrantisek · 2020-03-17T07:07:41Z

Makefile

@@ -38,6 +38,33 @@ check_in_container: test_image
 		-v $(CURDIR)/files/packit-service.yaml:/root/.config/packit-service.yaml \
 		$(TEST_IMAGE) make check

+# This is my target so don't touch it! :) How to:


so... the target for running tests in a container is not good enough for me and I keep changing it all the time when working on p-s: so I decided to start my own target (which everyone can use)

Oh, I understood that as "don't touch it and don't try to use it"..;)

lachmanfrantisek · 2020-03-17T07:08:39Z

files/deployment.yaml

+
+    # path_to_secrets = this is what we get from the outside
+    # internal_secrets_path = this is what we'll use here
+    - set_fact: secrets_path={{ deployment_dir }}/secrets/dev


Can you please send the deployment changes in another PR? (This started to be really unreviewable.)

lachmanfrantisek · 2020-03-17T07:10:28Z

packit_service/service/events.py

@@ -188,7 +188,9 @@ def __init__(self, trigger: TheJobTriggerType, project_url: str):
        )

    def get_project(self) -> GitProject:
-        return ServiceConfig.get_service_config().get_project(url=self.project_url)
+        project = ServiceConfig.get_service_config().get_project(url=self.project_url)
+        logger.debug(f"GitService = {project.service.__dict__}")


This is very dangerous -- print the str representation instead so we don't log service secrets.

dangerous where? that's literally the point here: print the secrets so we know what's the setup

I mean, I can drop that commit since the issue is resolved - it's just these log lines helped me resolve it

thinking... it's true that we share srpm logs with people and this could potentially be printed in there, I'll drop it

The str version contains the first and last character of the tokens to be sure we are using the right tokens...

softwarefactory-project-zuul · 2020-03-17T08:34:26Z

Build failed.

packit-service-pre-commit : SUCCESS in 2m 29s
packit-service-tests : SUCCESS in 15m 13s
packit-service-tests-requre : FAILURE in 18m 54s

softwarefactory-project-zuul · 2020-03-17T11:06:54Z

Build failed.

packit-service-pre-commit : SUCCESS in 2m 35s
packit-service-tests : SUCCESS in 16m 18s
packit-service-tests-requre : FAILURE in 21m 20s

softwarefactory-project-zuul · 2020-03-17T12:12:30Z

Build failed.

packit-service-pre-commit : SUCCESS in 2m 35s
packit-service-tests : SUCCESS in 13m 43s
packit-service-tests-requre : FAILURE in 20m 24s

softwarefactory-project-zuul · 2020-03-17T15:33:29Z

Build failed.

packit-service-pre-commit : SUCCESS in 3m 35s
packit-service-tests : SUCCESS in 14m 54s
packit-service-tests-requre : FAILURE in 21m 17s

softwarefactory-project-zuul · 2020-03-17T16:33:24Z

Build succeeded.

packit-service-pre-commit : SUCCESS in 3m 31s
packit-service-tests : SUCCESS in 15m 20s
packit-service-tests-requre : SUCCESS in 21m 41s

csomh · 2020-03-17T16:52:05Z

packit_service/worker/tasks.py

+    bind=True,
+    name="task.babysit_copr_build",
+    # 120 minutes
+    default_retry_delay=120 * 60,


Why is this set to 120 minutes? countdown in the retry() bellow will override this to be 60 seconds, and when the task is sent, it's also 60 seconds.

Another note: isn't doing this work every minute too often? The cadence is very close to what we would get just listening to the message bus. I would go with 5 minutes, but I can imagine even 15 minutes would be fine. Longer times seem to be a better fit for what this functionality would try to achieve: pick up the leftovers.

How many times will this retry? 3?

Oh, and how about making all these values part of the configuration, so that we don't have to touch the code, to change them? Would help a lot in fine tuning operations.

yeah, this is a good point

so I changed it to use the exponential backoff, current setup:

do the first check in 120s, then next one in 60s, next 120s, 240s, 480s, etc.

would that be okay?

(btw after we merge this, I'm stoked to see how it goes on stg and will fix all the issues - which should be much easier to do and review once we see it in action)

also, I set max retries to 7

thanks for such a thorough review

softwarefactory-project-zuul · 2020-03-17T19:22:43Z

Build succeeded.

packit-service-pre-commit : SUCCESS in 2m 42s
packit-service-tests : SUCCESS in 16m 19s
packit-service-tests-requre : SUCCESS in 14m 28s

TomasTomecek · 2020-03-17T19:30:24Z

please don't merge this before fixing #504

we probably need to deploy to prod tomorrow, we can merge this PR after we fix the prod TFT problems

softwarefactory-project-zuul · 2020-03-18T10:42:04Z

Build failed.

packit-service-pre-commit : SUCCESS in 6m 44s
packit-service-tests : RETRY_LIMIT in 11m 14s
packit-service-tests-requre : RETRY_LIMIT in 13m 36s

TomasTomecek · 2020-03-18T13:51:56Z

recheck

dhodovsk

LGTM

dhodovsk · 2020-03-18T12:44:00Z

packit_service/worker/tasks.py

+    name="task.babysit_copr_build",
+    retry_backoff=60,  # retry again in 60s, 120s, 240s, 480s...
+    retry_backoff_max=60 * 60 * 8,  # is 8 hours okay? gcc/kernel build really long
+    max_retries=7,


Does this work ok? In the celery doc, it is specified like this: retry_kwargs={'max_retries': 5}.

I was actually checking the celery code itself, and they are picking it from these kwargs, so, it should work

softwarefactory-project-zuul · 2020-03-18T14:05:46Z

Build failed.

packit-service-pre-commit : SUCCESS in 2m 39s
packit-service-tests : RETRY_LIMIT in 11m 34s
packit-service-tests-requre : RETRY_LIMIT in 12m 42s

no longer needed, builds in the past 2 weeks are in psql Signed-off-by: Tomas Tomecek <ttomecek@redhat.com>

Signed-off-by: Tomas Tomecek <ttomecek@redhat.com>

softwarefactory-project-zuul · 2020-03-18T17:26:43Z

Build succeeded.

packit-service-pre-commit : SUCCESS in 2m 51s
packit-service-tests : SUCCESS in 14m 30s
packit-service-tests-requre : SUCCESS in 16m 25s

softwarefactory-project-zuul · 2020-03-18T17:44:06Z

Build succeeded (gate pipeline).

packit-service-pre-commit : SUCCESS in 2m 43s
packit-service-tests : SUCCESS in 14m 39s
packit-service-tests-requre : SUCCESS in 16m 13s

TomasTomecek · 2020-03-18T17:45:10Z

looking fwd fixing stg tomorrow!

lachmanfrantisek reviewed Mar 13, 2020

View reviewed changes

TomasTomecek force-pushed the fedora-massaging branch from d2b1f3c to 144a231 Compare March 13, 2020 17:20

lachmanfrantisek mentioned this pull request Mar 16, 2020

[test] test requre test #491

Closed

TomasTomecek force-pushed the fedora-massaging branch from 144a231 to a58a971 Compare March 16, 2020 19:50

TomasTomecek changed the title ~~WIP: periodically check the status of a copr build~~ periodically check the status of a copr build Mar 16, 2020

TomasTomecek added the ready-for-review Pull request is ready for review. label Mar 16, 2020

lachmanfrantisek reviewed Mar 17, 2020

View reviewed changes

TomasTomecek changed the title ~~periodically check the status of a copr build~~ WIP: periodically check the status of a copr build Mar 17, 2020

TomasTomecek force-pushed the fedora-massaging branch from a58a971 to 22d509c Compare March 17, 2020 08:14

TomasTomecek force-pushed the fedora-massaging branch from 22d509c to 62a505c Compare March 17, 2020 10:44

TomasTomecek force-pushed the fedora-massaging branch from 62a505c to 13c16cc Compare March 17, 2020 11:50

TomasTomecek changed the title ~~WIP: periodically check the status of a copr build~~ periodically check the status of a copr build Mar 17, 2020

TomasTomecek force-pushed the fedora-massaging branch from 13c16cc to ab07d60 Compare March 17, 2020 15:10

TomasTomecek force-pushed the fedora-massaging branch from ab07d60 to 2afcad0 Compare March 17, 2020 16:10

TomasTomecek mentioned this pull request Mar 17, 2020

2 requre tests break after #490 is in #503

Closed

csomh reviewed Mar 17, 2020

View reviewed changes

TomasTomecek force-pushed the fedora-massaging branch from 2afcad0 to 6bc807f Compare March 17, 2020 19:05

csomh added the do-not-merge Work in progress. label Mar 18, 2020

TomasTomecek force-pushed the fedora-massaging branch from 6bc807f to da5d03c Compare March 18, 2020 10:27

TomasTomecek removed the do-not-merge Work in progress. label Mar 18, 2020

dhodovsk approved these changes Mar 18, 2020

View reviewed changes

TomasTomecek added 4 commits March 18, 2020 18:08

CoprBuildEndHandler: drop redis code

97f9300

no longer needed, builds in the past 2 weeks are in psql Signed-off-by: Tomas Tomecek <ttomecek@redhat.com>

rename: get_all_{,by_}build_id

65b8de2

Signed-off-by: Tomas Tomecek <ttomecek@redhat.com>

requre: merge two decorates together

41ec18b

Signed-off-by: Tomas Tomecek <ttomecek@redhat.com>

babysit copr builds we submit

bf6013b

Signed-off-by: Tomas Tomecek <ttomecek@redhat.com>

TomasTomecek force-pushed the fedora-massaging branch from da5d03c to bf6013b Compare March 18, 2020 17:08

TomasTomecek added the mergeit When set, zuul wil gate and merge the PR. label Mar 18, 2020

softwarefactory-project-zuul bot merged commit 8efe455 into packit:master Mar 18, 2020

TomasTomecek deleted the fedora-massaging branch March 18, 2020 17:45

periodically check the status of a copr build #490

periodically check the status of a copr build #490

Conversation

TomasTomecek commented Mar 13, 2020 • edited

softwarefactory-project-zuul bot commented Mar 13, 2020

lachmanfrantisek left a comment • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

softwarefactory-project-zuul bot commented Mar 13, 2020

TomasTomecek commented Mar 13, 2020

lachmanfrantisek commented Mar 16, 2020

sakalosj commented Mar 16, 2020

TomasTomecek commented Mar 16, 2020

softwarefactory-project-zuul bot commented Mar 16, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lachmanfrantisek Mar 17, 2020 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

softwarefactory-project-zuul bot commented Mar 17, 2020

softwarefactory-project-zuul bot commented Mar 17, 2020

softwarefactory-project-zuul bot commented Mar 17, 2020

softwarefactory-project-zuul bot commented Mar 17, 2020

softwarefactory-project-zuul bot commented Mar 17, 2020

Choose a reason for hiding this comment

csomh Mar 17, 2020 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

softwarefactory-project-zuul bot commented Mar 17, 2020

TomasTomecek commented Mar 17, 2020

softwarefactory-project-zuul bot commented Mar 18, 2020

TomasTomecek commented Mar 18, 2020

dhodovsk left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

TomasTomecek Mar 18, 2020 • edited

Choose a reason for hiding this comment

softwarefactory-project-zuul bot commented Mar 18, 2020

softwarefactory-project-zuul bot commented Mar 18, 2020

softwarefactory-project-zuul bot commented Mar 18, 2020

TomasTomecek commented Mar 18, 2020

TomasTomecek commented Mar 13, 2020 •

edited

lachmanfrantisek left a comment •

edited

lachmanfrantisek Mar 17, 2020 •

edited

csomh Mar 17, 2020 •

edited

TomasTomecek Mar 18, 2020 •

edited