Mark 404'd Repos to be Ignored and Re-collect Errored Repos #2678

IsaacMilarky · 2024-02-08T14:43:00Z

Description

Add a new CollectionState enum value called 'IGNORE'. This marks the repos to be ignored.
Fix some import issues where CollectionState was being defined multiple times. I have moved this class to its own file in an attempt to make sure that we only define enums once.
Add periodic task to set errored repos to pending daily

Signed commits

Yes, I signed my commits.

Signed-off-by: Isaac Milarsky <isaac.milarsky@hhs.cms.gov>

…ndby Signed-off-by: Isaac Milarsky <isaac.milarsky@hhs.cms.gov>

github-actions

Remaining comments which cannot be posted as a review comment to avoid GitHub Rate Limit

pylint

augur/tasks/util/collection_state.py|1| C0114: Missing module docstring (missing-module-docstring)
augur/tasks/util/collection_state.py|3| C0115: Missing class docstring (missing-class-docstring)
augur/tasks/util/collection_util.py|30| C0116: Missing function or method docstring (missing-function-docstring)

augur/tasks/git/facade_tasks.py

augur/tasks/github/detect_move/core.py

augur/tasks/init/celery_app.py

github-actions · 2024-02-08T14:43:45Z

augur/tasks/start_tasks.py

@@ -36,6 +36,7 @@
 from enum import Enum
 from augur.tasks.util.redis_list import RedisList
 from augur.application.db.models import CollectionStatus, Repo
+from augur.tasks.util.collection_state import CollectionState
 from augur.tasks.util.collection_util import *
 from augur.tasks.git.util.facade_worker.facade_worker.utilitymethods import get_facade_weight_time_factor


[pylint] _{reported by reviewdog 🐶}
C0413: Import "from augur.tasks.git.util.facade_worker.facade_worker.utilitymethods import get_facade_weight_time_factor" should be placed at the top of the module (wrong-import-position)

github-actions · 2024-02-08T14:43:45Z

augur/tasks/start_tasks.py

@@ -328,6 +329,22 @@ def augur_collection_update_weights():
            session.commit()
            #git_update_commit_count_weight(repo_git)

+@celery.task
+def retry_404_repos():


[pylint] _{reported by reviewdog 🐶}
C0116: Missing function or method docstring (missing-function-docstring)

augur/tasks/start_tasks.py

github-actions · 2024-02-08T14:43:45Z

augur/tasks/start_tasks.py

+        session.execute_sql(query)
+
+
+
 #Retry this task for every issue so that repos that were added manually get the chance to be added to the collection_status table.
 @celery.task(autoretry_for=(Exception,), retry_backoff=True, retry_backoff_max=300, retry_jitter=True, max_retries=None)
 def create_collection_status_records():


[pylint] _{reported by reviewdog 🐶}
C0116: Missing function or method docstring (missing-function-docstring)

github-actions · 2024-02-08T14:43:45Z

augur/tasks/start_tasks.py

@@ -36,6 +36,7 @@
 from enum import Enum


[pylint] _{reported by reviewdog 🐶}
C0412: Imports from package enum are not grouped (ungrouped-imports)

Signed-off-by: Isaac Milarsky <isaac.milarsky@hhs.cms.gov>

augur/tasks/github/detect_move/core.py

github-actions · 2024-02-08T15:01:23Z

augur/tasks/util/collection_state.py

@@ -0,0 +1,12 @@
+


[pylint] _{reported by reviewdog 🐶}
C0114: Missing module docstring (missing-module-docstring)

augur/tasks/util/collection_state.py

github-actions · 2024-02-08T15:01:23Z

augur/tasks/util/collection_util.py

-    INITIALIZING = "Initializing"
-    UPDATE = "Update"
-    FAILED_CLONE = "Failed Clone"
-
 def get_list_of_all_users(session):


[pylint] _{reported by reviewdog 🐶}
C0116: Missing function or method docstring (missing-function-docstring)

Signed-off-by: Isaac Milarsky <isaac.milarsky@hhs.cms.gov>

github-actions · 2024-02-08T15:55:14Z

augur/tasks/init/celery_app.py

@@ -209,7 +201,7 @@ def setup_periodic_tasks(sender, **kwargs):
    """
    from celery.schedules import crontab
    from augur.tasks.start_tasks import augur_collection_monitor, augur_collection_update_weights
-    from augur.tasks.start_tasks import non_repo_domain_tasks
+    from augur.tasks.start_tasks import non_repo_domain_tasks, retry_errored_repos


[pylint] _{reported by reviewdog 🐶}
C0415: Import outside toplevel (augur.tasks.start_tasks.non_repo_domain_tasks, augur.tasks.start_tasks.retry_errored_repos) (import-outside-toplevel)

github-actions · 2024-02-08T15:55:14Z

augur/tasks/start_tasks.py

@@ -328,6 +329,22 @@ def augur_collection_update_weights():
            session.commit()
            #git_update_commit_count_weight(repo_git)

+@celery.task
+def retry_errored_repos():


[pylint] _{reported by reviewdog 🐶}
C0116: Missing function or method docstring (missing-function-docstring)

Signed-off-by: Isaac Milarsky <isaac.milarsky@hhs.cms.gov>

github-actions · 2024-02-09T14:40:43Z

augur/tasks/start_tasks.py

+    from augur.tasks.init.celery_app import engine
+    logger = logging.getLogger(create_collection_status_records.__name__)
+
+    #TODO: Isaac needs to normalize the status's to be abstract in the 


[pylint] _{reported by reviewdog 🐶}
W0511: TODO: Isaac needs to normalize the status's to be abstract in the (fixme)

Signed-off-by: Isaac Milarsky <isaac.milarsky@hhs.cms.gov>

github-actions · 2024-02-09T14:51:39Z

augur/tasks/start_tasks.py

@@ -33,9 +33,9 @@
 from augur.tasks.init.celery_app import celery_app as celery


[pylint] _{reported by reviewdog 🐶}
C0413: Import "from augur.tasks.init.celery_app import celery_app as celery" should be placed at the top of the module (wrong-import-position)

github-actions · 2024-02-09T14:51:39Z

augur/tasks/start_tasks.py

@@ -33,9 +33,9 @@
 from augur.tasks.init.celery_app import celery_app as celery
 from augur.application.db.session import DatabaseSession


[pylint] _{reported by reviewdog 🐶}
C0413: Import "from augur.application.db.session import DatabaseSession" should be placed at the top of the module (wrong-import-position)

github-actions · 2024-02-09T14:51:39Z

augur/tasks/start_tasks.py

@@ -33,9 +33,9 @@
 from augur.tasks.init.celery_app import celery_app as celery
 from augur.application.db.session import DatabaseSession
 from logging import Logger


[pylint] _{reported by reviewdog 🐶}
C0413: Import "from logging import Logger" should be placed at the top of the module (wrong-import-position)

github-actions · 2024-02-09T14:51:39Z

augur/tasks/start_tasks.py

+
+    A special celery task that automatically retries itself and has no max retries.
+    """
+
    from augur.tasks.init.celery_app import engine


[pylint] _{reported by reviewdog 🐶}
C0415: Import outside toplevel (augur.tasks.init.celery_app.engine) (import-outside-toplevel)

Signed-off-by: Isaac Milarsky <isaac.milarsky@hhs.cms.gov>

ABrain7710

The logic in ping_github_for_repo_move doesn't appear to be quite right. It appears to not be setting the repos that return a 404 (deleted repos) to a status of IGNORE. It throwing an excpetion for 404, succeeding if it the status wasn't a 301, and setting the repos that are 301 (moved) to IGNORE. We need to be setting 404s to IGNORE and erroring repos that are 301 so they can be retried

sgoggins · 2024-02-12T13:40:20Z

@IsaacMilarky : Is the question @ABrain7710 raised addressed?

The logic in ping_github_for_repo_move doesn't appear to be quite right. It appears to not be setting the repos that return a 404 (deleted repos) to a status of IGNORE. It throwing an excpetion for 404, succeeding if it the status wasn't a 301, and setting the repos that are 301 (moved) to IGNORE. We need to be setting 404s to IGNORE and erroring repos that are 301 so they can be retried

sgoggins

LGTM, waiting to see if @ABrain7710 's comment is addressed before merging.

ABrain7710 · 2024-02-12T13:44:26Z

@sgoggins No the requested changes have not been addressed. I messaged Isaac and he agreed the logic is off

Signed-off-by: Isaac Milarsky <krabs@tilde.team>

github-actions · 2024-02-12T23:49:50Z

augur/application/cli/_multicommand.py

@@ -27,7 +27,7 @@ def get_command(self, ctx, name):
        try:
            module = importlib.import_module('.' + name, 'augur.application.cli')
            return module.cli
-        except ModuleNotFoundError:
+        except ModuleNotFoundError as e:


[pylint] _{reported by reviewdog 🐶}
W0612: Unused variable 'e' (unused-variable)

ABrain7710

I believe we need to be updating all the collection statuses to IGNORE so that we don't collect facade or ml on deleted repos

Signed-off-by: Isaac Milarsky <krabs@tilde.team>

IsaacMilarky added 4 commits February 6, 2024 16:36

start on changes I think andrew wanted

08666c9

Signed-off-by: Isaac Milarsky <isaac.milarsky@hhs.cms.gov>

subclasses not needed

a8bf6f7

Signed-off-by: Isaac Milarsky <isaac.milarsky@hhs.cms.gov>

implement change

1934a2d

Signed-off-by: Isaac Milarsky <isaac.milarsky@hhs.cms.gov>

fix enum import issues and add mechanism to retry repos marked as Sta…

01c5c73

…ndby Signed-off-by: Isaac Milarsky <isaac.milarsky@hhs.cms.gov>

github-actions bot reviewed Feb 8, 2024

View reviewed changes

add docstring to update_repo_with_dict

801e5e0

Signed-off-by: Isaac Milarsky <isaac.milarsky@hhs.cms.gov>

github-actions bot reviewed Feb 8, 2024

View reviewed changes

fix misunderstanding in implementation

9a76471

Signed-off-by: Isaac Milarsky <isaac.milarsky@hhs.cms.gov>

github-actions bot reviewed Feb 8, 2024

View reviewed changes

fix format

5a77d65

Signed-off-by: Isaac Milarsky <isaac.milarsky@hhs.cms.gov>

IsaacMilarky changed the title ~~Allow 404 Repos to Automatically be Re-collected~~ Mark 404 Errors to be Ignored and Re-collect Errored Repos Feb 8, 2024

IsaacMilarky changed the title ~~Mark 404 Errors to be Ignored and Re-collect Errored Repos~~ Mark 404'd Repos to be Ignored and Re-collect Errored Repos Feb 8, 2024

retry all status'

23f17e9

Signed-off-by: Isaac Milarsky <isaac.milarsky@hhs.cms.gov>

github-actions bot reviewed Feb 9, 2024

View reviewed changes

linting

b70f524

Signed-off-by: Isaac Milarsky <isaac.milarsky@hhs.cms.gov>

github-actions bot reviewed Feb 9, 2024

View reviewed changes

doc-string

68e50de

Signed-off-by: Isaac Milarsky <isaac.milarsky@hhs.cms.gov>

IsaacMilarky marked this pull request as ready for review February 9, 2024 18:40

IsaacMilarky assigned ABrain7710 Feb 9, 2024

IsaacMilarky added the add-feature Adds new features label Feb 9, 2024

IsaacMilarky requested review from ABrain7710 and sgoggins February 9, 2024 18:51

ABrain7710 requested changes Feb 11, 2024

View reviewed changes

sgoggins approved these changes Feb 12, 2024

View reviewed changes

implement requested changes

9aa9725

Signed-off-by: Isaac Milarsky <krabs@tilde.team>

IsaacMilarky requested a review from ABrain7710 February 12, 2024 16:24

IsaacMilarky added 2 commits February 12, 2024 10:26

Merge branch 'dev' into allow-repo-retry-collection-isaac

45be12f

typo causing cli error

4c8edae

Signed-off-by: Isaac Milarsky <krabs@tilde.team>

github-actions bot reviewed Feb 12, 2024

View reviewed changes

ABrain7710 requested changes Feb 12, 2024

View reviewed changes

IsaacMilarky added 2 commits February 12, 2024 18:08

make sure all status are updated on 404 as per andrew suggestion

be5c19c

Signed-off-by: Isaac Milarsky <krabs@tilde.team>

i need to normalize this for the love of gosh

0a52999

Signed-off-by: Isaac Milarsky <krabs@tilde.team>

ABrain7710 self-requested a review February 13, 2024 00:34

ABrain7710 approved these changes Feb 13, 2024

View reviewed changes

ABrain7710 merged commit 69788d1 into dev Feb 13, 2024
7 of 8 checks passed

sgoggins deleted the allow-repo-retry-collection-isaac branch February 20, 2024 00:55

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Mark 404'd Repos to be Ignored and Re-collect Errored Repos #2678

Mark 404'd Repos to be Ignored and Re-collect Errored Repos #2678

IsaacMilarky commented Feb 8, 2024 •

edited

github-actions bot left a comment

github-actions bot Feb 8, 2024

github-actions bot Feb 8, 2024

github-actions bot Feb 8, 2024

github-actions bot Feb 8, 2024

github-actions bot Feb 8, 2024

github-actions bot Feb 8, 2024

github-actions bot Feb 8, 2024

github-actions bot Feb 8, 2024

github-actions bot Feb 9, 2024

github-actions bot Feb 9, 2024

github-actions bot Feb 9, 2024

github-actions bot Feb 9, 2024

github-actions bot Feb 9, 2024

ABrain7710 left a comment

sgoggins commented Feb 12, 2024

sgoggins left a comment

ABrain7710 commented Feb 12, 2024

github-actions bot Feb 12, 2024

ABrain7710 left a comment

		@@ -33,9 +33,9 @@
		from augur.tasks.init.celery_app import celery_app as celery

		@@ -33,9 +33,9 @@
		from augur.tasks.init.celery_app import celery_app as celery
		from augur.application.db.session import DatabaseSession

Mark 404'd Repos to be Ignored and Re-collect Errored Repos #2678

Mark 404'd Repos to be Ignored and Re-collect Errored Repos #2678

Conversation

IsaacMilarky commented Feb 8, 2024 • edited

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot Feb 8, 2024

Choose a reason for hiding this comment

github-actions bot Feb 8, 2024

Choose a reason for hiding this comment

github-actions bot Feb 8, 2024

Choose a reason for hiding this comment

github-actions bot Feb 8, 2024

Choose a reason for hiding this comment

github-actions bot Feb 8, 2024

Choose a reason for hiding this comment

github-actions bot Feb 8, 2024

Choose a reason for hiding this comment

github-actions bot Feb 8, 2024

Choose a reason for hiding this comment

github-actions bot Feb 8, 2024

Choose a reason for hiding this comment

github-actions bot Feb 9, 2024

Choose a reason for hiding this comment

github-actions bot Feb 9, 2024

Choose a reason for hiding this comment

github-actions bot Feb 9, 2024

Choose a reason for hiding this comment

github-actions bot Feb 9, 2024

Choose a reason for hiding this comment

github-actions bot Feb 9, 2024

Choose a reason for hiding this comment

ABrain7710 left a comment

Choose a reason for hiding this comment

sgoggins commented Feb 12, 2024

sgoggins left a comment

Choose a reason for hiding this comment

ABrain7710 commented Feb 12, 2024

github-actions bot Feb 12, 2024

Choose a reason for hiding this comment

ABrain7710 left a comment

Choose a reason for hiding this comment

IsaacMilarky commented Feb 8, 2024 •

edited