Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Release Ready #2812

Merged
merged 139 commits into from
Jun 10, 2024
Merged

Release Ready #2812

merged 139 commits into from
Jun 10, 2024

Conversation

Signed-off-by: Andrew Brain <andrewbrain2019@gmail.com>
Signed-off-by: Andrew Brain <andrewbrain2019@gmail.com>
Signed-off-by: Andrew Brain <andrewbrain2019@gmail.com>
Signed-off-by: Andrew Brain <andrewbrain2019@gmail.com>
Signed-off-by: Andrew Brain <andrewbrain2019@gmail.com>
Signed-off-by: Andrew Brain <andrewbrain2019@gmail.com>
Signed-off-by: Andrew Brain <andrewbrain2019@gmail.com>
Signed-off-by: Andrew Brain <andrewbrain2019@gmail.com>
Signed-off-by: Andrew Brain <andrewbrain2019@gmail.com>
Signed-off-by: Andrew Brain <andrewbrain2019@gmail.com>
Signed-off-by: Andrew Brain <andrewbrain2019@gmail.com>
Signed-off-by: Andrew Brain <andrewbrain2019@gmail.com>
Signed-off-by: Andrew Brain <andrewbrain2019@gmail.com>
Signed-off-by: Andrew Brain <andrewbrain2019@gmail.com>
Signed-off-by: Andrew Brain <andrewbrain2019@gmail.com>
Signed-off-by: Andrew Brain <andrewbrain2019@gmail.com>
Signed-off-by: Andrew Brain <andrewbrain2019@gmail.com>
Signed-off-by: Andrew Brain <andrewbrain2019@gmail.com>
Signed-off-by: Andrew Brain <andrewbrain2019@gmail.com>
Signed-off-by: Andrew Brain <andrewbrain2019@gmail.com>
Signed-off-by: Andrew Brain <andrewbrain2019@gmail.com>
Signed-off-by: Andrew Brain <andrewbrain2019@gmail.com>
Signed-off-by: Andrew Brain <andrewbrain2019@gmail.com>
Signed-off-by: Andrew Brain <andrewbrain2019@gmail.com>
Signed-off-by: Andrew Brain <andrewbrain2019@gmail.com>
Signed-off-by: Andrew Brain <andrewbrain2019@gmail.com>
Signed-off-by: Andrew Brain <andrewbrain2019@gmail.com>
Signed-off-by: Andrew Brain <andrewbrain2019@gmail.com>
Signed-off-by: Andrew Brain <andrewbrain2019@gmail.com>
Signed-off-by: Sean P. Goggins <s@goggins.com>
Copy link

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remaining comments which cannot be posted as a review comment to avoid GitHub Rate Limit

pylint

augur/tasks/util/collection_util.py|300 col 30| R1704: Redefining argument with the local name 'session' (redefined-argument-from-local)
augur/tasks/util/collection_util.py|404 col 4| W0612: Unused variable 'engine' (unused-variable)
augur/tasks/util/collection_util.py|587 col 8| W0612: Unused variable 'augur_collection_list' (unused-variable)
tests/test_tasks/test_task_utlities/test_key_handler/test_github_api_key_handler.py|28 col 24| W0621: Redefining name 'key_handler' from outer scope (line 22) (redefined-outer-name)


return all_data


def process_clones_data(clones_data, task_name, repo_id, logger, augur_db) -> None:
def process_clones_data(clones_data, task_name, repo_id, logger) -> None:
clone_history_data = clones_data[0]['clones']

clone_history_data_dicts = extract_needed_clone_history_data(clone_history_data, repo_id)

clone_history_data = remove_duplicate_dicts(clone_history_data_dicts, 'clone_data_timestamp')
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[pylint] reported by reviewdog 🐶
E1121: Too many positional arguments for function call (too-many-function-args)

@@ -6,8 +6,7 @@
from sqlalchemy.orm import Session

from augur.tasks.util.redis_list import RedisList
from augur.application.db.session import DatabaseSession
from augur.application.db.lib import get_value
from augur.application.db.lib import get_value, get_worker_oauth_keys
from sqlalchemy import func
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[pylint] reported by reviewdog 🐶
C0412: Imports from package sqlalchemy are not grouped (ungrouped-imports)


if issue_data:
issue_ids = process_issues(issue_data, f"{owner}/{repo}: Gitlab Issue task", repo_id, logger, augur_db)
if issue_data:
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[pylint] reported by reviewdog 🐶
R1705: Unnecessary "else" after "return", remove the "else" and de-indent the code inside it (no-else-return)

@@ -87,7 +83,7 @@

return all_data

def process_issues(issues, task_name, repo_id, logger, augur_db) -> None:
def process_issues(issues, task_name, repo_id, logger) -> None:
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[pylint] reported by reviewdog 🐶
R0914: Too many local variables (31/30) (too-many-locals)

@@ -87,7 +83,7 @@

return all_data

def process_issues(issues, task_name, repo_id, logger, augur_db) -> None:
def process_issues(issues, task_name, repo_id, logger) -> None:
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[pylint] reported by reviewdog 🐶
R1710: Either all return statements in a function should return an expression, or none of them should. (inconsistent-return-statements)

@@ -217,7 +218,7 @@
request.get_valid_repos(session)
return request

def build_ml_repo_collect_request(session,enabled_phase_names, days_until_collect_again = 1):
def build_ml_repo_collect_request(session, logger, enabled_phase_names, days_until_collect_again = 1):
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[pylint] reported by reviewdog 🐶
W0613: Unused argument 'logger' (unused-argument)

@@ -217,7 +218,7 @@
request.get_valid_repos(session)
return request

def build_ml_repo_collect_request(session,enabled_phase_names, days_until_collect_again = 1):
def build_ml_repo_collect_request(session, logger, enabled_phase_names, days_until_collect_again = 1):
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[pylint] reported by reviewdog 🐶
W0613: Unused argument 'enabled_phase_names' (unused-argument)

@@ -217,7 +218,7 @@
request.get_valid_repos(session)
return request

def build_ml_repo_collect_request(session,enabled_phase_names, days_until_collect_again = 1):
def build_ml_repo_collect_request(session, logger, enabled_phase_names, days_until_collect_again = 1):
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[pylint] reported by reviewdog 🐶
W0613: Unused argument 'days_until_collect_again' (unused-argument)

@@ -278,7 +282,7 @@

logger.info("Updating stale collection weights")

with DatabaseSession(logger,engine) as session:
with get_session() as session:

core_weight_update_repos = session.query(CollectionStatus).filter(CollectionStatus.core_weight != None).all()
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[pylint] reported by reviewdog 🐶
C0121: Comparison 'CollectionStatus.core_weight != None' should be 'CollectionStatus.core_weight is not None' (singleton-comparison)

@@ -361,16 +364,17 @@
engine = self.app.engine
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[pylint] reported by reviewdog 🐶
W0612: Unused variable 'engine' (unused-variable)

Signed-off-by: Sean P. Goggins <s@goggins.com>

request.get_valid_repos(session)
return request


def build_facade_repo_collect_request(session,enabled_phase_names, days_until_collect_again = 1):
def build_facade_repo_collect_request(session, logger, enabled_phase_names, days_until_collect_again = 10):
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[pylint] reported by reviewdog 🐶
W0613: Unused argument 'logger' (unused-argument)


request.get_valid_repos(session)
return request


def build_facade_repo_collect_request(session,enabled_phase_names, days_until_collect_again = 1):
def build_facade_repo_collect_request(session, logger, enabled_phase_names, days_until_collect_again = 10):
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[pylint] reported by reviewdog 🐶
W0613: Unused argument 'enabled_phase_names' (unused-argument)


request.get_valid_repos(session)
return request


def build_facade_repo_collect_request(session,enabled_phase_names, days_until_collect_again = 1):
def build_facade_repo_collect_request(session, logger, enabled_phase_names, days_until_collect_again = 10):
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[pylint] reported by reviewdog 🐶
W0613: Unused argument 'days_until_collect_again' (unused-argument)


request.get_valid_repos(session)
return request

def build_ml_repo_collect_request(session,enabled_phase_names, days_until_collect_again = 1):
def build_ml_repo_collect_request(session, logger, enabled_phase_names, days_until_collect_again = 40):
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[pylint] reported by reviewdog 🐶
W0613: Unused argument 'logger' (unused-argument)


request.get_valid_repos(session)
return request

def build_ml_repo_collect_request(session,enabled_phase_names, days_until_collect_again = 1):
def build_ml_repo_collect_request(session, logger, enabled_phase_names, days_until_collect_again = 40):
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[pylint] reported by reviewdog 🐶
W0613: Unused argument 'enabled_phase_names' (unused-argument)


request.get_valid_repos(session)
return request

def build_ml_repo_collect_request(session,enabled_phase_names, days_until_collect_again = 1):
def build_ml_repo_collect_request(session, logger, enabled_phase_names, days_until_collect_again = 40):
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[pylint] reported by reviewdog 🐶
W0613: Unused argument 'days_until_collect_again' (unused-argument)

@@ -301,7 +297,7 @@ def issue_pr_task_update_weight_util(self, issue_and_pr_nums,repo_git=None,sessi
if session is not None:
update_issue_pr_weights(logger, session, repo_git, sum(issue_and_pr_nums))
else:
with DatabaseSession(logger,engine=engine) as session:
with get_session() as session:
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[pylint] reported by reviewdog 🐶
R1704: Redefining argument with the local name 'session' (redefined-argument-from-local)

@@ -407,7 +403,7 @@

engine = get_engine()
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[pylint] reported by reviewdog 🐶
W0612: Unused variable 'engine' (unused-variable)

with get_session() as session:

for repo_git, task_id, hook_name in self.send_messages():
self.update_status_and_id(repo_git,task_id,hook_name, session)

def send_messages(self):
augur_collection_list = []
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[pylint] reported by reviewdog 🐶
W0612: Unused variable 'augur_collection_list' (unused-variable)

@@ -23,7 +23,7 @@ def key_handler(test_db_session):

redis.flushdb()

yield GithubApiKeyHandler(test_db_session)
yield GithubApiKeyHandler(logger)

def test_get_config_key(key_handler, test_db_engine):
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[pylint] reported by reviewdog 🐶
W0621: Redefining name 'key_handler' from outer scope (line 22) (redefined-outer-name)

Signed-off-by: Sean P. Goggins <s@goggins.com>
@@ -153,7 +154,7 @@ def non_repo_domain_tasks(self):
tasks.apply_async()


def build_primary_repo_collect_request(session,enabled_phase_names, days_until_collect_again = 1):
def build_primary_repo_collect_request(session, logger, enabled_phase_names, days_until_collect_again = 15):
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[pylint] reported by reviewdog 🐶
W0613: Unused argument 'logger' (unused-argument)

@@ -153,7 +154,7 @@
tasks.apply_async()


def build_primary_repo_collect_request(session,enabled_phase_names, days_until_collect_again = 1):
def build_primary_repo_collect_request(session, logger, enabled_phase_names, days_until_collect_again = 15):
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[pylint] reported by reviewdog 🐶
W0613: Unused argument 'days_until_collect_again' (unused-argument)

@sgoggins sgoggins requested a review from Ulincsys June 7, 2024 22:40
sgoggins and others added 2 commits June 8, 2024 17:42
Signed-off-by: Sean P. Goggins <s@goggins.com>
…ndary data is falling behind. So I am tweaking the numbers.

Signed-off-by: Sean Goggins <seangoggins@outdoors@acm.org>
@@ -164,15 +165,15 @@ def determine_worker_processes(ratio,maximum):
process_list.append(subprocess.Popen(scheduling_worker.split(" ")))
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[pylint] reported by reviewdog 🐶
R1732: Consider using 'with' for resource-allocating operations (consider-using-with)

@@ -124,15 +124,15 @@ def determine_worker_processes(ratio,maximum):
process_list.append(subprocess.Popen(scheduling_worker.split(" ")))
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[pylint] reported by reviewdog 🐶
R1732: Consider using 'with' for resource-allocating operations (consider-using-with)

ABrain7710 and others added 8 commits June 9, 2024 18:47
Signed-off-by: Andrew Brain <andrewbrain2019@gmail.com>
Signed-off-by: Sean P. Goggins <s@goggins.com>
Signed-off-by: Sean P. Goggins <s@goggins.com>
Signed-off-by: Sean P. Goggins <s@goggins.com>
Signed-off-by: Sean P. Goggins <s@goggins.com>
Signed-off-by: Sean P. Goggins <s@goggins.com>
PR Files Merge Issue Fix
# Find missing commits and add them
#Get the huge list of commits to process.
absolute_path = get_absolute_repo_path(facade_helper.repo_base_directory, repo.repo_id, repo.repo_path,repo.repo_name)
repo_loc = (f"{absolute_path}/.git")

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[pylint] reported by reviewdog 🐶
C0325: Unnecessary parens after '=' keyword (superfluous-parens)

existing_commits = get_existing_commits_set(session, repo_id)
#Get the huge list of commits to process.
absolute_path = get_absolute_repo_path(facade_helper.repo_base_directory, repo.repo_id, repo.repo_path, repo.repo_name)
repo_loc = (f"{absolute_path}/.git")

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[pylint] reported by reviewdog 🐶
C0325: Unnecessary parens after '=' keyword (superfluous-parens)

quarterQueue = int(len(queue) / 4)
logger.info(f"Got to analysis!")
absolute_path = get_absolute_repo_path(facade_helper.repo_base_directory, repo.repo_id, repo.repo_path,repo.repo_name)
repo_loc = (f"{absolute_path}/.git")

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[pylint] reported by reviewdog 🐶
C0325: Unnecessary parens after '=' keyword (superfluous-parens)

sgoggins and others added 6 commits June 10, 2024 17:08
Signed-off-by: Sean P. Goggins <s@goggins.com>
Signed-off-by: Sean P. Goggins <s@goggins.com>
Signed-off-by: Sean P. Goggins <s@goggins.com>
Signed-off-by: Sean P. Goggins <s@goggins.com>
Xdev: Final Patches for Release
@@ -206,7 +214,6 @@ def process_pull_request_review_contributor(pr_review: dict, tool_source: str, t

return pr_review_cntrb


@celery.task(base=AugurSecondaryRepoCollectionTask)
def collect_pull_request_review_comments(repo_git: str) -> None:

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[pylint] reported by reviewdog 🐶
R0914: Too many local variables (36/30) (too-many-locals)

@@ -206,7 +214,6 @@ def process_pull_request_review_contributor(pr_review: dict, tool_source: str, t

return pr_review_cntrb


@celery.task(base=AugurSecondaryRepoCollectionTask)
def collect_pull_request_review_comments(repo_git: str) -> None:

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[pylint] reported by reviewdog 🐶
R0915: Too many statements (62/50) (too-many-statements)

element.decode('utf-8').replace('\x00', ' ') if isinstance(element, bytes) else element
for element in page_data
]
logger.info(f"NUL characters were found in PR Reviews and replaced with spaces.")

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[pylint] reported by reviewdog 🐶
W1309: Using an f-string that does not have any interpolated variables (f-string-without-interpolation)

logger.info(f"NUL characters were found in PR Reviews and replaced with spaces.")
elif isinstance(page_data, bytes):
page_data = page_data.decode('utf-8').replace('\x00', ' ')
logger.info(f"NUL characters were found in PR Reviews and replaced with spaces.")

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[pylint] reported by reviewdog 🐶
W1309: Using an f-string that does not have any interpolated variables (f-string-without-interpolation)

Copy link
Contributor

@IsaacMilarky IsaacMilarky left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good for the most part.


self.oauths = GithubRandomKeyAuth(logger)
#self.oauths = GithubRandomKeyAuth(self, logger) # Removed and replaced for the issue below in frontend.py
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

?

@sgoggins sgoggins merged commit 99ee81f into main Jun 10, 2024
9 of 10 checks passed
@Ulincsys Ulincsys mentioned this pull request Jun 21, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
add-feature Adds new features admin Administrative/housekeeping/community tasks API Related to Augur's metrics API bug-fix Fixes a bug database Related to Augur's unifed data model dependencies Pull requests that update a dependency file deployed version Live problems with deployed versions devops Development Operations documentation Updates documentation frontend Related to the frontend installation Related the installation process metric Related to metrics, new or otherwise python Pull requests that update Python code release Related to releasing a new version of Augur security Related to keeping Augur secure server Related to the Augur server
Projects
None yet
4 participants