Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Updates to Pull Request, Facade, and Contributor Logic #1614

Merged
merged 133 commits into from
Feb 7, 2022
Merged

Updates to Pull Request, Facade, and Contributor Logic #1614

merged 133 commits into from
Feb 7, 2022

Conversation

sgoggins
Copy link
Member

@sgoggins sgoggins commented Feb 7, 2022

Also includes a modification of to one API endpoint.

ABrain7710 and others added 30 commits October 15, 2021 12:06
@ABrain7710
Set up basic structure for worker_persistance tests
2dcb319 
Signed-off-by: Andrew Brain <61482022+ABrain7710@users.noreply.github.com>
@IsaacMilarky
Merge branch 'docker_test_data' into test-worker-persistance
9f3e983
@IsaacMilarky
Up to date database container
3ef55eb 
Signed-off-by: Isaac M <krabs@tilde.team>
@IsaacMilarky
Docker fixture written out
4f0f41f 
Signed-off-by: Isaac M <krabs@tilde.team>
@IsaacMilarky
More fixtures for sample data
82feb50 
Signed-off-by: Isaac M <krabs@tilde.team>
@IsaacMilarky
filename change
32bd59b 
Signed-off-by: Isaac M <krabs@tilde.team>
@IsaacMilarky
More progress
e7d2be2 
Signed-off-by: Isaac M <krabs@tilde.team>
@IsaacMilarky
Kind of a hack but so long as it works
2c6d317 
Signed-off-by: Isaac M <krabs@tilde.team>
@IsaacMilarky
Oh it was a typo
ca3fac5 
Signed-off-by: Isaac M <krabs@tilde.team>
@IsaacMilarky
Remove import
9ec43c1 
Signed-off-by: Isaac M <krabs@tilde.team>
@IsaacMilarky
build string typo
fdfc589 
Signed-off-by: Isaac M <krabs@tilde.team>
@IsaacMilarky
python scripting
7599d79 
Signed-off-by: Isaac M <krabs@tilde.team>
@IsaacMilarky
Debug
446efc6 
Signed-off-by: Isaac M <krabs@tilde.team>
@IsaacMilarky
syntax
0aca00f 
Signed-off-by: Isaac M <krabs@tilde.team>
@IsaacMilarky
typo
1e5283f 
Signed-off-by: Isaac M <krabs@tilde.team>
@IsaacMilarky
change default port
f72efcd 
Signed-off-by: Isaac M <krabs@tilde.team>
@IsaacMilarky
change default port
ab983a0 
Signed-off-by: Isaac M <krabs@tilde.team>
@IsaacMilarky
syntax
24d5d99 
Signed-off-by: Isaac M <krabs@tilde.team>
@IsaacMilarky
no bash nonsenese
b6bfc14 
Signed-off-by: Isaac M <krabs@tilde.team>
@IsaacMilarky
five seconds
67bd4af 
Signed-off-by: Isaac M <krabs@tilde.team>
@IsaacMilarky
Updated paths
283af33 
Signed-off-by: Isaac M <krabs@tilde.team>
@IsaacMilarky
syntax
750ed16 
Signed-off-by: Isaac M <krabs@tilde.team>
@IsaacMilarky
test
5127368 
Signed-off-by: Isaac M <krabs@tilde.team>
@IsaacMilarky
No detach
7978f19 
Signed-off-by: Isaac M <krabs@tilde.team>
@IsaacMilarky
figuring out how to deploy database without too much overhead
b3cbcef 
Signed-off-by: Isaac M <krabs@tilde.team>
@IsaacMilarky
syntax
ecefe5c 
Signed-off-by: Isaac M <krabs@tilde.team>
@IsaacMilarky
Progress
8ed499c 
Signed-off-by: Isaac M <krabs@tilde.team>
@IsaacMilarky
Debugging
b045709 
Signed-off-by: Isaac M <krabs@tilde.team>
@IsaacMilarky
Debugging
59eacc2 
Signed-off-by: Isaac M <krabs@tilde.team>
@IsaacMilarky
Doing insertion with right fields this time
11a3eb2 
Signed-off-by: Isaac M <krabs@tilde.team>
IsaacMilarky and others added 21 commits February 1, 2022 14:39
@IsaacMilarky
Set worker type contributor interface
584b5dc 
Signed-off-by: Isaac Milarsky <krabs@tilde.team>
@IsaacMilarky
Interface using query_github_contributors
a2bc8c1 
Signed-off-by: Isaac Milarsky <krabs@tilde.team>
@IsaacMilarky
typo
40614b1 
Signed-off-by: Isaac Milarsky <krabs@tilde.team>
@sgoggins
Merge pull request #1602 from chaoss/worker_git_integration_refactor
a456e02 
Worker git integration refactor
@IsaacMilarky
Catch bad url passed to query_github_contributors
94be422 
Signed-off-by: Isaac Milarsky <krabs@tilde.team>
@IsaacMilarky
Try to fix index error in add_repo_groups
887472e 
Signed-off-by: Isaac Milarsky <krabs@tilde.team>
@IsaacMilarky
Handle case where there's a hanging empty row
6cd5f01 
Signed-off-by: Isaac Milarsky <krabs@tilde.team>
@sgoggins
custom housekeeper setup for vista.
7d78712
@sgoggins
Merge branch 'facade-test' into spg-patch-aaq
b86a7c8
@sgoggins
Resolving this error:
5113f88 
2022-02-04 03:03:53,131,131ms [PID: 957399] workers.pull_request_worker.50471 [INFO] No more pages to check, breaking from pagination.

2022-02-04 03:03:53,131,131ms [PID: 957399] workers.pull_request_worker.50471 [DEBUG] Pull Requests model. ERROR: organize_needed_data() got multiple values for argument 'in_memory'
Traceback (most recent call last):
  File "/bigdisk/augur-vmware/workers/pull_request_worker/pull_request_worker.py", line 723, in pull_requests_model
    pk_source_prs = self._get_pk_source_prs()
  File "/bigdisk/augur-vmware/workers/pull_request_worker/pull_request_worker.py", line 676, in _get_pk_source_prs
    source_prs = self.paginate_endpoint(
  File "/bigdisk/augur-vmware/workers/worker_git_integration.py", line 1274, in paginate_endpoint
    need_insertion, need_update = self.organize_needed_data(
TypeError: organize_needed_data() got multiple values for argument 'in_memory'
2022-02-04 03:03:53,160,160ms [PID: 957399] workers.pull_request_worker.50471 [INFO] Worker completed this task in -72.24467587471008 seconds.

2022-02-04 03:03:53,172,172ms [PID: 957399] workers.pull_request_worker.50471 [INFO] Recorded job completion for: {'worker_id': 'workers.pull_request_worker.50471', 'job_type': 'MAINTAIN', 'repo_id': 27044, 'job_model': 'pull_requests', 'github_url': 'https://github.com/vmware/clarity'}
@sgoggins
Trying to fix this error:
d14cd60 
2022-02-04 03:10:32,904,904ms [PID: 1000904] workers.pull_request_worker.50471 [INFO] No more pages to check, breaking from pagination.

2022-02-04 03:10:32,904,904ms [PID: 1000904] workers.pull_request_worker.50471 [DEBUG] Pull Requests model. ERROR: organize_needed_data() got an unexpected keyword argument 'in_memory'
Traceback (most recent call last):
  File "/bigdisk/augur-vmware/workers/pull_request_worker/pull_request_worker.py", line 723, in pull_requests_model
    pk_source_prs = self._get_pk_source_prs()
  File "/bigdisk/augur-vmware/workers/pull_request_worker/pull_request_worker.py", line 676, in _get_pk_source_prs
    source_prs = self.paginate_endpoint(
  File "/bigdisk/augur-vmware/workers/worker_git_integration.py", line 1274, in paginate_endpoint
    need_insertion, need_update = self.organize_needed_data(
TypeError: organize_needed_data() got an unexpected keyword argument 'in_memory'
2022-02-04 03:10:32,933,933ms [PID: 1000904] workers.pull_request_worker.50471 [INFO] Worker completed this task in -70.22280406951904 seconds.

2022-02-04 03:10:32,944,944ms [PID: 1000904] workers.pull_request_worker.50471 [INFO] Recorded job completion for: {'worker_id': 'workers.pull_request_worker.50471', 'job_type': 'MAINTAIN', 'repo_id': 27044, 'job_model': 'pull_requests', 'github_url': 'https://github.com/vmware/clarity'}
@sgoggins
Trying to fix this error:
6136cf8 
22-02-04 03:22:31,750,750ms [PID: 1239046] workers.pull_request_worker.50471 [INFO] Updated rate limit, you have: 4674 requests remaining.
2022-02-04 03:22:31,755,755ms [PID: 1239046] workers.pull_request_worker.50471 [INFO] Analyzation of page 28 of 28 complete

2022-02-04 03:22:31,755,755ms [PID: 1239046] workers.pull_request_worker.50471 [INFO] No more pages to check, breaking from pagination.

2022-02-04 03:22:31,755,755ms [PID: 1239046] workers.pull_request_worker.50471 [DEBUG] Pull Requests model. ERROR: organize_needed_data() takes from 3 to 4 positional arguments but 5 were given
Traceback (most recent call last):
  File "/bigdisk/augur-vmware/workers/pull_request_worker/pull_request_worker.py", line 723, in pull_requests_model
    pk_source_prs = self._get_pk_source_prs()
  File "/bigdisk/augur-vmware/workers/pull_request_worker/pull_request_worker.py", line 676, in _get_pk_source_prs
    source_prs = self.paginate_endpoint(
  File "/bigdisk/augur-vmware/workers/worker_git_integration.py", line 1274, in paginate_endpoint
    need_insertion, need_update = self.organize_needed_data(
TypeError: organize_needed_data() takes from 3 to 4 positional arguments but 5 were given
2022-02-04 03:22:31,785,785ms [PID: 1239046] workers.pull_request_worker.50471 [INFO] Worker completed this task in -71.99200201034546 seconds.
@sgoggins
organize needed data does not take an in_memory parameter
b6d35b6 
2022-02-04 03:30:39,971,971ms [PID: 1278984] workers.pull_request_worker.50471 [DEBUG] Pull Requests model. ERROR: organize_needed_data() takes from 3 to 4 positional arguments but 5 were given
Traceback (most recent call last):
  File "/bigdisk/augur-vmware/workers/pull_request_worker/pull_request_worker.py", line 723, in pull_requests_model
    pk_source_prs = self._get_pk_source_prs()
  File "/bigdisk/augur-vmware/workers/pull_request_worker/pull_request_worker.py", line 676, in _get_pk_source_prs
    source_prs = self.paginate_endpoint(
  File "/bigdisk/augur-vmware/workers/worker_git_integration.py", line 1274, in paginate_endpoint
    need_insertion, need_update = self.organize_needed_data(
TypeError: organize_needed_data() takes from 3 to 4 positional arguments but 5 were given
2022-02-04 03:30:40,000,0ms [PID: 1278984] workers.pull_request_worker.50471 [INFO] Worker completed this task in -70.00859665870667 seconds.

2022-02-04 03:30:40,019,19ms [PID: 1278984] workers.pull_request_worker.50471 [INFO] Recorded job completion for: {'worker_id': 'workers.pull_request_worker.50471', 'job_type': 'MAINTAIN', 'repo_id': 27044, 'job_model': 'pull_requests', 'github_url': 'https://github.com/vmware/clarity'}
@sgoggins
Trying to still fix:
dd9376e 
2022-02-04 03:30:39,971,971ms [PID: 1278984] workers.pull_request_worker.50471 [DEBUG] Pull Requests model. ERROR: organize_needed_data() takes from 3 to 4 positional arguments but 5 were given
Traceback (most recent call last):
  File "/bigdisk/augur-vmware/workers/pull_request_worker/pull_request_worker.py", line 723, in pull_requests_model
    pk_source_prs = self._get_pk_source_prs()
  File "/bigdisk/augur-vmware/workers/pull_request_worker/pull_request_worker.py", line 676, in _get_pk_source_prs
    source_prs = self.paginate_endpoint(
  File "/bigdisk/augur-vmware/workers/worker_git_integration.py", line 1274, in paginate_endpoint
    need_insertion, need_update = self.organize_needed_data(
TypeError: organize_needed_data() takes from 3 to 4 positional arguments but 5 were given
2022-02-04 03:30:40,000,0ms [PID: 1278984] workers.pull_request_worker.50471 [INFO] Worker completed this task in -70.00859665870667 seconds.

2022-02-04 03:30:40,019,19ms [PID: 1278984] workers.pull_request_worker.50471 [INFO] Recorded job completion for: {'worker_id': 'workers.pull_request_worker.50471', 'job_type': 'MAINTAIN', 'repo_id': 27044, 'job_model': 'pull_requests', 'github_url': 'https://github.com/vmware/clarity'}
@sgoggins
Still trying to fix:
1660931 
2022-02-04 03:30:39,971,971ms [PID: 1278984] workers.pull_request_worker.50471 [DEBUG] Pull Requests model. ERROR: organize_needed_data() takes from 3 to 4 positional arguments but 5 were given
Traceback (most recent call last):
  File "/bigdisk/augur-vmware/workers/pull_request_worker/pull_request_worker.py", line 723, in pull_requests_model
    pk_source_prs = self._get_pk_source_prs()
  File "/bigdisk/augur-vmware/workers/pull_request_worker/pull_request_worker.py", line 676, in _get_pk_source_prs
    source_prs = self.paginate_endpoint(
  File "/bigdisk/augur-vmware/workers/worker_git_integration.py", line 1274, in paginate_endpoint
    need_insertion, need_update = self.organize_needed_data(
TypeError: organize_needed_data() takes from 3 to 4 positional arguments but 5 were given
2022-02-04 03:30:40,000,0ms [PID: 1278984] workers.pull_request_worker.50471 [INFO] Worker completed this task in -70.00859665870667 seconds.

2022-02-04 03:30:40,019,19ms [PID: 1278984] workers.pull_request_worker.50471 [INFO] Recorded job completion for: {'worker_id': 'workers.pull_request_worker.50471', 'job_type': 'MAINTAIN', 'repo_id': 27044, 'job_model': 'pull_requests', 'github_url': 'https://github.com/vmware/clarity'}
@sgoggins
Perhaps the missing self in paginate_endpoint is causing:
57e08e8 
2022-02-04 03:50:11,016,16ms [PID: 1542143] workers.pull_request_worker.50471 [DEBUG] Pull Requests model. ERROR: organize_needed_data() takes from 3 to 4 positional arguments but 5 were given
Traceback (most recent call last):
  File "/bigdisk/augur-vmware/workers/pull_request_worker/pull_request_worker.py", line 723, in pull_requests_model
    pk_source_prs = self._get_pk_source_prs()
  File "/bigdisk/augur-vmware/workers/pull_request_worker/pull_request_worker.py", line 676, in _get_pk_source_prs
    source_prs = self.paginate_endpoint(
  File "/bigdisk/augur-vmware/workers/worker_git_integration.py", line 1274, in paginate_endpoint
    need_insertion, need_update = self.organize_needed_data(
TypeError: organize_needed_data() takes from 3 to 4 positional arguments but 5 were given
2022-02-04 03:50:11,046,46ms [PID: 1542143] workers.pull_request_worker.50471 [INFO] Worker completed this task in -64.56478452682495 seconds.

I'm running out of logical guesses about what broke here.
@sgoggins
Revert "Perhaps the missing self in paginate_endpoint is causing:"
83ae33d 
This reverts commit 57e08e8.
@sgoggins
Revert "Trying to still fix:"
7c1fc15 
This reverts commit dd9376e.

I think this commit missed teh importance of table_pkey: #1601
@sgoggins
Merge branch 'spg-patch-aaq' into facade-test
7b85651
@sgoggins
Fixing, maybe:
e2f1aea 
2022-02-03 21:56:37,984,984ms [PID: 492008] workers.pull_request_worker.9099 [INFO] No more pages to check, breaking from pagination.

2022-02-03 21:56:37,985,985ms [PID: 492008] workers.pull_request_worker.9099 [DEBUG] Pull Requests model. ERROR: name 'in_memory' is not defined
Traceback (most recent call last):
  File "/home/sean/github/opsfirst/workers/pull_request_worker/pull_request_worker.py", line 723, in pull_requests_model
    pk_source_prs = self._get_pk_source_prs()
  File "/home/sean/github/opsfirst/workers/pull_request_worker/pull_request_worker.py", line 676, in _get_pk_source_prs
    source_prs = self.paginate_endpoint(
  File "/home/sean/github/opsfirst/workers/worker_git_integration.py", line 1274, in paginate_endpoint
    need_insertion, need_update = self.organize_needed_data(
  File "/home/sean/github/opsfirst/workers/worker_persistance.py", line 409, in organize_needed_data
    if not in_memory:
NameError: name 'in_memory' is not defined
2022-02-03 21:56:37,986,986ms [PID: 492008] workers.pull_request_worker.9099 [INFO] Worker completed this task in -1.6014952659606934 seconds.
@sgoggins
API Revision
090fa57
@sgoggins sgoggins added bug-fix Fixes a bug server Related to the Augur server triage Needs to be triaged labels Feb 7, 2022
@sgoggins sgoggins merged commit 59c5fca into main Feb 7, 2022
sgoggins
sgoggins commented Feb 22, 2022
View reviewed changes
Makefile
@@ -91,8 +91,10 @@ test: test-application test-metric-routes test-workers
test-application:
@ bash -c 'tox -e py-application'

#Worker's tests need a database from docker
#To use the docker daemon you need to be root so sudo is needed.
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good documentation.

@sgoggins sgoggins deleted the facade-test branch March 2, 2022 16:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@IsaacMilarky IsaacMilarky Awaiting requested review from IsaacMilarky

@ABrain7710 ABrain7710 Awaiting requested review from ABrain7710

Assignees
No one assigned
Labels
bug-fix Fixes a bug server Related to the Augur server triage Needs to be triaged
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@sgoggins @ABrain7710 @IsaacMilarky