Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

REDIS Patch + api docs update #2083

Merged
merged 3 commits into from
Dec 20, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 18 additions & 4 deletions augur/tasks/init/celery_app.py
Original file line number Diff line number Diff line change
Expand Up @@ -46,22 +46,36 @@

redis_db_number, redis_conn_string = get_redis_conn_values()

task_annotations = {'*': {'rate_limit': '5/s'}}

# initialize the celery app
BROKER_URL = f'{redis_conn_string}{redis_db_number}'
BACKEND_URL = f'{redis_conn_string}{redis_db_number+1}'

celery_app = Celery('tasks', broker=BROKER_URL, backend=BACKEND_URL, include=tasks,
CELERY_ANNOTATIONS=task_annotations, worker_pool_restarts=True)
celery_app = Celery('tasks', broker=BROKER_URL, backend=BACKEND_URL, include=tasks)

# define the queues that tasks will be put in (by default tasks are put in celery queue)
celery_app.conf.task_routes = {
'augur.tasks.git.facade_tasks.*': {'queue': 'cpu'}
}

#Setting to be able to see more detailed states of running tasks
celery_app.conf.task_track_started = True

#ignore task results by default
celery_app.conf.task_ignore_result = True

# store task erros even if the task result is ignored
celery_app.conf.task_store_errors_even_if_ignored = True

# set task default rate limit
celery_app.conf.task_default_rate_limit = '5/s'

# set tasks annotations for rate limiting specific tasks
celery_app.conf.task_annotations = None

# allow workers to be restarted remotely
celery_app.conf.worker_pool_restarts = True



def split_tasks_into_groups(augur_tasks: List[str]) -> Dict[str, List[str]]:
"""Split tasks on the celery app into groups.
Expand Down
88 changes: 88 additions & 0 deletions docs/source/rest-api/spec.yml
Original file line number Diff line number Diff line change
Expand Up @@ -5144,3 +5144,91 @@ paths:
type: object
tags:
- visualizations
/repo-groups/:repo_group_id/repo-messages:
get:
description: 'The number of messages exchanged for a repository group over a specified period.'
externalDocs:
description: CHAOSS Metric Definition
url:
operationId: Repository Messages (Repo Group)
parameters:
- description: Repository Group ID
in: path
name: repo_group_id
schema:
type: string
type: string
- description: 'Beginning date specification. E.g. values: 2018, 2018-05, 2019-05-01'
in: path
name: begin_date
schema:
type: string
type: string
- description: 'Ending date specification. E.g. values: 2018, 2018-05, 2019-05-01'
in: path
name: end_date
schema:
type: string
type: string
responses:
'200':
description: OK
schema:
items:
properties:
message_date:
description: 'Example: 2010-09-28T19:07:15.000Z'
type: string
count:
description: 'Example: 0 days 22:39:44.000000000'
type: string
repo_name:
description: 'Example: prototype-ujs'
type: string
type: array
tags:
- evolution
/repos/:repo_id/repo-messages:
get:
description: 'The number of messages exchanged for a repository over a specified period.'
externalDocs:
description: CHAOSS Metric Definition
url:
operationId: Repo Messages (Repo)
parameters:
- description: Repository ID.
in: path
name: repo_id
schema:
type: string
type: string
- description: 'Beginning date specification. E.g. values: 2018, 2018-05, 2019-05-01'
in: path
name: begin_date
schema:
type: string
type: string
- description: 'Ending date specification. E.g. values: 2018, 2018-05, 2019-05-01'
in: path
name: end_date
schema:
type: string
type: string
responses:
'200':
description: OK
schema:
items:
properties:
message_date:
description: 'Example: 2010-09-28T19:07:15.000Z'
type: string
count:
description: 'Example: 0 days 22:39:44.000000000'
type: string
repo_name:
description: 'Example: prototype-ujs'
type: string
type: array
tags:
- evolution
4 changes: 2 additions & 2 deletions metadata.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,8 @@

__short_description__ = "Python 3 package for free/libre and open-source software community metrics, models & data collection"

__version__ = "0.43.5"
__release__ = "v0.43.5 (Festivus!)"
__version__ = "0.43.6"
__release__ = "v0.43.6 (Festivus!)"

__license__ = "MIT"
__copyright__ = "University of Missouri, University of Nebraska-Omaha, CHAOSS, Brian Warner & Augurlabs 2023"