Maintain Tasks in Postgres

[snyaggarwal]

Context:
Redis is a broker for Celery. The states are maintained in Redis and OCL API queries Flower for to get tasks running/pending/successful/failed. This is currently maintained for 72 hours or next deployment.

Tasks:

• Postgres to be able to maintain the states of tasks. Redis + Celery stays as is.
• Handle all task events, start/success/failed/retry/etc
• API to get all tasks for a user
• Remove Flower dependency

[snyaggarwal]

@rkorytkowski @paynejd this is deployed on Dev. Please test pepfar scripts and any other manual testing on dev this week.

[paynejd]

Mismatch between summaries -- see summary in this screenshot (queried directly from the API):

Compare with result.detailed_summary in the verbose response, where it says it processed 261 out of 261:

Started: 2024-03-13 12:25:37.750349 | Processed: 261/261 | Created: 261 | Updated: 0 | Deleted: 0 | Existing: 0 | Permission Denied: 0 | Time: 78.23880672454834secs

result.report agrees with result.detailed_summary and shows 261 total resources instead of 295 or 278:

{'total': 261, 'processed': 261, 'created': 261, 'updated': 0, 'invalid': 0, 'exists': 0, 'failed': 0, 'exception': 0, 'deleted': 0, 'others': 0, 'unknown': 0, 'permission_denied': 0, 'elapsed_seconds': 78.23880672454834, 'start_time': '2024-03-13 12:25:37.750349', 'child_resource_time_distribution': {'concept': 15.847328662872314, 'mapping': 10.734777688980103, 'reference': 15.731571435928345}}

Also worth noting that the start times are different -- I'm assuming this is not a bug, but it's also probably not ideal behavior

[paynejd]

In the response in verbose mode (for a completed task), result should not be serialized

After unserializing result (e.g. json.loads), result.report is still serialized (e.g. it got serialized twice) -- this should also be unserialized

In result.report, the values in child_resource_time_distribution do not add up to the total elapsed time, which was 78 seconds -- why is there such a big difference? There are sources, collections, and versions in the import json -- should these be here as well, or are those not sent to parallel child processes?

'child_resource_time_distribution': {'concept': 15.847328662872314, 'mapping': 10.734777688980103, 'reference': 15.731571435928345}}

[paynejd]

Task is displayed as "PENDING" even though it is already starting to be processed:

[paynejd]

I ran a second import, and there is a big mismatch in the counts:

And there is a big difference between the numbers shown in this image and in the result.report field in the response:

{'total': 539, 'processed': 539, 'created': 539, 'updated': 0, 'invalid': 0, 'exists': 0, 'failed': 0, 'exception': 0, 'deleted': 0, 'others': 0, 'unknown': 0, 'permission_denied': 0, 'elapsed_seconds': 129.78452610969543, 'start_time': '2024-03-13 13:27:32.889688', 'child_resource_time_distribution': {'concept': 10.199201822280884, 'mapping': 30.620508670806885, 'reference': 35.50830268859863}}

[paynejd]

Finally, code to grab the task ID from a newly queued import no longer works in ocldev.oclfleximporter.OclBulkImporter.post(). Here's the code that is used to grab the task ID:

    bulk_import_response = ocldev.oclfleximporter.OclBulkImporter.post(
        input_list=resource_list, api_token=OCL_API_TOKEN,
        api_url_root=OCL_API_URL_ROOT, parallel=True)
    task_id = bulk_import_response.json()['task']

Is it possible to still support this without changing much under the hood? This will impact the PEPFAR integration scripts.

[paynejd]

This is fixed:

• No longer experiencing an issue retrieving the task ID when submitting a new bulk import

Three issues:

• response.result is still serialized -- I think we would prefer for this data to be returned simply as JSON (rather than serialized)?
• Import status shown as PENDING even though processing is already underway (this happened with both of my round 2 imports):

• Mismatch between counts, the total number processed, etc. -- in my first (smaller) import, this worked perfectly; in my second (larger) import, the counts did not match up (see only 540 processed in the result.report and 575 in the parent task, out of a total of 654 resources in the import script):

\"detailed_summary\": \"Started: 2024-03-15 15:56:09.509480 | Processed: 540/540 | Created: 539 | Updated: 0 | Deleted: 1 | Existing: 0 | Permission Denied: 0 | Time: 138.3544499874115secs\", \"report\": {\"total\": 540, \"processed\": 540, \"created\": 539, \"updated\": 0, \"invalid\": 0, \"exists\": 0, \"failed\": 0, \"exception\": 0, \"deleted\": 1, \"others\": 0, \"unknown\": 0, \"permission_denied\": 0, \"elapsed_seconds\": 138.3544499874115, \"start_time\": \"2024-03-15 15:56:09.509480\", \"child_resource_time_distribution\": {\"concept\": 10.30651330947876, \"mapping\": 20.206477403640747, \"reference\": 30.55776309967041}

[snyaggarwal]

@paynejd Serialization is fixed and its deployed on QA and Staging

[paynejd]

I tested serialization on dev and it's working well! Anything else to do on this ticket before we close it out?

[snyaggarwal]

@paynejd Lets keep this ticket until its clear of staging.

[paynejd]

The queue-specific query returns zero results:

GET https://api.staging.openconceptlab.org/importers/bulk-import/DATIM-MOH-NGA-DAA-FY22/

This means that the integration script is not able to detect if an import is already underway, so it is possible to have 2 conflicting IMAP imports overwriting each other or to start to build an IMAP export using incomplete content.

[snyaggarwal]

@paynejd This is fixed on Staging now.

[paynejd]

Round 2 of tests in staging worked flawlessly!

Only remaining step: Test out the subscription module in staging.

[snyaggarwal]

@rkorytkowski @paynejd Added a job to expire (remove) tasks from postgres when its older than a week.