Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add automatic retries to data-sink-worker #1842

Merged
merged 2 commits into from
Nov 14, 2023

Conversation

garrrikkotua
Copy link
Contributor

@garrrikkotua garrrikkotua commented Nov 13, 2023

Changes proposed ✍️

What

🤖 Generated by Copilot at 1ea4963

This pull request adds a feature to the data sink worker app and its related modules to retry and delay results that fail or take too long to process. It uses the DataSinkWorkerEmitter class from the @crowd/sqs library to communicate with the data sink worker queue and the DataSinkRepository class to update the result state and timestamps. It also adds new columns, types, and messages to support this feature.

🤖 Generated by Copilot at 1ea4963

Oh, we are the data sink workers, and we process the results
We use the DataSinkWorkerEmitter to send and receive the messages
But when a result is failing or slow, we don't give up so quick
We retry and delay it with the IResultData trick

Why

How

🤖 Generated by Copilot at 1ea4963

  • Add retries and delays for failed or slow results (link, link, link, link, link, link, link, link, link)
  • Add two new columns retries and delayedUntil to the integration.results table in V1699626027__add-retries-to-results.sql (link)
  • Add the retries and delayedUntil properties to the IResultData interface in dataSink.data.ts (link)
  • Set the delayedUntil column to null when marking a result as success in dataSink.repo.ts (link)
  • Add methods delayResult and getDelayedResults to the DataSinkRepository class in dataSink.repo.ts to update and select delayed results (link)
  • Add logic to check and increment the retries value and calculate the delayedUntil timestamp in the triggerResultError method of the DataSinkService class in dataSink.service.ts (link)
  • Add a new value DELAYED to the IntegrationResultState enum in integrations.ts (link)
  • Add a new value CHECK_RESULTS to the DataSinkWorkerQueueMessageType enum and a new class CheckResultsQueueMessage in index.ts (link, link)
  • Add communication with the data sink worker queue using the DataSinkWorkerEmitter class from the @crowd/sqs library (link, link, link, link, link, link, link, link, link, link, link, link, link, link, link, link, link, link, link, link, link, link, link, link, link)
    • Import the DataSinkWorkerEmitter class in integrationTickProcessor.ts, serviceSQS.ts, processOldResults.ts, main.ts, index.ts, dataSink.service.ts, and dataSinkWorker.ts (link, link, link, link, link, link, link)
    • Add a function getDataSinkWorkerEmitter in serviceSQS.ts to get an instance of the DataSinkWorkerEmitter class (link)
    • Add a property dataSinkWorkerEmitter to the IntegrationTickProcessor and WorkerQueueReceiver classes in integrationTickProcessor.ts and index.ts (link, link)
    • Assign the dataSinkWorkerEmitter property in the constructors of the IntegrationTickProcessor and WorkerQueueReceiver classes using the getDataSinkWorkerEmitter function or the parameter (link, link)
    • Pass the dataSinkWorkerEmitter parameter or constant to the DataSinkService constructor in integrationTickProcessor.ts, processOldResults.ts, and index.ts (link, link, link)
    • Pass the dataSinkWorkerEmitter constant to the processOldResultsJob function in main.ts (link)
    • Create a constant dataWorkerEmitter that holds an instance of the DataSinkWorkerEmitter class in main.ts (link)
    • Add the dataWorkerEmitter constant to the list of parameters passed to the WorkerQueueReceiver constructor in main.ts (link)
    • Call the init method of the dataWorkerEmitter constant in main.ts (link)
    • Add a method checkResults to the DataSinkWorkerEmitter class in dataSinkWorker.ts to send a message to the data sink worker queue to check for delayed results (link)
    • Call the checkResults method of the dataSinkWorkerEmitter property in integrationTickProcessor.ts (link)
    • Add a case for the CHECK_RESULTS message type in the handleQueueMessage method of the WorkerQueueReceiver class in index.ts to call the checkResults method of the DataSinkService instance (link)
    • Import the CheckResultsQueueMessage class in dataSinkWorker.ts (link)
    • Import the IResultData interface, the addSeconds function, and the WORKER_SETTINGS function in dataSink.service.ts (link)
    • Add a property dataSinkWorkerEmitter to the DataSinkService class in dataSink.service.ts (link)
    • Change the type of the resultId parameter and argument of the triggerResultError method of the DataSinkService class in dataSink.service.ts and index.ts to IResultData (link, link)
  • Import the PlatformType enum from the @crowd/types library in dataSink.repo.ts to identify the platform of a result (link)

Checklist ✅

  • Label appropriately with Feature, Improvement, or Bug.
  • Add screenshots to the PR description for relevant FE changes
  • New backend functionality has been unit-tested.
  • API documentation has been updated (if necessary) (see docs on API documentation).
  • Quality standards are met.

@garrrikkotua garrrikkotua merged commit ce64e09 into main Nov 14, 2023
8 checks passed
@garrrikkotua garrrikkotua deleted the feature/automatic-results-retries branch November 14, 2023 10:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants