Production performance

[themarolt]

Changes proposed ✍️

What

🤖 Generated by Copilot at 3286eff

This pull request contains various changes to improve the performance, logging, debugging and data integrity of the backend and the data sink, integration data, integration run and integration stream workers. It reduces the database updates, removes unnecessary indexes, filters out deleted or irrelevant data, and adds conditions to avoid duplicate or redundant data. It also adds a new script to manually trigger the stream processed event for a run. It modifies several files, such as webhookProcessor.ts, activity.repo.ts, integrationDataService.ts, nodeWorkerSQS.ts, and the migration files.
​

🤖 Generated by Copilot at 3286eff

We are the masters of the data stream
We drop the indexes, we delete the data
We optimize the queries, we reduce the noise
We trigger the events, we process the data

Why

How

🤖 Generated by Copilot at 3286eff

• Drop and create indexes on activities, segments and conversations tables to improve performance and avoid duplicates (link, link, link, link, link, link)
• Exclude deleted activities from queries in activity.repo.ts and incomingWebhookRepository.ts to fix bugs and enforce data integrity (link, link, link)
• Exclude webhooks with type GITHUB or DISCORD from processing in incomingWebhookRepository.ts and webhookProcessor.ts to support different types of webhooks and avoid conflicts (link, link)
• Remove or comment out unnecessary touch and update calls in dataSink.service.ts, integrationDataService.ts, integrationRunService.ts and integrationStreamService.ts to reduce database updates and rely on stream processed event (link, link, link, link, link, link, link, link, link, link)
• Refactor update queries in member.repo.ts, member.service.ts and settings.repo.ts to exclude unchanged or empty columns and reduce query size (link, link, link, link)
• Pass resultId and sourceId to triggerResultProcessing method in integrationDataService.ts to pass to data sink worker for logging and debugging (link, link, link)
• Use transaction context in member.service.ts to ensure consistency and avoid errors (link, link)
• Delete processed data from database in integrationData.repo.ts to reduce table size (link)
• Add script to package.json in integration_run_worker to manually trigger stream processed event for debugging (link)
• Lower log level from info to debug in webhookProcessor.ts and nodeWorkerSQS.ts to reduce noise and verbosity (link, link)
• Remove unused argument from super constructor call in queue/index.ts to simplify queue settings (link)

Checklist ✅

• Label appropriately with Feature, Improvement, or Bug.
• Add screehshots to the PR description for relevant FE changes
• New backend functionality has been unit-tested.
• API documentation has been updated (if necessary) (see docs on API documentation).
• Quality standards are met.