LinkedIn bugfixes, small refactor of integration processor and individual stream processing script

[themarolt]

Changes proposed ✍️

• In some cases linkedin get member API endpoint returns 404 if member has private profile and we needed to handle that
• Organization URN ids can also be in format urn:li:organizationBrand so added this option to the processing
• Individual stream processing script for easier debugging of errored streams
• Small memory footprint optimization when processing db operations one by one - no need to keep records that were already processed in the memory

What

🤖 Generated by Copilot at 345852b

This pull request improves the integration processing logic for various platforms, such as LinkedIn, Twitter, and GitHub. It adds new classes and files to handle the periodic checks, ticks, and webhooks of integrations, and to send messages to the node worker queue. It also refactors some existing functions and files to handle different URN formats, batch database operations, and profile not found errors. It also adds a new script command and a trace log statement for convenience and debugging.
​

🤖 Generated by Copilot at 345852b

We're the crew of the node worker queue
We process the streams and the webhooks too
We check the integrations and run them with ease
We heave away, heave away, on the count of three

Why

How

🤖 Generated by Copilot at 345852b

• Add new script command to run process-stream.ts script with SERVICE environment variable (link)
• Replace for loops with while loops in operationsWorker.ts to avoid memory issues when processing large arrays of records (link, link, link, link, link)
• Add new classes to handle periodic checks, ticks, and processing of integrations in integrationCheckProcessor.ts, integrationTickProcessor.ts, and webhookProcessor.ts (link, link, link)
• Use nodeWorkerSQS module to send messages to node worker queue for integration check or processing logic (link, link, link)
• Handle 404 status from LinkedIn API in getMember function and return null (link)
• Handle 'urn:li:organizationBrand:' URNs in isLinkedInOrganization and getLinkedInOrganizationId functions (link, link)
• Add trace log statement to createWithMember function in activityService.ts (link)
• Add optional streamId parameter to NodeWorkerIntegrationProcessMessage class (link)

Checklist ✅

• Label appropriately with Feature, Improvement, or Bug.
• Add screehshots to the PR description for relevant FE changes
• New backend functionality has been unit-tested.
• API documentation has been updated (if necessary) (see docs on API documentation).
• Quality standards are met.