feat: activityRelations and PRs lambda architecture (IN-740) by epipav · Pull Request #3624 · linuxfoundation/crowd.dev

epipav · 2025-11-20T13:12:24Z

Note

Introduce snapshot-based Lambda Architecture for Tinybird: add MVs/copy pipes/datasources with snapshotId, build PR baseline-merge pipeline, and update queries to filter latest snapshots.

Docs/Guidance
- Add LAMBDA_ARCHITECTURE.md detailing the real-time enrichment, hourly merge, and serving layers with snapshot-based deduplication.
- Update README.md to reference the new architecture and clarify operational notes.
Datasources (Serving Layer)
- activityRelations_deduplicated_cleaned_ds: add snapshotId (partition key), TTL (6h), adjust partitioning to snapshotId.
- Add MV output datasource activityRelations_enrich_clean_snapshot_MV_ds with TTL (1d).
- pull_requests_analyzed: add snapshotId.
Enrichment (Real-time MVs)
- New MV activityRelations_enrich_clean_snapshot_MV enriching/validating rows and assigning hourly snapshotId.
- New PR analysis baseline-merge MVs: pull_request_analysis_MV_baseline_merge and pull_request_analysis_baseline_merge_MV producing merged PR snapshots.
- Convert segmentsAggregatedMV MV input to use MV output (activityRelations_enrich_clean_snapshot_MV_ds).
Merge Layer (Copy Pipes / Scheduling)
- Add activityRelations_snapshot_merger_copy (hourly :10, append) to merge realtime and previous snapshots.
- Add pull_request_analysis_snapshot_merger_copy (hourly :00, replace) to refresh PR snapshots.
- Add initial snapshot copy pipes: activityRelations_enrich_clean_initial_snapshot, pull_request_analysis_initial_snapshot, segmentId_aggregates_initial_snapshot (@on-demand).
- Adjust schedules where needed (e.g., contributions_with_local_time to 33 * * * *).
Query Layer updates
- Update numerous pipes (filters, health scores, leaderboards, org/project insights, issues/pr analysis, data copilot, maintainers/members, org dashboards) to filter serving datasets by latest snapshot: WHERE snapshotId = (SELECT max(snapshotId) FROM ...).
Misc
- Minor schema/tag cleanups and wording fixes across datasource/pipe files.

^{Written by Cursor Bugbot for commit 66232c6. This will update automatically on new commits. Configure here.}

github-actions · 2025-11-20T13:12:34Z

⚠️ Jira Issue Key Missing

Your PR title doesn't contain a Jira issue key. Consider adding it for better traceability.

Example:

feat: add user authentication (CM-123)
feat: add user authentication (IN-123)

Projects:

CM: Community Data Platform
IN: Insights

Please add a Jira issue key to your PR title.

github-actions

Conventional Commits FTW!

github-actions · 2025-11-20T13:12:40Z

⚠️ Jira Issue Key Missing

Your PR title doesn't contain a Jira issue key. Consider adding it for better traceability.

Example:

feat: add user authentication (CM-123)
feat: add user authentication (IN-123)

Projects:

CM: Community Data Platform
IN: Insights

Please add a Jira issue key to your PR title.

github-actions · 2025-11-20T13:13:16Z

⚠️ Jira Issue Key Missing

Your PR title doesn't contain a Jira issue key. Consider adding it for better traceability.

Example:

feat: add user authentication (CM-123)
feat: add user authentication (IN-123)

Projects:

CM: Community Data Platform
IN: Insights

Please add a Jira issue key to your PR title.

cursor

This PR is being reviewed by Cursor Bugbot

Details

Your team is on the Bugbot Free tier. On this plan, Bugbot will review limited PRs each billing cycle for each member of your team.

To receive Bugbot reviews on all of your PRs, visit the Cursor dashboard to activate Pro and start your 14-day free trial.

github-actions · 2025-11-20T13:15:39Z

⚠️ Jira Issue Key Missing

Your PR title doesn't contain a Jira issue key. Consider adding it for better traceability.

Example:

feat: add user authentication (CM-123)
feat: add user authentication (IN-123)

Projects:

CM: Community Data Platform
IN: Insights

Please add a Jira issue key to your PR title.

github-actions · 2025-11-20T13:16:12Z

⚠️ Jira Issue Key Missing

Your PR title doesn't contain a Jira issue key. Consider adding it for better traceability.

Example:

feat: add user authentication (CM-123)
feat: add user authentication (IN-123)

Projects:

CM: Community Data Platform
IN: Insights

Please add a Jira issue key to your PR title.

github-actions · 2025-11-20T13:16:47Z

⚠️ Jira Issue Key Missing

Your PR title doesn't contain a Jira issue key. Consider adding it for better traceability.

Example:

feat: add user authentication (CM-123)
feat: add user authentication (IN-123)

Projects:

CM: Community Data Platform
IN: Insights

Please add a Jira issue key to your PR title.

github-actions · 2025-11-20T13:17:13Z

⚠️ Jira Issue Key Missing

Your PR title doesn't contain a Jira issue key. Consider adding it for better traceability.

Example:

feat: add user authentication (CM-123)
feat: add user authentication (IN-123)

Projects:

CM: Community Data Platform
IN: Insights

Please add a Jira issue key to your PR title.

github-actions · 2025-11-21T08:18:24Z

⚠️ Jira Issue Key Missing

Your PR title doesn't contain a Jira issue key. Consider adding it for better traceability.

Example:

feat: add user authentication (CM-123)
feat: add user authentication (IN-123)

Projects:

CM: Community Data Platform
IN: Insights

Please add a Jira issue key to your PR title.

github-actions · 2025-11-21T08:18:43Z

⚠️ Jira Issue Key Missing

Your PR title doesn't contain a Jira issue key. Consider adding it for better traceability.

Example:

feat: add user authentication (CM-123)
feat: add user authentication (IN-123)

Projects:

CM: Community Data Platform
IN: Insights

Please add a Jira issue key to your PR title.

github-actions · 2025-11-24T09:47:03Z

⚠️ Jira Issue Key Missing

Your PR title doesn't contain a Jira issue key. Consider adding it for better traceability.

Example:

feat: add user authentication (CM-123)
feat: add user authentication (IN-123)

Projects:

CM: Community Data Platform
IN: Insights

Please add a Jira issue key to your PR title.

github-actions · 2025-11-24T09:47:35Z

⚠️ Jira Issue Key Missing

Your PR title doesn't contain a Jira issue key. Consider adding it for better traceability.

Example:

feat: add user authentication (CM-123)
feat: add user authentication (IN-123)

Projects:

CM: Community Data Platform
IN: Insights

Please add a Jira issue key to your PR title.

github-actions · 2025-11-24T09:48:04Z

⚠️ Jira Issue Key Missing

Your PR title doesn't contain a Jira issue key. Consider adding it for better traceability.

Example:

feat: add user authentication (CM-123)
feat: add user authentication (IN-123)

Projects:

CM: Community Data Platform
IN: Insights

Please add a Jira issue key to your PR title.

github-actions · 2025-11-24T09:49:29Z

⚠️ Jira Issue Key Missing

Your PR title doesn't contain a Jira issue key. Consider adding it for better traceability.

Example:

feat: add user authentication (CM-123)
feat: add user authentication (IN-123)

Projects:

CM: Community Data Platform
IN: Insights

Please add a Jira issue key to your PR title.

github-actions · 2025-11-24T09:50:14Z

⚠️ Jira Issue Key Missing

Your PR title doesn't contain a Jira issue key. Consider adding it for better traceability.

Example:

feat: add user authentication (CM-123)
feat: add user authentication (IN-123)

Projects:

CM: Community Data Platform
IN: Insights

Please add a Jira issue key to your PR title.

github-actions · 2025-11-24T09:50:29Z

⚠️ Jira Issue Key Missing

Your PR title doesn't contain a Jira issue key. Consider adding it for better traceability.

Example:

feat: add user authentication (CM-123)
feat: add user authentication (IN-123)

Projects:

CM: Community Data Platform
IN: Insights

Please add a Jira issue key to your PR title.

github-actions · 2025-11-24T09:52:24Z

⚠️ Jira Issue Key Missing

Your PR title doesn't contain a Jira issue key. Consider adding it for better traceability.

Example:

feat: add user authentication (CM-123)
feat: add user authentication (IN-123)

Projects:

CM: Community Data Platform
IN: Insights

Please add a Jira issue key to your PR title.

github-actions · 2025-11-24T12:45:53Z

⚠️ Jira Issue Key Missing

Your PR title doesn't contain a Jira issue key. Consider adding it for better traceability.

Example:

feat: add user authentication (CM-123)
feat: add user authentication (IN-123)

Projects:

CM: Community Data Platform
IN: Insights

Please add a Jira issue key to your PR title.

github-actions · 2025-11-24T12:46:14Z

⚠️ Jira Issue Key Missing

Your PR title doesn't contain a Jira issue key. Consider adding it for better traceability.

Example:

feat: add user authentication (CM-123)
feat: add user authentication (IN-123)

Projects:

CM: Community Data Platform
IN: Insights

Please add a Jira issue key to your PR title.

github-actions · 2025-11-24T12:57:27Z

⚠️ Jira Issue Key Missing

Your PR title doesn't contain a Jira issue key. Consider adding it for better traceability.

Example:

feat: add user authentication (CM-123)
feat: add user authentication (IN-123)

Projects:

CM: Community Data Platform
IN: Insights

Please add a Jira issue key to your PR title.

github-actions · 2025-11-24T12:57:54Z

⚠️ Jira Issue Key Missing

Your PR title doesn't contain a Jira issue key. Consider adding it for better traceability.

Example:

feat: add user authentication (CM-123)
feat: add user authentication (IN-123)

Projects:

CM: Community Data Platform
IN: Insights

Please add a Jira issue key to your PR title.

github-actions · 2025-11-24T13:01:42Z

⚠️ Jira Issue Key Missing

Your PR title doesn't contain a Jira issue key. Consider adding it for better traceability.

Example:

feat: add user authentication (CM-123)
feat: add user authentication (IN-123)

Projects:

CM: Community Data Platform
IN: Insights

Please add a Jira issue key to your PR title.

github-actions · 2025-11-24T13:02:28Z

⚠️ Jira Issue Key Missing

Your PR title doesn't contain a Jira issue key. Consider adding it for better traceability.

Example:

feat: add user authentication (CM-123)
feat: add user authentication (IN-123)

Projects:

CM: Community Data Platform
IN: Insights

Please add a Jira issue key to your PR title.

cursor · 2025-11-24T13:04:41Z

+            'pull_request-merged',
+            'merge_request-merged',
+            'changeset-merged'
+        )


Bug: Unused node in pipe definition

The node new_pull_request_related_activity is defined but never referenced by any other node in the pipe. The new_events_aggregated node queries directly from activityRelations_enrich_clean_snapshot_MV_ds instead of using this node. This unused node wastes resources and suggests incomplete refactoring or debugging code left behind.

Additional Locations (1)

services/libs/tinybird/pipes/pull_request_analysis_MV_baseline_merge.pipe#L5-L43

…boards

github-actions · 2025-11-25T10:38:44Z

⚠️ Jira Issue Key Missing

Your PR title doesn't contain a Jira issue key. Consider adding it for better traceability.

Example:

feat: add user authentication (CM-123)
feat: add user authentication (IN-123)

Projects:

CM: Community Data Platform
IN: Insights

Please add a Jira issue key to your PR title.

…bda-arch

github-actions · 2025-11-25T10:39:28Z

⚠️ Jira Issue Key Missing

Your PR title doesn't contain a Jira issue key. Consider adding it for better traceability.

Example:

feat: add user authentication (CM-123)
feat: add user authentication (IN-123)

Projects:

CM: Community Data Platform
IN: Insights

Please add a Jira issue key to your PR title.

github-actions · 2025-11-25T10:39:36Z

⚠️ Jira Issue Key Missing

Your PR title doesn't contain a Jira issue key. Consider adding it for better traceability.

Example:

feat: add user authentication (CM-123)
feat: add user authentication (IN-123)

Projects:

CM: Community Data Platform
IN: Insights

Please add a Jira issue key to your PR title.

github-actions · 2025-11-25T10:40:25Z

⚠️ Jira Issue Key Missing

Your PR title doesn't contain a Jira issue key. Consider adding it for better traceability.

Example:

feat: add user authentication (CM-123)
feat: add user authentication (IN-123)

Projects:

CM: Community Data Platform
IN: Insights

Please add a Jira issue key to your PR title.

cursor · 2025-11-25T10:41:56Z

        countDistinctState(memberId) AS contributorCount,
        countDistinctState(organizationId) AS organizationCount
-    FROM activityRelations_deduplicated_cleaned_ds
+    FROM activityRelations_enrich_clean_snapshot_MV_ds


Bug: Missing snapshotId filter in segment aggregates MV

The segmentId_aggregates_mv reads from activityRelations_enrich_clean_snapshot_MV_ds without filtering by snapshotId. Since the MV datasource contains multiple snapshots (partitioned by toYYYYMM(snapshotId) with 1-day TTL), aggregations will include all historical snapshots rather than the latest one, producing incorrect contributor and organization counts. This should filter by the latest snapshotId like other query pipes do.

…f github.com:CrowdDotDev/crowd.dev into feat/activityRelations-and-pull-requests-lambda-arch

github-actions · 2025-11-25T10:53:21Z

⚠️ Jira Issue Key Missing

Your PR title doesn't contain a Jira issue key. Consider adding it for better traceability.

Example:

feat: add user authentication (CM-123)
feat: add user authentication (IN-123)

Projects:

CM: Community Data Platform
IN: Insights

Please add a Jira issue key to your PR title.

github-actions · 2025-11-25T10:54:06Z

⚠️ Jira Issue Key Missing

Your PR title doesn't contain a Jira issue key. Consider adding it for better traceability.

Example:

feat: add user authentication (CM-123)
feat: add user authentication (IN-123)

Projects:

CM: Community Data Platform
IN: Insights

Please add a Jira issue key to your PR title.

…bda-arch

github-actions · 2025-11-25T12:10:03Z

⚠️ Jira Issue Key Missing

Your PR title doesn't contain a Jira issue key. Consider adding it for better traceability.

Example:

feat: add user authentication (CM-123)
feat: add user authentication (IN-123)

Projects:

CM: Community Data Platform
IN: Insights

Please add a Jira issue key to your PR title.

…bda-arch

epipav added 2 commits November 20, 2025 14:09

feat: lambda architecture for activityRelations and PRs

3df69c0

chore: formatting

0aa95dd

github-actions Bot reviewed Nov 20, 2025

View reviewed changes

cursor Bot reviewed Nov 20, 2025

View reviewed changes

Comment thread services/libs/tinybird/pipes/pull_request_analysis_initial_snapshot.pipe Outdated

Comment thread services/libs/tinybird/pipes/pull_request_analysis_initial_snapshot.pipe Outdated

Comment thread services/libs/tinybird/pipes/pull_request_analysis_MV.pipe Outdated

chore: cleaning and more formatting

0fc8194

chore: remove commented out sql piece

6665373

cursor Bot reviewed Nov 20, 2025

View reviewed changes

Comment thread services/libs/tinybird/pipes/pull_request_analysis_MV.pipe Outdated

Comment thread services/libs/tinybird/datasources/segmentsAggregatedMV.datasource Outdated

chore: documentation start

42c7914

cursor Bot reviewed Nov 21, 2025

View reviewed changes

Comment thread services/libs/tinybird/pipes/pull_request_analysis_MV.pipe Outdated

Comment thread services/libs/tinybird/pipes/activityRelations_snapshot_merger_copy.pipe

chore: improve docs

0fc58f1

epipav changed the title ~~feat: activityRelations and pull requests with ingestion time enrichment and lambda architecture~~ feat: activityRelations and pull requests lambda architecture Nov 24, 2025

chore: improve docs

b4a9d16

epipav changed the title ~~feat: activityRelations and pull requests lambda architecture~~ feat: activityRelations lambda architecture Nov 24, 2025

chore: formatting docs

b6347ff

cursor Bot reviewed Nov 24, 2025

View reviewed changes

Comment thread services/libs/tinybird/pipes/pull_request_analysis_snapshot_merger_copy.pipe

chore: remove obsolete MV definition

5ff555a

cursor Bot reviewed Nov 24, 2025

View reviewed changes

Comment thread services/libs/tinybird/pipes/org_dash_metric_copy_pipe.pipe

Comment thread services/libs/tinybird/pipes/pull_request_analysis_baseline_merge_MV.pipe

fix: sql precedence typo

be45a94

chore: documentation improvements

7be5943

cursor Bot reviewed Nov 24, 2025

View reviewed changes

themarolt approved these changes Nov 25, 2025

View reviewed changes

fix: second sorting for deterministic tiebreaks in contributor leader…

f0a19c2

…boards

Merge branch 'main' into feat/activityRelations-and-pull-requests-lam…

94a2e24

…bda-arch

cursor Bot reviewed Nov 25, 2025

View reviewed changes

epipav added 2 commits November 25, 2025 11:52

chore: main documentation updates and improvements

695fd92

Merge branch 'feat/activityRelations-and-pull-requests-lambda-arch' o…

66232c6

…f github.com:CrowdDotDev/crowd.dev into feat/activityRelations-and-pull-requests-lambda-arch

Merge branch 'main' into feat/activityRelations-and-pull-requests-lam…

85d4d15

…bda-arch

epipav changed the title ~~feat: activityRelations and PRs lambda architecture~~ feat: activityRelations and PRs lambda architecture (IN-740) Nov 25, 2025

Merge branch 'main' into feat/activityRelations-and-pull-requests-lam…

10f6a4b

…bda-arch

epipav merged commit 95bd360 into main Nov 25, 2025
14 checks passed

epipav deleted the feat/activityRelations-and-pull-requests-lambda-arch branch November 25, 2025 12:25

Conversation

epipav commented Nov 20, 2025 • edited by cursor Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Nov 20, 2025

Uh oh!

github-actions Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Nov 20, 2025

Uh oh!

github-actions Bot commented Nov 20, 2025

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

This PR is being reviewed by Cursor Bugbot

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Nov 20, 2025

Uh oh!

github-actions Bot commented Nov 20, 2025

Uh oh!

github-actions Bot commented Nov 20, 2025

Uh oh!

github-actions Bot commented Nov 20, 2025

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Nov 21, 2025

Uh oh!

github-actions Bot commented Nov 21, 2025

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Nov 24, 2025

Uh oh!

github-actions Bot commented Nov 24, 2025

Uh oh!

github-actions Bot commented Nov 24, 2025

Uh oh!

github-actions Bot commented Nov 24, 2025

Uh oh!

github-actions Bot commented Nov 24, 2025

Uh oh!

github-actions Bot commented Nov 24, 2025

Uh oh!

github-actions Bot commented Nov 24, 2025

Uh oh!

Uh oh!

github-actions Bot commented Nov 24, 2025

Uh oh!

github-actions Bot commented Nov 24, 2025

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Nov 24, 2025

Uh oh!

github-actions Bot commented Nov 24, 2025

Uh oh!

github-actions Bot commented Nov 24, 2025

Uh oh!

github-actions Bot commented Nov 24, 2025

Uh oh!

cursor Bot Nov 24, 2025

Choose a reason for hiding this comment

Bug: Unused node in pipe definition

Uh oh!

github-actions Bot commented Nov 25, 2025

Uh oh!

github-actions Bot commented Nov 25, 2025

Uh oh!

github-actions Bot commented Nov 25, 2025

Uh oh!

github-actions Bot commented Nov 25, 2025

Uh oh!

cursor Bot Nov 25, 2025

Choose a reason for hiding this comment

epipav commented Nov 20, 2025 •

edited by cursor Bot

Loading

github-actions Bot left a comment •

edited

Loading