feat: allow filtering contributions and collaborations [IN-707]

[borfast]

This replaces the isContribution filter in the pipes that use activities, and adds the ability to filter by activity type "category", by including a new includeCodeContributions and includeCollaborations parameters.

By default, if neither parameter is passed to the pipe, it considers includeCodeContributions = 1.

The changes in services/libs/tinybird/pipes/top_member_org_copy.pipe were done so that the "Top contributors" and "Top organizations" widgets in the app front page use filtered data.

I'm not sure what to do for the "Top LF projects" widget, though. It uses the projects_list pipe from Tinybird, but that one does not touch activities, so we can't filter its data by activity type directly. The pipe gets its data from the insightsProjects_filtered pipe, which in turn gets its data from insights_projects_populated_ds data source, and that one is populated by a copy pipe, which does get activities data - but if we filter the data before it is copied, we affect every other pipe downstream from that one, which is no good.

---

Note

Replaces isContribution filters with activity type-based filtering via new activityTypes_filtered pipe and updates related pipes/widgets to support includeCodeContributions/includeCollaborations.

• Pipes: activity filtering overhaul
  • Replace isContribution checks with AND (type, platform) IN (SELECT activityType, platform FROM activityTypes_filtered) in:
    • activities_filtered.pipe, activities_filtered_historical_cutoff.pipe, activities_filtered_retention.pipe
    • health_score_active_contributors.pipe, health_score_contributor_dependency.pipe, health_score_organization_dependency.pipe
    • segmentId_aggregates_mv.pipe
  • Add params inherited from activityTypes_filtered: includeCodeContributions, includeCollaborations (and docs tweaks to repos).
• New pipe
  • activityTypes_filtered.pipe: parameterized filter over activityTypes with includeCodeContributions (default 1), includeCollaborations, includeOtherContributions; outputs (activityType, platform).
• Widgets data sources
  • top_member_org_copy.pipe: constrain member/org activity counts to code contributions or collaborations using activityTypes (exclude other activity types).

^{Written by Cursor Bugbot for commit 1b769fa. This will update automatically on new commits. Configure here.}

[joanagmaia]

@gaspergrom can you check @borfast comment from the PR?

I'm not sure what to do for the "Top LF projects" widget, though. It uses the projects_list pipe from Tinybird, but that one does not touch activities, so we can't filter its data by activity type directly. The pipe gets its data from the insightsProjects_filtered pipe, which in turn gets its data from insights_projects_populated_ds data source, and that one is populated by a copy pipe, which does get activities data - but if we filter the data before it is copied, we affect every other pipe downstream from that one, which is no good.

Raúl raised this on the backend weekly sync.

@epipav if you can also check this one out, I think it will help. This is a change that will affect almost all metrics we display on Insights.

[epipav]

Overall looks ok - I added comments to a few places where we changed the parameter name from onlyContributions to includeX style - In these places where we consume the tinybird API, client should also be changed. Have we already made this change somewhere?

Also added a readability nit to the pipe

[epipav]

Readability NIT: If we get the parameters to the query using CTEs, it can be shorter and easier to read

%
WITH
  {{ UInt8(includeCodeContributions, default=1) }} AS icc,
  {{ UInt8(includeCollaborations, default=0) }} AS icol,
  {{ UInt8(includeOtherContributions, default=0) }} AS ioc
SELECT activityType, platform
FROM activityTypes
WHERE
      (icc = 1  AND isCodeContribution = 1)
   OR (icol = 1 AND isCollaboration   = 1)
   OR (ioc = 1  AND isCodeContribution = 0 AND isCollaboration = 0)

[borfast]

Good call, I'll change it 👍

[epipav]

this'll break on the client part because client still uses onlyContributions ?

[epipav]

this'll break on the client part because client still uses onlyContributions ?

[epipav]

this'll break on the client part because client still uses onlyContributions ?

[borfast]

In these places where we consume the tinybird API, client should also be changed. Have we already made this change somewhere?

Yes, there'a a PR for Insights that covers this. As far as I know, there's no other client that uses onlyContributions.

[cursor]

Bug: Filter Inclusion Error

The filter for "other contributions" doesn't verify that activities are actual contributions. This means the pipe can return non-contribution activities, which differs from the original behavior of only including isContribution = 1 activities.