Skip to content

feat: add daily activities counter#3445

Merged
ulemons merged 2 commits into
mainfrom
feat/activities-daily-count
Oct 5, 2025
Merged

feat: add daily activities counter#3445
ulemons merged 2 commits into
mainfrom
feat/activities-daily-count

Conversation

@ulemons
Copy link
Copy Markdown
Contributor

@ulemons ulemons commented Oct 2, 2025

PR: Add activities_daily_counts.pipe and optimize with PREWHERE

What

  • Add a Tinybird pipe to compute daily per-segment unique activity counts required by the cache-worker as part of deprecating QuestDB.
  • Optimize the query by using PREWHERE on segmentId, timestamp (and platform when present) so the engine reads only filter columns first, discards non-matching rows early, and then loads activityId only for survivors.

Why

  • The cache-worker needs a canonical time series of daily counts per segment to replace the equivalent queries previously served by QuestDB.
  • Using PREWHERE reduces scanned bytes, decompression, and memory pressure during uniq* aggregation—especially when filters are selective (not “high cardinality”, but high selectivity).

Checklist ✅

  • Label appropriately with Feature, Improvement, or Bug.

@ulemons ulemons self-assigned this Oct 2, 2025
@ulemons ulemons requested review from borfast and epipav as code owners October 2, 2025 09:02
@ulemons ulemons added the Feature Created by Linear-GitHub Sync label Oct 2, 2025
Copy link
Copy Markdown
Contributor

@borfast borfast left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TIL about PREWHERE 😁
👍

@ulemons ulemons merged commit 4f96a8f into main Oct 5, 2025
15 of 16 checks passed
@ulemons ulemons deleted the feat/activities-daily-count branch October 5, 2025 09:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Feature Created by Linear-GitHub Sync

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants