Skip to content

feat: mentions handling on tinybird#3614

Merged
gaspergrom merged 5 commits intomainfrom
feat/tb-mentions
Nov 19, 2025
Merged

feat: mentions handling on tinybird#3614
gaspergrom merged 5 commits intomainfrom
feat/tb-mentions

Conversation

@gaspergrom
Copy link
Copy Markdown
Contributor

@gaspergrom gaspergrom commented Nov 18, 2025

Note

Introduces a new mentions datasource, integrates aggregated community fields into insights projects, adds a mentions listing pipe, and standardizes Kafka sink configs.

  • Data Sources
    • mentions: New datasource capturing community mentions with sentiment, relevance, and metadata.
  • Insights Projects Enrichment
    • Schema: Extend insights_projects_populated_ds with communityPlatforms, communityKeywords, communityLanguages.
    • Copy Pipe (insights_projects_populated_copy.pipe): Aggregate mentions per projectSlug and LEFT JOIN into results.
    • Filtered Pipe (insightsProjects_filtered.pipe): Select and return new community fields.
  • Pipes
    • mentions_list.pipe: List mentions with filters for projectSlug, platforms, keywords, sentiments, languages, with pagination.
  • Sinks
    • health_score_sink.pipe, insights_projects_populated_sink.pipe: Normalize TYPE SINK; add EXPORT_FORMAT csv and EXPORT_STRATEGY @new; keep Kafka topic and schedule.

Written by Cursor Bugbot for commit ed79abe. This will update automatically on new commits. Configure here.

Signed-off-by: Gašper Grom <gasper.grom@gmail.com>
Signed-off-by: Gašper Grom <gasper.grom@gmail.com>
Signed-off-by: Gašper Grom <gasper.grom@gmail.com>
@gaspergrom gaspergrom requested a review from epipav November 18, 2025 17:55
@gaspergrom gaspergrom self-assigned this Nov 18, 2025
@github-actions
Copy link
Copy Markdown
Contributor

⚠️ Jira Issue Key Missing

Your PR title doesn't contain a Jira issue key. Consider adding it for better traceability.

Example:

  • feat: add user authentication (CM-123)
  • feat: add user authentication (IN-123)

Projects:

  • CM: Community Data Platform
  • IN: Insights

Please add a Jira issue key to your PR title.

2 similar comments
@github-actions
Copy link
Copy Markdown
Contributor

⚠️ Jira Issue Key Missing

Your PR title doesn't contain a Jira issue key. Consider adding it for better traceability.

Example:

  • feat: add user authentication (CM-123)
  • feat: add user authentication (IN-123)

Projects:

  • CM: Community Data Platform
  • IN: Insights

Please add a Jira issue key to your PR title.

@github-actions
Copy link
Copy Markdown
Contributor

⚠️ Jira Issue Key Missing

Your PR title doesn't contain a Jira issue key. Consider adding it for better traceability.

Example:

  • feat: add user authentication (CM-123)
  • feat: add user authentication (IN-123)

Projects:

  • CM: Community Data Platform
  • IN: Insights

Please add a Jira issue key to your PR title.

Copy link
Copy Markdown

@cursor cursor Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the final PR Bugbot will review for you during this billing cycle

Your free Bugbot reviews will reset on December 12

Details

Your team is on the Bugbot Free tier. On this plan, Bugbot will review limited PRs each billing cycle for each member of your team.

To receive Bugbot reviews on all of your PRs, visit the Cursor dashboard to activate Pro and start your 14-day free trial.

Comment thread services/libs/tinybird/pipes/insights_projects_populated_copy.pipe Outdated
Signed-off-by: Gašper Grom <gasper.grom@gmail.com>
@github-actions
Copy link
Copy Markdown
Contributor

⚠️ Jira Issue Key Missing

Your PR title doesn't contain a Jira issue key. Consider adding it for better traceability.

Example:

  • feat: add user authentication (CM-123)
  • feat: add user authentication (IN-123)

Projects:

  • CM: Community Data Platform
  • IN: Insights

Please add a Jira issue key to your PR title.

1 similar comment
@github-actions
Copy link
Copy Markdown
Contributor

⚠️ Jira Issue Key Missing

Your PR title doesn't contain a Jira issue key. Consider adding it for better traceability.

Example:

  • feat: add user authentication (CM-123)
  • feat: add user authentication (IN-123)

Projects:

  • CM: Community Data Platform
  • IN: Insights

Please add a Jira issue key to your PR title.

`language` String `json:$.language` DEFAULT '',
`projectSlug` LowCardinality(String) `json:$.projectSlug` DEFAULT '',
`createdAt` DateTime64(3) `json:$.createdAt` DEFAULT now64(3),
`bookmarked` UInt8 `json:$.bookmarked`,
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bug: Missing DEFAULT value for bookmarked field

The bookmarked field is defined as UInt8 without a DEFAULT value. All other UInt8 fields in the codebase consistently have DEFAULT 0 or DEFAULT 1. Without a default, the field becomes required in incoming JSON, which will cause insertion failures when the field is missing from webhook payloads.

Fix in Cursor Fix in Web

`projectSlug` LowCardinality(String) `json:$.projectSlug` DEFAULT '',
`createdAt` DateTime64(3) `json:$.createdAt` DEFAULT now64(3),
`bookmarked` UInt8 `json:$.bookmarked`,
`keywords` Array(String) `json:$.keywords[:]`
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bug: Missing DEFAULT value for keywords array

The keywords field is defined as Array(String) without a DEFAULT value. All other Array(String) fields in the codebase consistently have DEFAULT []. Without a default, the field becomes required in incoming JSON, which will cause insertion failures when the field is missing from webhook payloads.

Fix in Cursor Fix in Web

Copy link
Copy Markdown
Collaborator

@epipav epipav left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added two comments

Comment thread services/libs/tinybird/pipes/insights_projects_populated_copy.pipe Outdated
Comment thread services/libs/tinybird/datasources/mentions.datasource Outdated
Signed-off-by: Gašper Grom <gasper.grom@gmail.com>
@github-actions
Copy link
Copy Markdown
Contributor

⚠️ Jira Issue Key Missing

Your PR title doesn't contain a Jira issue key. Consider adding it for better traceability.

Example:

  • feat: add user authentication (CM-123)
  • feat: add user authentication (IN-123)

Projects:

  • CM: Community Data Platform
  • IN: Insights

Please add a Jira issue key to your PR title.

@gaspergrom gaspergrom merged commit ffe96c3 into main Nov 19, 2025
14 checks passed
@gaspergrom gaspergrom deleted the feat/tb-mentions branch November 19, 2025 13:17
@github-actions
Copy link
Copy Markdown
Contributor

⚠️ Jira Issue Key Missing

Your PR title doesn't contain a Jira issue key. Consider adding it for better traceability.

Example:

  • feat: add user authentication (CM-123)
  • feat: add user authentication (IN-123)

Projects:

  • CM: Community Data Platform
  • IN: Insights

Please add a Jira issue key to your PR title.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants