Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve cubejs #1391

Merged
merged 17 commits into from
Aug 29, 2023
Merged

Improve cubejs #1391

merged 17 commits into from
Aug 29, 2023

Conversation

sausage-todd
Copy link
Contributor

@sausage-todd sausage-todd commented Aug 29, 2023

Changes proposed ✍️

  • Instead of trying to make perfect pre-aggregations, we just create materialized views for "members", "activities" and "organizations"
  • These materalized views have only the necessary information, which is why they're fast
  • The materialized views are refreshed periodically
  • Cubejs data model uses these materialized views to execute queries without pre-aggregations

What

🤖 Generated by Copilot at 4455d7c

This pull request refactors and simplifies the Cube.js schema files to use materialized views and table names instead of raw SQL queries, and to remove unnecessary or redundant fields. This improves the performance, readability, and maintainability of the Cube.js application. It also adds a SQL script to create and index the materialized views, and modifies the CrowdJob interface and the logging utility to use a custom logger.

🤖 Generated by Copilot at 4455d7c

The cube schemas were quite a mess
With raw queries and excess
They used materialized views
And cleaned up the cues
Now the performance is much better, I guess

Why

How

🤖 Generated by Copilot at 4455d7c

  • Use materialized views instead of raw SQL queries for Activities, Members, Organizations, and Segments cube schemas to improve performance and scalability (link, link, link, link, link)
  • Simplify measures and dimensions of Activities, Conversations, Members, and Sentiment cube schemas by removing unnecessary or redundant fields and using existing fields from materialized views or SQL queries (link, link, link, link, link)
  • Replace backticks with single quotes for string literals in Conversations, MemberOrganizations, and OrganizationSegments cube schemas to follow Cube.js convention and avoid confusion with template literals (link, link, link, link)
  • Use table names instead of raw SQL queries for MemberOrganizations, MemberSegments, MemberTags, OrganizationSegments, and Tags cube schemas as they do not require any complex transformations or aggregations (link, link, link, link, link, link)
  • Remove measures and dimensions of MemberOrganizations, MemberSegments, and OrganizationSegments cube schemas as they are only used as join tables for other cubes and do not have any meaningful metrics or attributes (link, link, link, link)
  • Add id field to Sentiment cube schema SQL query to ensure a unique identifier for each row and avoid potential errors or duplicates (link)
  • Delete Identities and MemberIdentities cube schemas as they are no longer used by the application (link, link)
  • Add log parameter to onTrigger function of CrowdJob interface to allow job functions to use custom logger from @crowd/logging library (link)
  • Add logExecutionTimeV2 utility function that uses log parameter to log execution time of a given process and replace existing logExecutionTime function that uses default console logger (link)
  • Add .cubecloud folder to .gitignore file to ignore Cube Cloud service folder that may contain sensitive information or configuration details (link)

Checklist ✅

  • Label appropriately with Feature, Improvement, or Bug.
  • Add screehshots to the PR description for relevant FE changes
  • New backend functionality has been unit-tested.
  • API documentation has been updated (if necessary) (see docs on API documentation).
  • Quality standards are met.

@joanreyero joanreyero merged commit 88a3718 into main Aug 29, 2023
9 checks passed
@joanreyero joanreyero deleted the improve-cubejs branch August 29, 2023 12:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants