Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

improvement to errors_joined_vw perf #8457

Merged
merged 2 commits into from
May 6, 2024
Merged

improvement to errors_joined_vw perf #8457

merged 2 commits into from
May 6, 2024

Conversation

mayberryzane
Copy link
Contributor

@mayberryzane mayberryzane commented May 3, 2024

Summary

  • small change to errors_joined_vw replaces the line
    SELECT eg.ProjectID as ProjectId, with SELECT ProjectID as ProjectId,
    • this filters using the error_objects' primary key, and since it is the larger table in the join this makes a big perf difference
  • also remove a where clause that was only to handle some bad data
    • in the process of deleting this data (~250 error groups) with the following:
CREATE TABLE errors_to_delete
(
    `ID` Int64,
    `CreatedAt` DateTime64(6)
)
ORDER BY (ID, CreatedAt)

INSERT INTO errors_to_delete
SELECT ID, CreatedAt
FROM default.error_groups
WHERE (ID, CreatedAt) NOT IN (
    SELECT
        ID,
        max(CreatedAt)
    FROM default.error_groups
    GROUP BY ID
)

DELETE
FROM default.error_groups
WHERE (ID, CreatedAt) IN (
    SELECT
        ID,
        CreatedAt FROM
    errors_to_delete
) SETTINGS allow_nondeterministic_mutations=1

How did you test this change?

  • created a new view and tested query perf against it

Are there any deployment considerations?

  • will wait to confirm those ~250 error groups are deleted

Does this work require review from our design team?

  • no

Copy link

changeset-bot bot commented May 3, 2024

⚠️ No Changeset found

Latest commit: d02e0ef

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

💥 An error occurred when fetching the changed packages and changesets in this PR
Some errors occurred when validating the changesets config:
The package or glob expression "rrdom" is specified in the `ignore` option but it is not found in the project. You may have misspelled the package name or provided an invalid glob expression. Note that glob expressions must be defined according to https://www.npmjs.com/package/micromatch.
The package or glob expression "rrdom-nodejs" is specified in the `ignore` option but it is not found in the project. You may have misspelled the package name or provided an invalid glob expression. Note that glob expressions must be defined according to https://www.npmjs.com/package/micromatch.
The package or glob expression "rrweb" is specified in the `ignore` option but it is not found in the project. You may have misspelled the package name or provided an invalid glob expression. Note that glob expressions must be defined according to https://www.npmjs.com/package/micromatch.
The package or glob expression "rrweb-player" is specified in the `ignore` option but it is not found in the project. You may have misspelled the package name or provided an invalid glob expression. Note that glob expressions must be defined according to https://www.npmjs.com/package/micromatch.
The package or glob expression "rrweb-snapshot" is specified in the `ignore` option but it is not found in the project. You may have misspelled the package name or provided an invalid glob expression. Note that glob expressions must be defined according to https://www.npmjs.com/package/micromatch.
The package or glob expression "@rrweb/types" is specified in the `ignore` option but it is not found in the project. You may have misspelled the package name or provided an invalid glob expression. Note that glob expressions must be defined according to https://www.npmjs.com/package/micromatch.
The package or glob expression "@rrweb/web-extension" is specified in the `ignore` option but it is not found in the project. You may have misspelled the package name or provided an invalid glob expression. Note that glob expressions must be defined according to https://www.npmjs.com/package/micromatch.
The package or glob expression "rrvideo" is specified in the `ignore` option but it is not found in the project. You may have misspelled the package name or provided an invalid glob expression. Note that glob expressions must be defined according to https://www.npmjs.com/package/micromatch.

@mayberryzane mayberryzane requested review from a team and Vadman97 and removed request for a team May 3, 2024 20:18
Copy link
Member

@Vadman97 Vadman97 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

don't quite understand why we had those duplicate error groups with the same id... but the view change looks great

@mayberryzane
Copy link
Contributor Author

don't quite understand why we had those duplicate error groups with the same id... but the view change looks great

@Vadman97 I think there was a change at some point to send higher granularity timestamps w/ microsecond instead of second precision, and because CreatedAt is part of the order by key, for some error groups we ended up w/ two rows

@mayberryzane mayberryzane merged commit 4a0cc0b into main May 6, 2024
21 checks passed
@mayberryzane mayberryzane deleted the zane/errors_perf branch May 6, 2024 19:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants