Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reduce sqlalchemy reflection overhead at recorder setup time #113989

Merged
merged 1 commit into from Mar 23, 2024

Conversation

bdraco
Copy link
Member

@bdraco bdraco commented Mar 22, 2024

Proposed change

Only reflect back the table we are checking instead of all tables

Type of change

  • Dependency upgrade
  • Bugfix (non-breaking change which fixes an issue)
  • New integration (thank you!)
  • New feature (which adds functionality to an existing integration)
  • Deprecation (breaking change to happen in the future)
  • Breaking change (fix/feature causing existing functionality to break)
  • Code quality improvements to existing code or addition of tests

Additional information

  • This PR fixes or closes issue: fixes #
  • This PR is related to issue:
  • Link to documentation pull request:

Checklist

  • The code change is tested and works locally.
  • Local tests pass. Your PR cannot be merged unless tests pass
  • There is no commented out code in this PR.
  • I have followed the development checklist
  • I have followed the perfect PR recommendations
  • The code has been formatted using Ruff (ruff format homeassistant tests)
  • Tests have been added to verify that the new code works.

If user exposed functionality or configuration variables are added/changed:

If the code communicates with devices, web services, or third-party tools:

  • The manifest file has all fields filled out correctly.
    Updated and included derived files by running: python3 -m script.hassfest.
  • New or updated dependencies have been added to requirements_all.txt.
    Updated by running python3 -m script.gen_requirements_all.
  • For the updated dependencies - a link to the changelog, or at minimum a diff between library versions is added to the PR description.
  • Untested files have been added to .coveragerc.

To help with the load of incoming pull requests:

Only reflect back the table we are checking instead of all tables
@home-assistant
Copy link

Hey there @home-assistant/core, mind taking a look at this pull request as it has been labeled with an integration (recorder) you are listed as a code owner for? Thanks!

Code owner commands

Code owners of recorder can trigger bot actions by commenting:

  • @home-assistant close Closes the pull request.
  • @home-assistant rename Awesome new title Renames the pull request.
  • @home-assistant reopen Reopen the pull request.
  • @home-assistant unassign recorder Removes the current integration label and assignees on the pull request, add the integration domain after the command.
  • @home-assistant add-label needs-more-information Add a label (needs-more-information, problem in dependency, problem in custom component) to the pull request.
  • @home-assistant remove-label needs-more-information Remove a label (needs-more-information, problem in dependency, problem in custom component) on the pull request.

@bdraco
Copy link
Member Author

bdraco commented Mar 22, 2024

There was a lot of show create table happening

Now each check is only checking the table it cares about and now reflecting all of them for each table

2024-03-22 04:12:00.636 INFO (Recorder) [sqlalchemy.engine.Engine] ROLLBACK
2024-03-22 04:12:00.639 INFO (Recorder) [sqlalchemy.engine.Engine] BEGIN (implicit)
2024-03-22 04:12:00.639 INFO (Recorder) [sqlalchemy.engine.Engine] SHOW CREATE TABLE `events`
2024-03-22 04:12:00.639 INFO (Recorder) [sqlalchemy.engine.Engine] [raw sql] ()
2024-03-22 04:12:00.644 INFO (Recorder) [sqlalchemy.engine.Engine] SHOW CREATE TABLE `event_data`
2024-03-22 04:12:00.644 INFO (Recorder) [sqlalchemy.engine.Engine] [raw sql] ()
2024-03-22 04:12:00.648 INFO (Recorder) [sqlalchemy.engine.Engine] ROLLBACK

before we got

2024-03-22 03:50:27.770 INFO (Recorder) [sqlalchemy.engine.Engine] SHOW FULL TABLES FROM `homeassistant`
2024-03-22 03:50:27.770 INFO (Recorder) [sqlalchemy.engine.Engine] [raw sql] ()
2024-03-22 03:50:27.772 INFO (Recorder) [sqlalchemy.engine.Engine] SHOW CREATE TABLE `event_data`
2024-03-22 03:50:27.773 INFO (Recorder) [sqlalchemy.engine.Engine] [raw sql] ()
2024-03-22 03:50:27.774 INFO (Recorder) [sqlalchemy.engine.Engine] SHOW CREATE TABLE `event_types`
2024-03-22 03:50:27.774 INFO (Recorder) [sqlalchemy.engine.Engine] [raw sql] ()
2024-03-22 03:50:27.776 INFO (Recorder) [sqlalchemy.engine.Engine] SHOW CREATE TABLE `events`
2024-03-22 03:50:27.776 INFO (Recorder) [sqlalchemy.engine.Engine] [raw sql] ()
2024-03-22 03:50:27.778 INFO (Recorder) [sqlalchemy.engine.Engine] SHOW CREATE TABLE `migration_changes`
2024-03-22 03:50:27.778 INFO (Recorder) [sqlalchemy.engine.Engine] [raw sql] ()
2024-03-22 03:50:27.780 INFO (Recorder) [sqlalchemy.engine.Engine] SHOW CREATE TABLE `recorder_runs`
2024-03-22 03:50:27.780 INFO (Recorder) [sqlalchemy.engine.Engine] [raw sql] ()
2024-03-22 03:50:27.782 INFO (Recorder) [sqlalchemy.engine.Engine] SHOW CREATE TABLE `schema_changes`
2024-03-22 03:50:27.783 INFO (Recorder) [sqlalchemy.engine.Engine] [raw sql] ()
2024-03-22 03:50:27.784 INFO (Recorder) [sqlalchemy.engine.Engine] SHOW CREATE TABLE `state_attributes`
2024-03-22 03:50:27.784 INFO (Recorder) [sqlalchemy.engine.Engine] [raw sql] ()
2024-03-22 03:50:27.786 INFO (Recorder) [sqlalchemy.engine.Engine] SHOW CREATE TABLE `states`
2024-03-22 03:50:27.786 INFO (Recorder) [sqlalchemy.engine.Engine] [raw sql] ()
2024-03-22 03:50:27.788 INFO (Recorder) [sqlalchemy.engine.Engine] SHOW CREATE TABLE `states_meta`
2024-03-22 03:50:27.788 INFO (Recorder) [sqlalchemy.engine.Engine] [raw sql] ()
2024-03-22 03:50:27.790 INFO (Recorder) [sqlalchemy.engine.Engine] SHOW CREATE TABLE `statistics`
2024-03-22 03:50:27.790 INFO (Recorder) [sqlalchemy.engine.Engine] [raw sql] ()
2024-03-22 03:50:27.792 INFO (Recorder) [sqlalchemy.engine.Engine] SHOW CREATE TABLE `statistics_meta`
2024-03-22 03:50:27.792 INFO (Recorder) [sqlalchemy.engine.Engine] [raw sql] ()
2024-03-22 03:50:27.794 INFO (Recorder) [sqlalchemy.engine.Engine] SHOW CREATE TABLE `statistics_runs`
2024-03-22 03:50:27.794 INFO (Recorder) [sqlalchemy.engine.Engine] [raw sql] ()
2024-03-22 03:50:27.796 INFO (Recorder) [sqlalchemy.engine.Engine] SHOW CREATE TABLE `statistics_short_term`
2024-03-22 03:50:27.796 INFO (Recorder) [sqlalchemy.engine.Engine] [raw sql] ()

@bdraco bdraco marked this pull request as ready for review March 22, 2024 17:17
@bdraco bdraco requested a review from a team as a code owner March 22, 2024 17:17
@bdraco bdraco added the smash Indicator this PR is close to finish for merging or closing label Mar 22, 2024
Copy link
Member

@frenck frenck left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, @bdraco 👍

../Frenck

@frenck frenck merged commit de62b77 into dev Mar 23, 2024
35 checks passed
@frenck frenck deleted the reflect_overhead branch March 23, 2024 10:25
@bdraco
Copy link
Member Author

bdraco commented Mar 23, 2024

thanks

@github-actions github-actions bot locked and limited conversation to collaborators Mar 24, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
cla-signed code-quality core integration: recorder Quality Scale: internal small-pr PRs with less than 30 lines. smash Indicator this PR is close to finish for merging or closing
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants