Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Recover from previously failed statistics migrations #101781

Merged
merged 19 commits into from Oct 23, 2023
Merged

Recover from previously failed statistics migrations #101781

merged 19 commits into from Oct 23, 2023

Conversation

bdraco
Copy link
Member

@bdraco bdraco commented Oct 10, 2023 •
edited

Proposed change

The stats timestamp migration had some somewhat rare failure cases that were not accounted for:

  • If a user has duplicate rows that were manually inserted or inserted by external stats before we had the unique index they could fail to migrate. We now fallback to a one by one migration when this happens.
  • If the user downgraded HA from 2023.3.x to an older version without restoring the database and they upgrade again with the same database they will have un-migrated statistics columns since we only migrate them once.

Because this involves migrating data, this should go through beta. While it means the users in the linked issue will have to wait a month for a solution, we need to make sure it doesn't have any side effects.

Type of change

  • Dependency upgrade
  • Bugfix (non-breaking change which fixes an issue)
  • New integration (thank you!)
  • New feature (which adds functionality to an existing integration)
  • Deprecation (breaking change to happen in the future)
  • Breaking change (fix/feature causing existing functionality to break)
  • Code quality improvements to existing code or addition of tests

Additional information

Checklist

  • The code change is tested and works locally.
  • Local tests pass. Your PR cannot be merged unless tests pass
  • There is no commented out code in this PR.
  • I have followed the development checklist
  • I have followed the perfect PR recommendations
  • The code has been formatted using Black (black --fast homeassistant tests)
  • Tests have been added to verify that the new code works.

If user exposed functionality or configuration variables are added/changed:

If the code communicates with devices, web services, or third-party tools:

  • The manifest file has all fields filled out correctly.
    Updated and included derived files by running: python3 -m script.hassfest.
  • New or updated dependencies have been added to requirements_all.txt.
    Updated by running python3 -m script.gen_requirements_all.
  • For the updated dependencies - a link to the changelog, or at minimum a diff between library versions is added to the PR description.
  • Untested files have been added to .coveragerc.

To help with the load of incoming pull requests:

@bdraco
Handle statistics columns being unmigrated from previous downgrades
c78c20f 
If the user downgraded HA from 2023.3.x to an older version without
restoring the database and they upgrade again with the same database
they will have unmigrated statistics columns since we only migrate them
once.

As its expensive to check, we do not want to check every time
at startup, so we will only do this one more time since the
risk that someone will downgrade to an older version is very
low at this point.
@home-assistant
Copy link

Hey there @home-assistant/core, mind taking a look at this pull request as it has been labeled with an integration (recorder) you are listed as a code owner for? Thanks!

Code owner commands

Code owners of recorder can trigger bot actions by commenting:

  • @home-assistant close Closes the pull request.
  • @home-assistant rename Awesome new title Renames the pull request.
  • @home-assistant reopen Reopen the pull request.
  • @home-assistant unassign recorder Removes the current integration label and assignees on the pull request, add the integration domain after the command.

@bdraco
Copy link
Member Author

bdraco commented Oct 10, 2023

This needs a test to ensure rows get preserved

Insert old rows
Run migration code
Insert new rows
Run migration code
Insert old rows
Run migration code

Verify we can see all rows in new format

@bdraco bdraco mentioned this pull request Oct 10, 2023
Closed
@bdraco bdraco marked this pull request as ready for review October 10, 2023 21:40
@bdraco bdraco requested a review from a team as a code owner October 10, 2023 21:40
@bdraco bdraco marked this pull request as draft October 15, 2023 18:10
@bdraco bdraco changed the title Handle statistics columns being unmigrated from previous downgrades Fix failed statistics migrations Oct 15, 2023
@bdraco bdraco changed the title Fix failed statistics migrations Recover from failed statistics migrations Oct 15, 2023
@bdraco bdraco changed the title Recover from failed statistics migrations Recover from previously failed statistics migrations Oct 15, 2023
@bdraco bdraco marked this pull request as ready for review October 16, 2023 01:44
@bdraco
Copy link
Member Author

bdraco commented Oct 19, 2023

merge in dev to make sure this still passes with the other changes

@frenck frenck added the smash Indicator this PR is close to finish for merging or closing label Oct 22, 2023
@balloob balloob merged commit 268425b into dev Oct 23, 2023
53 checks passed
@balloob balloob deleted the migrate_stats branch October 23, 2023 03:34
@github-actions github-actions bot locked and limited conversation to collaborators Oct 24, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@balloob balloob balloob approved these changes

Assignees
No one assigned
Labels
cla-signed core Hacktoberfest integration: recorder Quality Scale: internal smash Indicator this PR is close to finish for merging or closing
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Energy dashboard stopped working after updating to 2023.10.0: TypeError when compiling stats
3 participants
@bdraco @balloob @frenck