Fix derivative integration showing unexpected spikes

[sophof]

Proposed change

Fixes the bug as described in #31579 (and others) for the derivative integration by implementing a more robust algorithm with respect to unevenly spaced updates. In essence applies a simple moving average weighted by time as described in http://www.eckner.com/papers/Algorithms%20for%20Unevenly%20Spaced%20Time%20Series.pdf in the SMAlast section. For more info, see the issue and my posts there. My code has changed a little from what I've described there, but the principle hasn't really changed.

I have added a test for the issue (that fails for the old algorithm) and added a few more tests for unevenly spaced derivatives. Existing tests also show a higher accuracy than before. In addition I've run this code on my own installation for the past two weeks and it has solved the issue without impacting the signal.

Note that the old code essentially had an unpredictable smoothing window (it only had a maximum size), which means that the parameter has sort of changed in effectiveness (is that a word?). What I mean is, I expect that some people will want to reduce their window size with this new code, but the change is not strictly breaking. If this PR is accepted I'll happily update the documentation too (because my PRs have been denied in the past for surprising reasons I'd like to wait with that).

NOTE: I wasn't sure if I should add myself as a codeowner, so I did not, is that ok?

Type of change

• Dependency upgrade
• Bugfix (non-breaking change which fixes an issue)
• New integration (thank you!)
• New feature (which adds functionality to an existing integration)
• Breaking change (fix/feature causing existing functionality to break)
• Code quality improvements to existing code or addition of tests

Additional information

• This PR fixes or closes issue: fixes Derivative sensor doesn't update correctly for non-changing values #31579
• This PR is related to issue: Derivative sensor doesn't update correctly for non-changing values #31579
• Link to documentation pull request:

Checklist

• The code change is tested and works locally.
• Local tests pass. Your PR cannot be merged unless tests pass
• There is no commented out code in this PR.
• I have followed the development checklist
• The code has been formatted using Black (black --fast homeassistant tests)
• Tests have been added to verify that the new code works.

If user exposed functionality or configuration variables are added/changed:

• Documentation added/updated for www.home-assistant.io

If the code communicates with devices, web services, or third-party tools:

• The manifest file has all fields filled out correctly.
  Updated and included derived files by running: python3 -m script.hassfest.
• New or updated dependencies have been added to requirements_all.txt.
  Updated by running python3 -m script.gen_requirements_all.
• For the updated dependencies - a link to the changelog, or at minimum a diff between library versions is added to the PR description.
• Untested files have been added to .coveragerc.

The integration reached or maintains the following Integration Quality Scale:

• No score or internal
• 🥈 Silver
• 🥇 Gold
• 🏆 Platinum

To help with the load of incoming pull requests:

• I have reviewed two other open pull requests in this repository.

[probot-home-assistant]

Hey there @afaucogney, mind taking a look at this pull request as it has been labeled with an integration (derivative) you are listed as a code owner for? Thanks!
_{^{(message by CodeOwnersMention)}}

[balloob]

Thanks

[afaucogney]

@sophof I'm happy you found a dedicated educated solution. I'm the author of the original algorithme, but not about the time windows mechanism (that I find partial).

I'm not sure I understand 100% of the scientist doc, but at least it seems experienced peoples have been around.

I would be glad if you could add you name as an author. I'm not a python dev, so help is welcome. You may bring your experience and feedback if anyone need. But I do not know if this is the way.

[sophof]

@sophof I'm happy you found a dedicated educated solution. I'm the author of the original algorithme, but not about the time windows mechanism (that I find partial).

I'm not sure I understand 100% of the scientist doc, but at least it seems experienced peoples have been around.

I would be glad if you could add you name as an author. I'm not a python dev, so help is welcome. You may bring your experience and feedback if anyone need. But I do not know if this is the way.

This specific bit is not my field, but I work in statistics, so it was probably a bit more natural to me. Since the code is now merged (thx!) I can't add myself as the author anymore, but will if I do any other changes in the future. And of course, feel free to contact me if this turns out to be full of bugs (I'm always a bit scared actually sending it out into the wild :)).

I'll have at least a look at the documentation to see if it can be improved a little for the window. I think it is important if people have the right intuition for these things.