Refactor dispatcher to reduce run time and memory overhead

[bdraco]

Proposed change

• When we removed the last job/callable from the dict for the signal we did not remove the dict for the signal which meant it leaked memory
• Since we tend to create 10000+ of these, avoid the closures as well to reduce memory usage.
• Avoid calling .append() over in over when sending and instead do one list() call since the python overhead of calling .append() was much more expensive than the single list() call

Type of change

• Dependency upgrade
• Bugfix (non-breaking change which fixes an issue)
• New integration (thank you!)
• New feature (which adds functionality to an existing integration)
• Deprecation (breaking change to happen in the future)
• Breaking change (fix/feature causing existing functionality to break)
• Code quality improvements to existing code or addition of tests

Additional information

• This PR fixes or closes issue: fixes #
• This PR is related to issue:
• Link to documentation pull request:

Checklist

• The code change is tested and works locally.
• Local tests pass. Your PR cannot be merged unless tests pass
• There is no commented out code in this PR.
• I have followed the development checklist
• I have followed the perfect PR recommendations
• The code has been formatted using Black (black --fast homeassistant tests)
• Tests have been added to verify that the new code works.

If user exposed functionality or configuration variables are added/changed:

• Documentation added/updated for www.home-assistant.io

If the code communicates with devices, web services, or third-party tools:

• The manifest file has all fields filled out correctly.
  Updated and included derived files by running: python3 -m script.hassfest.
• New or updated dependencies have been added to requirements_all.txt.
  Updated by running python3 -m script.gen_requirements_all.
• For the updated dependencies - a link to the changelog, or at minimum a diff between library versions is added to the PR description.
• Untested files have been added to .coveragerc.

To help with the load of incoming pull requests:

• I have reviewed two other open pull requests in this repository.

[bdraco]

partial creates a new function wrapper with different arguments but the underlying function body reference is the same https://github.com/python/cpython/blob/cf19e8ea3a232086ff1e0d7d5e2a092d3d96fc7c/Modules/_functoolsmodule.c#L84 so we don't end up with a new function body per dispatcher connect in memory.

Memory used per closure: 408.7808
Memory used per partial: 250.01984

from functools import partial

import psutil

process = psutil.Process()
before = process.memory_info().rss
closures = []


def gen_closure(i, y, q):
    def this_is_a_closure():
        z = i + 1
        print(z)
        z = y + 1
        print(z)
        z += q
        new_dict = {}
        new_dict["key"] = "value"
        return i

    return this_is_a_closure


for i in range(100000):
    closures.append(gen_closure(i, 6, 7))

after = process.memory_info().rss

print(f"Memory used per closure: {(after - before) / len(closures):,}")


before = process.memory_info().rss


def use_for_partial(i, y, q):
    z = i + 1
    print(z)
    z = y + 1
    print(z)
    z += q
    new_dict = {}
    new_dict["key"] = "value"
    return i


partials = []
for i in range(100000):
    partials.append(partial(use_for_partial, i, 6, 7))

after = process.memory_info().rss
print(f"Memory used per partial: {(after - before) / len(partials):,}")

[bdraco]

Better to just make a copy of the items instead of appending it all to a list one at a time since the python function overhead of calling .append() over and over it much worse

[bdraco]

thanks