fix: unlisten removes all listeners, closes #2264

[FabianLars]

What kind of change does this PR introduce? (check at least one)

• Bugfix
• Feature
• Docs
• New Binding Issue #___
• Code style update
• Refactor
• Build-related changes
• Other, please describe:

Does this PR introduce a breaking change? (check one)

• Yes. Issue #___
• No

The PR fulfills these requirements:

• When resolving a specific issue, it's referenced in the PR's title (e.g. fix: #xxx[,#xxx], where "xxx" is the issue number)
• A change file is added if any packages will require a version bump due to this PR per the instructions in the readme.

If adding a new feature, the PR's description includes:

• A convincing reason for adding this feature (to avoid wasting your time, it's best to open a suggestion issue first and wait for approval before working on it)

Other information:
That one took longer to find than i would like to admit lol. (Sorry for the PR spam)