Skip to content

Conversation

@3r01
Copy link

@3r01 3r01 commented Mar 16, 2025

Implements permutation generation for duplicate messages.

In fast-moving chats, messages are often dropped. Since we currently generate only two message variations, we can easily trigger this error if any of our messages are dropped:

Your message was not sent because it is identical to the previous one you sent, less than 30 seconds ago.

Since we can't reliably detect if a message was delivered, we can instead increase the number of permutations to reduce the rate of identical messages delivered in sequence, even if some are dropped.


For a message with N usable spaces and using M magic characters, the total permutations are 2^(N + M).

The amount of permutations that we actually generate and use can be controlled by the constants TARGET_PERMUTATION_COUNT and MAX_MAGIC_CHARACTERS in TwitchChannel.cpp.

We generate the minimum number of variants needed to reach the target permutation count, in the following order:

  1. Original message: The base message, using single spaces only.
  2. Space-only Variations: All permutations created by selectively doubling usable spaces
    • For a message with N usable spaces, there are 2^N possible space-only variants (including the original message).
    • Computed in order from least edits to most edits of the original message.
  3. Magic Character Variants:
    • If space-only variants are not enough to reach the target permutation count, a magic suffix is appended to each previous variant.
    • Each magic append comes in two forms: one directly appended, and one with an extra space preceding the magic suffix.
    • This effectively doubles the count of variants per magic level.
    • The maximum number of magic characters can be specified in TwitchChannel.cpp.
      • In this case, we stop generating variants as soon as we reach either:
        • TARGET_PERMUTATION_COUNT, OR
        • MAX_MAGIC_CHARACTERS, whichever is reached first.

In these examples, the target permutation count is 4, and the max magic characters is 1.
The magic character is represented by ?

p0      NaM NaM NaM NaM NaM NaM NaM NaM
p1      NaM  NaM NaM NaM NaM NaM NaM NaM
p2      NaM NaM  NaM NaM NaM NaM NaM NaM
p3      NaM NaM NaM  NaM NaM NaM NaM NaM
p0      NaM NaM NaM NaM NaM NaM NaM NaM

p0      NaM NaM NaM
p1      NaM  NaM NaM
p2      NaM NaM  NaM
p3      NaM  NaM  NaM
p0      NaM NaM NaM

p0      NaM NaM
p1      NaM  NaM
p2      NaM NaM ?
p3      NaM  NaM ?
p0      NaM NaM

p0      NaM
p1      NaM ?
p2      NaM  ?
p0      NaM
NB: here we only reached 3 permutations because we hit the magic character limit (MAX_MAGIC_CHARACTERS) at 1

When messages are dropped, the chance of landing on a duplicate when delivery resumes is 1 / TARGET_PERMUTATION_COUNT, assuming that there are enough spaces available to reach the target permutation count. (This represents the worst case -- however, in practise the chance is lower than this unless you are spamming very fast during a period where delivery is failing).

The existing implementation has 2 permutations, so a 50% chance of landing on a duplicate.

This can inform how to tune TARGET_PERMUTATION_COUNT and MAX_MAGIC_CHARACTERS.

TARGET_PERMUTATION_COUNT = 3 -> duplicate chance = 33%
TARGET_PERMUTATION_COUNT = 4 -> duplicate chance = 25%
TARGET_PERMUTATION_COUNT = 5 -> duplicate chance = 20%
TARGET_PERMUTATION_COUNT = 6 -> duplicate chance = 16.66% (repeating of course)

In this PR, MAX_MAGIC_CHARACTERS = 1 to limit obstructive variants, and TARGET_PERMUTATION_COUNT = 4 as a conservative value—halving the denial rate compared to the existing implementation in the very worst case. However, a higher TARGET_PERMUTATION_COUNT would be fine, since MAX_MAGIC_CHARACTERS already prevents obstructive variants. I leave this decision to existing maintainers.


Duplicate of #6082 with improvements - I was being lazy and not creating an account, so @apa420 submitted for me.
Thank you @apa420 - #6082 can be closed.

Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

@3r01 3r01 force-pushed the duplicate-msg-variants branch from a1d0493 to b7f605f Compare March 16, 2025 21:15
@3r01
Copy link
Author

3r01 commented Mar 16, 2025

warnings and formatting should now have been fixed

@Nerixyz
Copy link
Contributor

Nerixyz commented Mar 16, 2025

This is quite complex. Could we make this simpler? For example, counting in binary (LSB first):

 0: a b c d
 1: a  b c d
 2: a b  c d
 3: a  b  c d
 4: a b c  d
 5: a  b c  d
    ...
15: a  b  c  d ?
 0: a b c d

I don't know if two spaces in multiple locations count as different for Twitch. If they don't, we could still do something similar to the space-only permutation, except that we also have one with the suffix.

@Mm2PL
Copy link
Collaborator

Mm2PL commented Mar 17, 2025 via email

@3r01
Copy link
Author

3r01 commented Mar 17, 2025

I don't know if two spaces in multiple locations count as different for Twitch.

They do.

Could we make this simpler?

Yes, if we accept that we will only ever use a maximum of one magic character (which pajlada had requested anyway), then I can remove the nesting logic.

I would prefer to keep the logic for sorting by least edits to most edits (graded order). I think this is preferable because the unsorted bitmask will add multiple double-spaces before they are necessary, which might be an issue for longer messages which are close to the character limit(?).

However, I think I can still simplify it with a stateful approach using Gosper's hack, which would remove the need for all of the combinatorial logic—that logic is for random access, which we do not need. So, I will make some changes.

@3r01 3r01 force-pushed the duplicate-msg-variants branch from b7f605f to 2e4654d Compare March 17, 2025 06:04
@3r01
Copy link
Author

3r01 commented Mar 17, 2025

As described above. Unfortunately, the ordering no longer strictly follows what was written in the OP.


We now iterate over a single bitmask in graded order using Gosper's hack, up to a maximum of MAX_VARIANT_COUNT = 8 unique variations (incl. the original message). Graded order meaning from fewest bits set to most. So, all combinations with a single edit, then all combinations with two edits, and so on.

The bitmask has two sections:

  • Space bits - bits representing the positions of usable spaces in the message. A set bit at index x indicates that an extra space should be inserted at the corresponding space position (spacePositions[x]). The number of space bits is limited to 62 to avoid overflow in the 64-bit mask.
  • Magic group bits - two bits immediately following the space bits. These control the magic suffix.
    • 00: no magic suffix
    • 01: append magic suffix
    • 10: append a space + magic suffix
    • 11: invalid combo, skipped

The cycle restarts from the original message when either MAX_VARIANT_COUNT is reached, or all valid bitmask combinations for the number of usable spaces have been exhausted.


Outputs
MAX_VARIANT_COUNT = 8

NaM
NaM ?
NaM  ?
NaM

NaM NaM
NaM  NaM
NaM NaM ?
NaM NaM  ?
NaM  NaM ?
NaM  NaM  ?
NaM NaM

NaM NaM NaM
NaM  NaM NaM
NaM NaM  NaM
NaM NaM NaM ?
NaM NaM NaM  ?
NaM  NaM  NaM
NaM  NaM NaM ?
NaM NaM  NaM ?
NaM NaM NaM

NaM NaM NaM NaM
NaM  NaM NaM NaM
NaM NaM  NaM NaM
NaM NaM NaM  NaM
NaM NaM NaM NaM ?
NaM NaM NaM NaM  ?
NaM  NaM  NaM NaM
NaM  NaM NaM  NaM
NaM NaM NaM NaM


NaM NaM NaM NaM NaM
NaM  NaM NaM NaM NaM
NaM NaM  NaM NaM NaM
NaM NaM NaM  NaM NaM
NaM NaM NaM NaM  NaM
NaM NaM NaM NaM NaM ?
NaM NaM NaM NaM NaM  ?
NaM  NaM  NaM NaM NaM
NaM NaM NaM NaM NaM

NaM NaM NaM NaM NaM NaM NaM NaM NaM NaM NaM NaM NaM
NaM  NaM NaM NaM NaM NaM NaM NaM NaM NaM NaM NaM NaM
NaM NaM  NaM NaM NaM NaM NaM NaM NaM NaM NaM NaM NaM
NaM NaM NaM  NaM NaM NaM NaM NaM NaM NaM NaM NaM NaM
NaM NaM NaM NaM  NaM NaM NaM NaM NaM NaM NaM NaM NaM
NaM NaM NaM NaM NaM  NaM NaM NaM NaM NaM NaM NaM NaM
NaM NaM NaM NaM NaM NaM  NaM NaM NaM NaM NaM NaM NaM
NaM NaM NaM NaM NaM NaM NaM  NaM NaM NaM NaM NaM NaM
NaM NaM NaM NaM NaM NaM NaM NaM NaM NaM NaM NaM NaM

@3r01 3r01 force-pushed the duplicate-msg-variants branch from 2e4654d to d3f8168 Compare August 29, 2025 09:58
Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

@3r01
Copy link
Author

3r01 commented Aug 31, 2025

changed magic char to \u{34f} in accordance with #6417

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants