Reactions not being returned correctly for a message

[GamingGeek]

Description
It seems the most recent incident with reactions have broken some messages. Users are unable to see the reaction on a message (including myself) and when I react with the emoji that used to be there, I can see the users that had reacted before the incident but the reaction does not persist. If I open the same channel in the browser version or reload the client, I can't see the reaction anymore

Steps to Reproduce
I personally only have one message where reactions are broken but others have mentioned having issues with reactions since the incident in the Discord Developers server.

https://canary.discord.com/channels/613425648685547541/697489244649816084/826417781330149406
https://canary.discord.com/channels/613425648685547541/697489244649816084/826447264867155989
https://canary.discord.com/channels/613425648685547541/697489244649816084/826464233192685578

Below are the IDs for the specific message I am seeing the issue with:
Guild ID: 755794954743185438
Channel ID: 819580357053251604
Message ID: 819580586183229460
Emoji: <a:ablobtonguewink:738118274067464293>

Expected Behavior
The reaction that used to be there should be returned by the API when fetching the message.

Current Behavior
No reactions are returned when fetching the message, reacting with the emoji that used to be there doesn't work but when I react and hover over the message, the client is able to fetch the users who had reacted before the incident, indicating the reaction still exists but is in a broken state.

Screenshots/Videos

Before reacting:

After reacting:

Fetching the message with my bot and checking reactions:

Fetching the reaction specifically with my bot and listing users:

Adding the reaction appears to be successful:

but upon reloading the client, it is no longer there

Just like my bot, the client is able to fetch and display the users who have reacted just fine:


yet the reaction count stays at 1

A video showing me reacting, the client fetching users who had previously reacted, reloading the client and seeing the reaction is no longer there:
https://user-images.githubusercontent.com/29981918/113142883-586d4980-9223-11eb-8710-7aa8c8f2a334.mp4

A video showing me adding a different reaction, reloading the client and seeing both the reaction I just added and the original reaction appearing but once I remove the new reaction and reload the old one is gone too:
https://user-images.githubusercontent.com/29981918/113143182-c3b71b80-9223-11eb-8856-65e0cff40311.mp4

Client and System Information
N/A, reproducible on any client/system.

[elderlabs]

Adding a bit here. In my testing yesterday with this bug while testing to see if it's connected to the long-running bug that affects at least large guilds spread across multiple nodes, it would appear the client renders reactions on a message in some way attached to the reaction's name. After the outage yesterday, many current reactions lost their names, thus when you add an additional reaction to a message, the client hits the reaction endpoint directly, and renders all reactions on the message, but leaves the broken reaction with a name of :null:. Re-adding the broken reaction after adding an additional one seems to correct the null name and count bug, but once you re-add the reaction, it must stay there to properly restore the name, leading me to assume it's only rendering now because the new reaction contains reaction name data. It may be best to have a bot add the reaction if it's used for an automated process, as we use them.

The long-running bug that's affected us for years and is mentioned somewhere in Discord.js's documentation seems to be a caching bug on Discord's end, which may also have a hand in this bug. Essentially, pulling the affected message via message history or directly from the API shows an empty list of reactions, but if you pull the reaction directly from the message, it returns a list of users that have reacted to the message. Essentially, the caching of reactions on the message side in this case isn't working, though the data is there if queried directly.

For internal staffer use and investigation, I've saved a message affected by this long-term bug here: /375436620578684930/375446800863330305/776939249106419752

Thought I'd throw in my two cents here as you've documented the new issue fairly well and left out almost nothing at all.
Thanks for documenting and filing this issue.

[lytefast]

A video showing me adding a different reaction, reloading the client and seeing both the reaction I just added and the original reaction appearing but once I remove the new reaction and reload the old one is gone too:
https://user-images.githubusercontent.com/29981918/113143182-c3b71b80-9223-11eb-8856-65e0cff40311.mp4

@GamingGeek This is super interesting. Could you show me what the message payload looks like after you add the extra emoji?
I also noticed that (as the comment above states as well) the emoji is :null:

A potential fix for the ghost reactions is to remove a reaction (as a single user) and add it back. This should fix the msg.reaction which tells our services to fetch counts

[GamingGeek]

Could you show me what the message payload looks like after you add the extra emoji?

https://h.inv.wtf/apagexoqew.json

A potential fix for the ghost reactions is to remove a reaction (as a single user) and add it back. This should fix the msg.reaction which tells our services to fetch counts

This seems to have worked, thank you @lytefast :D

[lytefast]

After the outage yesterday, many current reactions lost their names, thus when you add an additional reaction to a message, the client hits the reaction endpoint directly, and renders all reactions on the message, but leaves the broken reaction with a name of :null:. Re-adding the broken reaction after adding an additional one seems to correct the null name and count bug, but once you re-add the reaction, it must stay there to properly restore the name, leading me to assume it's only rendering now because the new reaction contains reaction name data. It may be best to have a bot add the reaction if it's used for an automated process, as we use them.

First off thanks for helping. This inspired me to look at another path, and I think I figured it out:
This was definitely related to the outage issue we had where emojis names for custom emojis were being dropped.

Impact

This effects all custom reactions for all messages regardless of if the client is a user or bot. If any user elected to remove a reaction on a custom emoji during this timeframe would have the custom emoji reaction deleted.

Note: this only wipes the msg.reactions which we use as a hint to load the users + count that you see in the end
that's why removing youself + adding the emoji again fixes it:
after a remove the next add is not short-circuited, and we see that the message doesn't that emoji and adds it

Cause

• we were looking through our DB with wrong emoji id
• in order to detect if we need to remove an emoji msg.reactions, let's say we use a function called is_last_reaction(emoji)

If a any request came to remove a reaction to a custom emoji we check to see if it's the last emoji using is_last_reaction(emoji). However due to an error in how we now do this check, this would always return True since our emoji key that we use was incorrect. So any remove action from any user on a reaction would trigger the API to delete the msg.reaction for that custom emoji.

Fix (already out)

This flow was detected during the incident and patched. Additional safeguards at lower levels will also prevent this from occuring in the future.