Stuck at messages containing invite links

[Zwelf]

Version

v2.40.0

Flavor

CLI (Command-Line Interface)

Export format

HTML

Details

Exported a channel. Was stuck for >5 hours. When looking at what messages were currently exported, it were spam messages containing a link to an invalid channel invite. (and saw it happen to other links to invalid invites as well, e.g. https://discord.gg/wJuFkEkME9)

Steps to reproduce

• Join https://ddnet.org/discord
• discord-chat-exporter export -c 293493549758939136 -t "${TOKEN}" --after 338128201123495937 --before 338129425365336065

[Zwelf]

Under further testing, it also gets stuck for me on valid invites

[Zwelf]

I do get further today. So maybe it is stricter rate limiting on invite link resolving api? I'm still stuck on exporting channel history with many invites (they are all the same invite link). Also I get this exception on master:

System.InvalidOperationException: Cannot read the specified JSON element as a non-empty and non-whitespace string value.
  at JsonExtensions.Reading.StringExtensions.GetNonWhiteSpaceString(JsonElement element) in /_/JsonExtensions/Reading/StringExtensions.cs:55
  at DiscordChatExporter.Core.Discord.Data.Channel.Parse(JsonElement json, ChannelCategory categoryHint, Nullable`1 positionHint, String parentName, Nullable`1 parentPosition) in /home/user/dev/DiscordChatExporter/DiscordChatExporter.Core/Discord/Data/Channel.cs:39
  at DiscordChatExporter.Core.Discord.DiscordClient.GetGuildChannelsAsync(Snowflake guildId, CancellationToken cancellationToken)+MoveNext() in /home/user/dev/DiscordChatExporter/DiscordChatExporter.Core/Discord/DiscordClient.cs:241
  at DiscordChatExporter.Core.Discord.DiscordClient.GetGuildChannelsAsync(Snowflake guildId, CancellationToken cancellationToken)+System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult() 
  at DiscordChatExporter.Core.Exporting.ExportContext.PopulateChannelsAndRolesAsync(CancellationToken cancellationToken) in /home/user/dev/DiscordChatExporter/DiscordChatExporter.Core/Exporting/ExportContext.cs:40
  at DiscordChatExporter.Core.Exporting.ExportContext.PopulateChannelsAndRolesAsync(CancellationToken cancellationToken) in /home/user/dev/DiscordChatExporter/DiscordChatExporter.Core/Exporting/ExportContext.cs:40
  at DiscordChatExporter.Core.Exporting.ChannelExporter.ExportChannelAsync(ExportRequest request, IProgress`1 progress, CancellationToken cancellationToken) in /home/user/dev/DiscordChatExporter/DiscordChatExporter.Core/Exporting/ChannelExporter.cs:31
  at DiscordChatExporter.Cli.Commands.Base.ExportCommandBase.<>c__DisplayClass55_2.<<ExecuteAsync>b__2>d.MoveNext() in /home/user/dev/DiscordChatExporter/DiscordChatExporter.Cli/Commands/Base/ExportCommandBase.cs:219
  at DiscordChatExporter.Cli.Utils.Extensions.ConsoleExtensions.StartTaskAsync(ProgressContext progressContext, String description, Func`2 performOperationAsync) in /home/user/dev/DiscordChatExporter/DiscordChatExporter.Cli/Utils/Extensions/ConsoleExtensions.cs:43
  at DiscordChatExporter.Cli.Commands.Base.ExportCommandBase.<>c__DisplayClass55_1.<<ExecuteAsync>b__1>d.MoveNext() in /home/user/dev/DiscordChatExporter/DiscordChatExporter.Cli/Commands/Base/ExportCommandBase.cs:197
  at System.Threading.Tasks.Parallel.<>c__50`1.<<ForEachAsync>b__50_0>d.MoveNext() 
  at DiscordChatExporter.Cli.Commands.Base.ExportCommandBase.<>c__DisplayClass55_0.<<ExecuteAsync>b__0>d.MoveNext() in /home/user/dev/DiscordChatExporter/DiscordChatExporter.Cli/Commands/Base/ExportCommandBase.cs:186
  at Spectre.Console.Progress.<>c__DisplayClass27_0.<<StartAsync>b__0>d.MoveNext() in /_/src/Spectre.Console/Live/Progress/Progress.cs:98
  at Spectre.Console.Progress.<>c__DisplayClass28_0`1.<<StartAsync>b__0>d.MoveNext() in /_/src/Spectre.Console/Live/Progress/Progress.cs:133
  at Spectre.Console.Internal.DefaultExclusivityMode.RunAsync[T](Func`1 func) in /_/src/Spectre.Console/Internal/DefaultExclusivityMode.cs:40
  at Spectre.Console.Progress.StartAsync[T](Func`2 action) in /_/src/Spectre.Console/Live/Progress/Progress.cs:116
  at Spectre.Console.Progress.StartAsync(Func`2 action) in /_/src/Spectre.Console/Live/Progress/Progress.cs:96
  at DiscordChatExporter.Cli.Commands.Base.ExportCommandBase.ExecuteAsync(IConsole console, IReadOnlyList`1 channels) in /home/user/dev/DiscordChatExporter/DiscordChatExporter.Cli/Commands/Base/ExportCommandBase.cs:184
  at DiscordChatExporter.Cli.Commands.Base.ExportCommandBase.ExecuteAsync(IConsole console, IReadOnlyList`1 channelIds) in /home/user/dev/DiscordChatExporter/DiscordChatExporter.Cli/Commands/Base/ExportCommandBase.cs:307
  at DiscordChatExporter.Cli.Commands.ExportChannelsCommand.ExecuteAsync(IConsole console) in /home/user/dev/DiscordChatExporter/DiscordChatExporter.Cli/Commands/ExportChannelsCommand.cs:26
  at CliFx.CliApplication.RunAsync(ApplicationSchema applicationSchema, CommandInput commandInput) in /_/CliFx/CliApplication.cs:153
  at CliFx.CliApplication.RunAsync(IReadOnlyList`1 commandLineArguments, IReadOnlyDictionary`2 environmentVariables) in /_/CliFx/CliApplication.cs:193

[Zwelf]

I bisected it to be a regression from this commit: 191294b. Related issue: #649

[Zwelf]

I see two possible solutions:

1. cache data for invites (because this is triggered here probably only because there are roughly 3750 consecutive invites to the same channel)
2. Add a command line argument option to disable invite gathering

[Zwelf]

So it seems real steps to reproduce include exporting the full invite range

dotnet run -- export -c 293493549758939136 --after 2017-07-22T00:00 --before 2017-07-23T00:00 -t ${TOKEN}

[Tyrrrz]

It does pull invite metadata for each discord.gg/... link. It shouldn't take long enough to be noticeable, but I guess if there are lots of spam invites, it can happen.

[Zwelf]

The problem is that after getting rate limited once, I can't resolve any discord.gg links anymore. As soon as I get to any message containing such a link the export stops making progress for ever. Even when switching accounts. So probably the IP gets rate limited.

[Zwelf]

I solved it for me by removing the code resolving the discord invite link for exporting the days with the spam. Unfortunately that is not a general solution.

Diff to 48eece5

diff --git a/DiscordChatExporter.Core/Exporting/MessageGroupTemplate.cshtml b/DiscordChatExporter.Core/Exporting/MessageGroupTemplate.cshtml
index 44d8b75..c9491de 100644
--- a/DiscordChatExporter.Core/Exporting/MessageGroupTemplate.cshtml
+++ b/DiscordChatExporter.Core/Exporting/MessageGroupTemplate.cshtml
@@ -310,49 +310,6 @@
                         </div>
                     }
 
-                    @{/* Invites */}
-                    @{
-                        var inviteCodes = MarkdownParser
-                            .ExtractLinks(message.Content)
-                            .Select(l => l.Url)
-                            .Select(Invite.TryGetCodeFromUrl)
-                            .WhereNotNull()
-                            .ToArray();
-
-                        foreach (var inviteCode in inviteCodes)
-                        {
-                            var invite = await Context.Discord.TryGetGuildInviteAsync(inviteCode, CancellationToken);
-                            if (invite is null)
-                            {
-                                continue;
-                            }
-
-                            <div class="chatlog__embed">
-                                <div class="chatlog__embed-invite-container">
-                                    <div class="chatlog__embed-invite-title">@(invite.Channel?.Kind.IsDirect() == true ? "Invite to join a group DM" : "Invite to join a server")</div>
-                                    <div class="chatlog__embed-invite">
-                                        <div class="chatlog__embed-invite-guild-icon-container">
-                                            <img class="chatlog__embed-invite-guild-icon" src="@await ResolveAssetUrlAsync(invite.Channel?.IconUrl ?? invite.Guild.IconUrl)" alt="Guild icon" loading="lazy">
-                                        </div>
-                                        <div class="chatlog__embed-invite-info">
-                                            <div class="chatlog__embed-invite-guild-name">
-                                                <a href="https://discord.gg/@invite.Code">
-                                                    @(invite.Guild.Name)
-                                                </a>
-                                            </div>
-                                            <div class="chatlog__embed-invite-channel-name">
-                                                <svg class="chatlog__embed-invite-channel-icon">
-                                                    <use href="#channel-icon"></use>
-                                                </svg>
-                                                <span> @(invite.Channel?.Name ?? "Unknown Channel")</span>
-                                            </div>
-                                        </div>
-                                    </div>
-                                </div>
-                            </div>
-                        }
-                    }
-
                     @{/* Embeds */}
                     @foreach (var embed in message.Embeds)
                     {

[Tyrrrz]

You can now skip invite links by adding the -has:invite message filter