Missing data due to unstable network connection

[rEnr3n]

$ discord-chat-exporter-cli --version
v2.26.1

I have an unstable internet connection. I tested exporting some channels twice to see if there is a difference. I saw some files have big difference in file size. I diffed them. I saw this at the end:

<   "messageCount": 3800
---
>   "messageCount": 5269

[Tyrrrz]

The number of exported messages has no correlation with the stability of your connection.

[rEnr3n]

I forgot to mention that I export per 1 day. This is the command I use.
discord-chat-exporter-cli export -f Json --dateformat u -o "/tmp/${servername[$channel]} - ${channelname[$channel]} ($date to $nextdate).json" -c "${channelid[$channel]}" -t "$token" --after "$date" --before "$nextdate"

The files that I diffed are <SERVERNAME> - <CHANNELNAME> (2021-03-05 to 2021-03-06).json. When I diffed them, the left file is the first attempt and the right file is the second attempt. That means some data was not retrieved on first attempt. The data in question is a few weeks old. It's not possible for the messageCount to increase (at least not in normal circumstances).

[96-LB]

I often have my daily backups interrupted by internet outages, but the only way I've seen it affect the output is by totally aborting it. The file doesn't reach the end, simply cutting off prematurely (and thus there is no message count listed). As I haven't seen this before, I'm not sure if internet stability is the cause.

[rEnr3n]

This is the script I use:

#!/bin/bash 

usage() {
  echo "Export discord channel history

Usage:
`basename $0` [-c <CHANNEL>] -t <TOKEN> [-d <STARTDATE>]

Options:
  CHANNEL    Name of channel to export
  TOKEN      Your user token
  STARTDATE  Date when to start exporting"
}

help() {
  usage
  exit 0
}

exit_abnormal() {
  usage
  exit 1
}

getall() {
  for key in "${!channelid[@]}"; do
    echo "Retrieving $date..."
    discord-chat-exporter-cli export -f Json --dateformat u -o "/tmp/${servername[$key]} - ${channelname[$key]} ($date to $nextdate).json" -c "${channelid[$key]}" -t "$token" --after "$date" --before "$nextdate"
  done
}

while getopts ":c:t:d:h" opt; do
  case ${opt} in
    c) channel="$OPTARG" ;;
    t) token="$OPTARG" ;;
    d) date="$OPTARG" ;;
    h) help ;;
    *) exit_abnormal ;;
  esac
done

declare -A servername channelname channelid
servername[blackpink-blackpinkchat]="BLACKPINK"
channelname[blackpink-blackpinkchat]="blackpink-chat"
channelid[blackpink-blackpinkchat]="196186445004800000"
### REDACTED THE REST OF THE ENTRIES ###

if [[ -z "$token" ]]; then
  exit_abnormal
fi

if [[ -z "$date" ]]; then
  date=$(date +%F -d $date-1day)
fi
nextdate=$(date +%F -d $date+1day)

until [[ "$date" == $(date +%F) ]]; do
  if [[ -z "$channel" ]]; then
    getall
    date="$nextdate"
    nextdate=$(date +%F -d $date+1day)
  else
    echo "Retrieving $date..."
    discord-chat-exporter-cli export -f Json --dateformat u -o "/tmp/${servername[$channel]} - ${channelname[$channel]} ($date to $nextdate).json" -c "${channelid[$channel]}" -t "$token" --after "$date" --before "$nextdate"
    date="$nextdate"
    nextdate=$(date +%F -d $date+1day)
  fi
done

cp /tmp/*.json .
#grep -i error /tmp/dce.log | grep 'No messages for the specified period' | uniq -cd
#grep -i error /tmp/dce.log | grep -v 'No messages for the specified period'

I use this script inside an archlinux container. This is the package I use: https://aur.archlinux.org/packages/discord-chat-exporter-cli/

[96-LB]

Is there anything abnormal about the channel's contents or the messages that are getting skipped? Are you able to reproduce this with other channels? I did a little poking around but wasn't able to find anything that looked like it would cause the problem.

On an unrelated note, you might want to consider using templates to build your filename instead of array literals so that you don't have to edit the script if a referenced channel/server is renamed.

[rEnr3n]

Is there anything abnormal about the channel's contents or the messages that are getting skipped?

I didn't see anything suspicious about it.

Are you able to reproduce this with other channels?

Yes. See below.

I made a new test yesterday same as before (two tests). It took a long time because of crap internet 😂. I compared these tests and I did not see any difference in messageCount. All of the tests I did had 100+ json files. Out of all of them only one pair had a different messageCount. So I thought there must be a network issue dce was not able to catch.

I use dhcpcd and I checked the logs. Nothing was there. I don't know if the router rebooted at that time. But I made another test today. I ran my script as is. I turned off the router while data was being downloaded. After a minute I saw the error below. The incomplete messageCount was also appended at the bottom of the json file.

Retrieving 2021-02-27...
Exporting channel '<CATEGORY> / <CHANNEL>'... ERROR: System.Threading.Tasks.TaskCanceledException: The operation was canceled.
    System.IO.IOException: Unable to read data from the transport connection: Operation canceled.
        System.Net.Sockets.SocketException: Operation canceled

      at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken) 
      at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.GetResult(Int16 token) 
      at System.Net.Security.SslStream.<FillBufferAsync>g__InternalFillBufferAsync|215_0[TReadAdapter](TReadAdapter adap, ValueTask`1 task, Int32 min, Int32 initial) 
      at System.Net.Security.SslStream.ReadAsyncInternal[TReadAdapter](TReadAdapter adapter, Memory`1 buffer) 
      at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken) 
   at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithNtConnectionAuthAsync(HttpConnection connection, HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncUnbuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at DiscordChatExporter.Domain.Discord.DiscordClient.<>c__DisplayClass5_0.<<GetResponseAsync>b__0>d.MoveNext() in D:\a\DiscordChatExporter\DiscordChatExporter\DiscordChatExporter.Domain\Discord\DiscordClient.cs:line 39
--- End of stack trace from previous location where exception was thrown ---
   at Polly.Retry.AsyncRetryEngine.ImplementationAsync[TResult](Func`3 action, Context context, CancellationToken cancellationToken, ExceptionPredicates shouldRetryExceptionPredicates, ResultPredicates`1 shouldRetryResultPredicates, Func`5 onRetryAsync, Int32 permittedRetryCount, IEnumerable`1 sleepDurationsEnumerable, Func`4 sleepDurationProvider, Boolean continueOnCapturedContext)
   at Polly.AsyncPolicy`1.ExecuteAsync(Func`3 action, Context context, CancellationToken cancellationToken, Boolean continueOnCapturedContext)
   at DiscordChatExporter.Domain.Discord.DiscordClient.GetResponseAsync(String url) in D:\a\DiscordChatExporter\DiscordChatExporter\DiscordChatExporter.Domain\Discord\DiscordClient.cs:line 34
   at DiscordChatExporter.Domain.Discord.DiscordClient.TryGetJsonResponseAsync(String url) in D:\a\DiscordChatExporter\DiscordChatExporter\DiscordChatExporter.Domain\Discord\DiscordClient.cs:line 62
   at DiscordChatExporter.Domain.Discord.DiscordClient.TryGetGuildMemberAsync(Snowflake guildId, User user) in D:\a\DiscordChatExporter\DiscordChatExporter\DiscordChatExporter.Domain\Discord\DiscordClient.cs:line 169
   at DiscordChatExporter.Domain.Exporting.ChannelExporter.ExportChannelAsync(ExportRequest request, IProgress`1 progress) in D:\a\DiscordChatExporter\DiscordChatExporter\DiscordChatExporter.Domain\Exporting\ChannelExporter.cs:line 48
   at DiscordChatExporter.Domain.Exporting.ChannelExporter.ExportChannelAsync(ExportRequest request, IProgress`1 progress) in D:\a\DiscordChatExporter\DiscordChatExporter\DiscordChatExporter.Domain\Exporting\ChannelExporter.cs:line 40
   at DiscordChatExporter.Domain.Exporting.ChannelExporter.ExportChannelAsync(ExportRequest request, IProgress`1 progress) in D:\a\DiscordChatExporter\DiscordChatExporter\DiscordChatExporter.Domain\Exporting\ChannelExporter.cs:line 62
   at DiscordChatExporter.Cli.Commands.Base.ExportCommandBase.ExportAsync(IConsole console, Guild guild, Channel channel) in D:\a\DiscordChatExporter\DiscordChatExporter\DiscordChatExporter.Cli\Commands\Base\ExportCommandBase.cs:line 61
   at DiscordChatExporter.Cli.Commands.Base.ExportCommandBase.ExportAsync(IConsole console, Channel channel) in D:\a\DiscordChatExporter\DiscordChatExporter\DiscordChatExporter.Cli\Commands\Base\ExportCommandBase.cs:line 68
   at DiscordChatExporter.Cli.Commands.Base.ExportCommandBase.ExportAsync(IConsole console, Snowflake channelId) in D:\a\DiscordChatExporter\DiscordChatExporter\DiscordChatExporter.Cli\Commands\Base\ExportCommandBase.cs:line 74
   at DiscordChatExporter.Cli.Commands.ExportChannelCommand.ExecuteAsync(IConsole console) in D:\a\DiscordChatExporter\DiscordChatExporter\DiscordChatExporter.Cli\Commands\ExportChannelCommand.cs:line 18
   at CliFx.CliApplication.RunAsync(IReadOnlyList`1 commandLineArguments, IReadOnlyDictionary`2 environmentVariables)
Retrieving 2021-02-27...

I waited a few more minutes to see if it's going to spit the same error but it did not. It just kept on waiting at Retrieving 2021-02-27... The issue only manifested when the router was turned off while data was in the middle of being downloaded.

---

On an unrelated note, you might want to consider using templates to build your filename instead of array literals so that you don't have to edit the script if a referenced channel/server is renamed.

Some channels put emoji on their channel names. I didn't want it. So I maintain it myself.

[96-LB]

Actually, if I recall correctly, the writers print the postamble (which includes the message count) when being disposed, so it actually makes sense that the file appears "complete" despite failing to export all messages. I was just able to reproduce this by turning off my Wi-Fi connection during a manual export.

but the only way I've seen it affect the output is by totally aborting it.

As such, this statement of mine from earlier is probably unrelated to my unreliable internet and not relevant to the issue at hand.

To confirm: the messages that are missing are the latest messages (i.e. the end of the export is cut off)? Is the issue that DCE is failing silently? I received the same exception that you posted when testing, but if the program reports it, I'm not sure what else can be done besides increasing the timeout.

[rEnr3n]

To confirm: the messages that are missing are the latest messages (i.e. the end of the export is cut off)? Is the issue that DCE is failing silently?

Yes. That's what it looks like.

I received the same exception that you posted when testing, but if the program reports it, I'm not sure what else can be done besides increasing the timeout.

DCE doesn't seem to timeout when it tries to export when the router was already off. It should be the same when the router is turned off midway.

[rEnr3n]

Has this been fixed?

[Tyrrrz]

No, I just closed all issues labeled as questions. Reopened now. I don't know if we can do anything here. Ideas?

[96-LB]

The only solution I see is endless retrying when connection fails instead of eventually timing out, but I don't know if that would be good behaviour. Perhaps if the user is notified that DCE is having trouble connecting, either through a CLI progress message or a GUI pop-up?

[Tyrrrz]

I think they should be already getting the exception mention in this thread. It's also pretty generic (IOException with SocketException inside) so it's hard to wrap it in a more specific error.

[96-LB]

I think they should be already getting the exception mention in this thread.

If I understood correctly, @rEnr3n's request is that the program catches this exception and retries until a connection is available again, as an unstable internet connection has the capability to stop an export and force the user to restart from the beginning. To my knowledge, DCE is already fairly lenient, but I don't know to what scale OP's internet outages are. I'm personally not too keen on attempting to retry indefinitely (at least, not without input from the user), but maybe there's a better way to handle long interruptions.

[rEnr3n]

If I understood correctly, @rEnr3n's request is that the program catches this exception and retries until a connection is available again, as an unstable internet connection has the capability to stop an export and force the user to restart from the beginning.

Internet connection was not the issue. The data I get was consistent despite connectivity issues. The issue was with the router turned off while downloading was in progress. That's the only time I got the error and missing data. I could monitor the logs for this error but I thought there might be a better way.

Could this be an issue with .NET?