Ways to optimise bulk exporting (7 million messages or so)?

[davidhamer]

Hi there,

I was wondering if anyone's had to download large quantities of messages and what would be fastest to do so? As far as I can tell my Internet isn't the bottleneck. I'm considering using multiple accounts to get the downloading done. I'm currently downloading in csv format but not sure if this is the best.

Thank you for any help or advice.

[Tyrrrz]

The best approach is to separate exports in time ranges, for example 1 export per 7 days of messages. Then you can export multiple ranges at the same time by launching multiple instances of DiscordChatExporter. Be careful when you do that though.

[davidhamer]

Gotcha, thank you for the quick response. Would it still be better to do separate exports in time ranges if each channel I'm exporting has a maximum of around 50k messages? Also I'm assuming you're saying to be careful re multiple instances on the same account? Or would it be more multiple instances on the same IP address / something different?

[Tyrrrz]

Would it still be better to do separate exports in time ranges if each channel I'm exporting has a maximum of around 50k messages?

Ah, I thought you had a single channel with 7m messages.
If those are all separate channels, you can export them in parallel (in GUI there's an option for that in settings, in CLI it's a command line option). To select multiple channels, use Shift or Control in GUI or just pass multiple IDs in CLI. Then there's probably no need to separate them, as 50k messages shouldn't be too much for a single file.

Also I'm assuming you're saying to be careful re multiple instances on the same account? Or would it be more multiple instances on the same IP address / something different?

I can't say with absolute certainty, but I'd assume Discord would find it concerning if there are a lot of concurrent requests coming from different accounts on the same IP. Generally speaking, parallelization makes it so you end up sending much more requests per second, so it might be suspicious (hence why it's disabled by default).

[davidhamer]

Ah, I thought you had a single channel with 7m messages.
Haha no, luckily not.

Thank you for all the help - love the tool!