Import-tool: Improve API to extract categories and channels #6690
Assignees
Comments
0x-r4bbit
added a commit
to status-im/status-go
that referenced
this issue
Aug 2, 2022
As part of the new Discord <-> Status Community Import functionality, we're adding an API that extracts all discord categories and channels from a previously exported discord export file. These APIs can be used in clients to show the user what categories and channels will be imported later on. There are two APIs: 1. `Messenger.ExtractDiscordCategoriesAndChannels(filesToimport []string) (*MessengerResponse, map[string]*discord.ImportError)` This takes a list of exported discord export (JSON) files (typically one per channel), reads them, and extracts the categories and channels into dedicated data structures (`[]DiscordChannel` and `[]DiscordCategory`) It also returns the oldest message timestamp found in all extracted channels. The API is synchronous and returns the extracted data as a `*MessengerResponse`. This allows to make the API available status-go's RPC interface. The error case is a `map[string]*discord.ImportError` where each key is a file path of a JSON file that we tried to extract data from, and the value a `discord.ImportError` which holds an error message and an error code, allowing for distinguishing between "critical" errors and "non-critical" errors. 2. `Messenger.RequestExtractDiscordCategoriesAndChannels(filesToImport []string)` This is the asynchronous counterpart to `ExtractDiscordCategoriesAndChannels`. The reason this API has been added is because discord servers can have a lot of message and channel data, which causes `ExtractDiscordCategoriesAndChannels` to block the thread for too long, making apps potentially feel like they are stuck. This API runs inside a go routine, eventually calls `ExtractDiscordCategoriesAndChannels`, and then emits a newly introduced `DiscordCategoriesAndChannelsExtractedSignal` that clients can react to. Failure of extraction has to be determined by the `discord.ImportErrors` emitted by the signal. **A note about exported discord history files** We expect users to export their discord histories via the [DiscordChatExporter](https://github.com/Tyrrrz/DiscordChatExporter/wiki/GUI%2C-CLI-and-Formats-explained#exportguild) tool. The tool allows to export the data in different formats, such as JSON, HTML and CSV. We expect users to have their data exported as JSON. Closes: status-im/status-desktop#6690
0x-r4bbit
added a commit
to status-im/status-go
that referenced
this issue
Aug 4, 2022
As part of the new Discord <-> Status Community Import functionality, we're adding an API that extracts all discord categories and channels from a previously exported discord export file. These APIs can be used in clients to show the user what categories and channels will be imported later on. There are two APIs: 1. `Messenger.ExtractDiscordCategoriesAndChannels(filesToimport []string) (*MessengerResponse, map[string]*discord.ImportError)` This takes a list of exported discord export (JSON) files (typically one per channel), reads them, and extracts the categories and channels into dedicated data structures (`[]DiscordChannel` and `[]DiscordCategory`) It also returns the oldest message timestamp found in all extracted channels. The API is synchronous and returns the extracted data as a `*MessengerResponse`. This allows to make the API available status-go's RPC interface. The error case is a `map[string]*discord.ImportError` where each key is a file path of a JSON file that we tried to extract data from, and the value a `discord.ImportError` which holds an error message and an error code, allowing for distinguishing between "critical" errors and "non-critical" errors. 2. `Messenger.RequestExtractDiscordCategoriesAndChannels(filesToImport []string)` This is the asynchronous counterpart to `ExtractDiscordCategoriesAndChannels`. The reason this API has been added is because discord servers can have a lot of message and channel data, which causes `ExtractDiscordCategoriesAndChannels` to block the thread for too long, making apps potentially feel like they are stuck. This API runs inside a go routine, eventually calls `ExtractDiscordCategoriesAndChannels`, and then emits a newly introduced `DiscordCategoriesAndChannelsExtractedSignal` that clients can react to. Failure of extraction has to be determined by the `discord.ImportErrors` emitted by the signal. **A note about exported discord history files** We expect users to export their discord histories via the [DiscordChatExporter](https://github.com/Tyrrrz/DiscordChatExporter/wiki/GUI%2C-CLI-and-Formats-explained#exportguild) tool. The tool allows to export the data in different formats, such as JSON, HTML and CSV. We expect users to have their data exported as JSON. Closes: status-im/status-desktop#6690
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Right now there's an API pending in status-im/status-go#2752 that allows for extracting channels and categories from discord files.
We want to provide more information in case something went wrong during extraction. Namely, it's possible that:
Case 2) is not an error, but simply data we can't really deal with, still should probably treated as error from a UI perspective.
To make the above work, we need to extend the API to keep track of individual errors
We could also consider parallelizing the parsing work on a per channel file basis.
The message response needs to include errors that map each import file to the error
The text was updated successfully, but these errors were encountered: