Have a command to have the bot rename a thread based on distinctive words

[jkomoros]

For example, use TFIDF to pick out diistinctive words to rename a thread distinctively.

There could be a slash command that could be run in the thread to update it.

Threads created by forking (see #34) could also automatically be named with this.

Generating a TFIDF index is hard for the whole corpus. We could use a TFIDF map extractec from https://thecompendium.cards as a default to start from. And use the same stemming.

[jkomoros]

• Strip out URLs
• Strip out markdown syntax
• Strip out mentions
• Stem
• Test
• Rebuild by fetching all messages
• Consider having a "document" be a channel's messages, and compare TFIDF compared to other whole channels (currently TFIDF is all done at the level of individual message)
• OPs and messages with multiple of the important/significant emjois are counted more times when computing TFIDF for channel (store with a Multiplier)
• Store an actual discorgo.Message in the JSON
• JSON should be a .data object
• Switch to a flat messageID -> word count, and a channelID -> []messageID
• When a message is edited or an emoji reaction happens, reindex the message
• Every 15 minutes, if the IDfIndex for a given guild has had changed, persist to disk (in case the bot doesn't exit cleanly)
• Store out IDF in a JSON file and reimport on bot boot (embed a constant for json version in the struct and check it's the same on boot, if not, discard)
• in JSON consider having the channels be an array of messages in order (especially if this cache grows to be the archival format for old channels)... No need, the IDs since they're snowflakes can already be sorted based on ID to have the proper order
• If a message comes in during boot it might interfere with the fetching of old messages
• Given a TFIDF, print out the suggested name (the top n items of idf)
• restem TopWords
• BUG: IDFs loaded up from json when autosaved save to a nil location
• have TopWords also automatically calculate a cutoff of words, picking UP TO count
• Only rebuild the missing messages when bot loaded with existing JSON (as in, look to see if we have messages already in the cache, but missed some new ones, and fetch the ones that came in while the bot was not running)
• Have an /auto-title-thread command
• Switch to the batch-idf method as described in a comment below. New branch on top of idf?
• When processing a message, if we're close to the max messages size, then don't store messages that would be deleted in the next batch. (Store the lowest known ID at all times and compare the ID to that, maybe?)
• Respond with a pending interaction response if IDF has to be rebuilt (it could take a long time) (although it should have been built at server start)
• Handle races where an IDF is asked for while one for that guild is being rebuilt
• Don't create guild-1 idf in the test suite
• Fix permissions issues where messages in a private channel are forbidden
• Remove stop words. They still show up in production!
• some restemmed words have punctuation
• Rebuild IDF every day or so automatically
• delete the channelsForMessage when deleting a message to make space. Need to store channelID in the message struct.
• A flag to allow to easily use the snapshot idf cache
• When rebuilding a stale IDF cache, only fetch messages that are more recent than the timestamp (might require parsing the snowflake IDs for timestamps)
• Allow configuring which emojis count as important
• OP messages get a TFIDF multiplier that is higher, too
• A way to accept the suggestion from the auto thread title command (currently it just suggests one)
• when suggesting multiple titles, respond with multiple alts as buttons to rename the thread

[jkomoros]

See also #10 which captures some of the TFIDF stuff

[jkomoros]

Actually, all of this behavior about expiring old messages from the cache implies I'm making this way too hard. Especially if we're moving to #41 where there's a more long-term backup data store.

IDF rarely changes (especially for large guilds). Instead of keeping it live to generate it on demand, just do a batch process every so often (every day?) where it fetches all messages and calculates the IDF and saves that to disk in a cache, along with a timestamp so it can figure out when loading back up if it needs to regenerate it. And then when asking for a channel's suggested title, live fetch the messages, calculate the TFIDF, and return that.