bot: a shared memory for recent lines

[Exirel]

Requested Feature

A built-in, official API, to store and retrieve recent lines:

• short-term memory ("recent" lines)
• in-memory only
• store PreTrigger objects
• mostly PRIVMSGs and NOTICEs
• per-channel and global configuration
• configurable by bot owner if necessary to limit memory consumption (might be useful for instance with many/huge channels)
• sane default (to be defined?)
• search API (destination, origin, maybe a regex?)

Problems Solved

As of today, a plugin that wants to have a list of recent lines has to create its own data structure and have a parse all rule (.*), and do all the work itself. For example, the find built-in plugin does that, as well as the sopel-spongemock plugin.

By having an official, built-in, public API to store and retrieve PreTrigger for recent lines per channel, we would allow plugin authors to concentrate on what they want to do, and not the gory details of registering everything that happens.

This would also (slightly) reduce the memory consumption of Sopel and its plugins, as it would de-duplicate what is stored by plugins.

Alternatives

No response

Notes

23:29:15 <+mal> this should probably start at "how will this feature be used" and built to support that, not "this is the easiest implementation"

Important Reminder that we must have a clear list of use-cases before diving into any API design.

23:39:04 <+dgw> SnoopJ, length of the queue shouldn't be a user's problem, but a plugin developer's problem
23:39:49 <+SnoopJ> dgw, I can sorta see how a plugin might be able to declare that it needs at least such-and-such length of history, and you consider every plugin's requests when settling on a value, but it's definitely something that merits more thought.
23:39:49 <+dgw> and the current way of doing things (make your own deque or whatever) perfectly accomplishes that, at the cost of duplicating storage between plugins with different needs
23:40:01 <+SnoopJ> yea
23:40:10 <+SnoopJ> maybe what we really want is a central store that de-dupes :P
23:40:22 <+dgw> the store could decide to drop a line based on whether it still falls within the union of all plugin constraints
23:40:37 <+dgw> I thought about bringing that up, then we got distracted by the long-term-storage red herring
23:40:42 <+SnoopJ> my concern there is that one greedy plugin can drastically shoot up the storage
23:40:50 <+dgw> so you limit it
23:40:53 <+SnoopJ> but there's a lot of texture there
23:41:01 <+dgw> "you can have as many lines as you want, as long as it's <= 100" for example

Good point here by @dgw and @SnoopJ about the idea that Sopel should provide a way for plugins to tell "I need 100 recent lines per channel", or "I need only one per nick". Not every use-case is possible, so at some point we'll have to compromise and say "yeah, we store 10 times too many data for this use-case, but it's the right amount for this one", etc.

Also, we must take care of not allowing a plugin to request more than the bot owner is confident with.

It's OK, tradeoff will be made!

23:41:42 <+SnoopJ> but truthfully, users probably don't want the raw lines either
23:41:51 <+SnoopJ> or uh, the not-quite-raw-but-the-message-text lines
23:41:59 <+SnoopJ> plugin authors care about Rules

Alongside the PreTrigger, having some metadata with it would be nice (like "this triggered these rules").

23:43:28 <+dgw> persistence came up later, and I wasn't clear on whether it was the same API or not

We more or less all agree that we don't want to deal with a long-term memory of lines in Sopel. This is out of scope of this feature, and so is persistent storage for now.