Grouping in Email Digest Agent

[a10kiloham]

I've got a bunch of forum threads I want to bundle up and send to myself periodically with Email Digest Agent. It seems there's a little bit of grouping but order is hard coded as ASC by event.id and there's no sub grouping. I'd like to extend the agent by adding a group_by key that would allow a grouping in the digest if the group_by key is present and then perform a sort by this key. There should probably also be a variable sort_by for ASC or DESC at a minimum. Thoughts on this structure?

[dsander]

Being about to sort the event sounds like a useful addition, you probably could utilize the SortableEvents concern. How do you imagine the sub grouping to work, should it be a replacement to the build-in grouping keys or be performed before/after that?

[a10kiloham]

Is the built in events_order already functioning in this agent? If so then this is probably redundant. I didn't see any sign of it and trying it, I can't really tell if it's working or not.

[dsander]

I don't think the concern is already in the EmailDigestAgent.

[a10kiloham]

I've been thinking about this for a while now over the holidays. I think the most elegant solution is sorting as in the Data Output Agent where keys for sorting can be input and then sorted. So in the case I've got in mind, I can sort by {{ event.title }} then {{ event.id }} which would group the payloads under each title and they would be in normal sort by id inside that grouping.

However I can't really figure out how to dig in deeper as the app is a bit more complex than my skills allow for. How can I even see the methods available for the various objects? Normally I'd just the .methods operator to do introspection but that doesn't seem possible to do in an app like this (or else I'm just too much of a noob). Even at the most basic level I'd like to try something like changing
payloads = received_events.reorder("events.id ASC").where(id: self.memory['events']).pluck(:payload).to_a
to reorder based on the event.title but it's not clear where/how I can access the individual components of each event. :/

[dsander]

The payloads variable is an Array of event payloads which are always a Hash. You can call sort_by on that array: ....pluck(:payload).sort_by {|p| p['title'] }

[a10kiloham]

Cool thanks. will give it a try this week to start extending

…

On Fri, Jan 27, 2017 at 8:24 PM Dominik Sander ***@***.***> wrote: The payloads variable is an Array of event payloads which are always a Hash. You can call sort_by <https://ruby-doc.org/core-2.2.0/Enumerable.html#method-i-sort_by> on that array: ....pluck(:payload).sort_by {|p| p['title'] } — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#1775 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AASBwYuLZSxwxqlYXG0gu3nHmYVYeRLzks5rWlKLgaJpZM4KoUsc> .

[nogre]

@bobbysteel I have a similar use case. My solution was to use a Digest Agent for each source and have its well-formatted output go to a Credential via a JavaScript agent:

Agent.receive = function() {
  var events = this.incomingEvents();
  for(var i = 0; i < events.length; i++) {
    this.credential('_src1', events[i].payload.message);
  }
}

Later, when I want the aggregated digest, I have a scheduled JavaScript Agent send an event to an Event Formatting Agent, with contents like this:

"instructions": {
    "message": "{% credential _src1 %} {% credential _src2 %} {% credential _src3 %} "
  },
...

This message will be a nicely formatted digest of events, which you can then email.

[a10kiloham]

good idea, thanks for this!

[a10kiloham]

Digging this up as I have a few spare cycles now.
The credential method seems to not work as the number of events can be in the dozens and I can't assume a defined # of credentials I'd have to statically define.
For the sort_by @dsander how would you envision that in the agent? On line 50 of the agent I guess I'd add another if there and allow a sort by the input sort_item selected?

[a10kiloham]

@nogre is this storing the events as an Array? it's hard to tell but it looks like the Credential is holding multiple messages?

[dsander]

@a10kiloham Changing this line to

payloads = received_events.reorder("events.id ASC").where(id: self.memory['events']).pluck(:payload).to_a.sort_by {|p| p['title'] }

should work. title is a key in the payload of your Events.

You probably also could use a DigestAgent, do the reordering and formatting in a JavascriptAgent and send the result to a EmailAgent.

[a10kiloham]

Thanks @dsander
For the DigestAgent, how would you deal with a multidimensional array? It seems to only accept one entry, message, so in my case I have from a forum, multiple threads each with multiple posts. I can collect each post individually in the DigestAgent but there's no way I can see to group by any other key - is there a way to access the existing DigestAgent data from JS and to store other keys in the input event to the DigestAgent? I suppose I could concat the thread identifier into each message, but can't see how to edit that in JS before emitting.

[nogre]

I haven't looked at this in forever and don't remember exactly what's going on, but what is happening in one of my Javascript Agents is:

Agent.receive = function() {
var events = this.incomingEvents();
for(var i = 0; i < events.length; i++) {
var ii = events[i].payload.message.indexOf('>');
this.credential('_'+events[i].payload.message.slice(1, ii), events[i].payload.message.slice(ii+1));
}
}

Basically it as you thought, the identifier is concatenated onto the front of the message with a separator > and then it is split later.

As for multiple array entries, I have other Digest Agents that accept the new entries and clear when the digest is created. So if forum A has multiple posts in the allotted time, then those events get sent to the "forum_A" Digest Agent. Then something else takes the data from all those Digest Agents (forum_A, forum_B, etc.) and makes sense of that. There are filters and formatters somewhere along the line there, too.

[a10kiloham]

Thanks. I've made it work for now by doing this and adding the keys at the front so I can add a sort | filter to the Digest Agent and I can use that as my grouping.