Optional secondary index?

[archonic]

I'm enjoying simple feed quite a bit so far. Right now I've created a new feed for each type of activity. I'm uncertain if that's a good idea but here's my reasoning. I need to be able to group together certain activity types by regions of time. Instead of a feed indexed only by time, I could primarily index by time but group similar activities to say things like "5 new comments on resource 'example'" where each new comment is an event. I think that's a pretty common desire for an activity feed.

I could query for activities in the last 30 days and then do processing on the results by parsing the value, but I think that's quite inefficient. It would be better to query the last 30 days and use group_date, then merge various feeds with array merge. In order to group by activity type, I need some kind of secondary index. Right now I'm just using different feeds as my secondary index.

Redis is capable of secondary indexes and offering an optional secondary index would really clean up this use case. Do you think that's within scope for this gem?

[archonic]

Speaking of indexes, I'm unsure how to query the primary index. Can I not specify a time range of activities to fetch?

[kigster]

Hey, @archonic! Thanks for the kind words and trying out simple feed! I’d like to help you as much as I can, but let me think about your question and I’ll post my thoughts here later today.

[kigster]

Ok here are my initial thoughts.

Time-based pagination

Right now simple feed does not support pagination by a time range, such as “return events that are between 30 and 15 days old”. However, there is an API call for fetching the "last N days" of events:

# Returns un-paginated list of all items since a timestamp provided in :since argument.
@activity.fetch(since: nil, reset_last_read: false)
# Note that the argument can also be a symbol :unread indicating that all unread items 
# should be returned.

Would that work for you, or do you need to go beyond the first page of time-sorted events?

Grouping similar events

Another feature that’s not currently implemented. However, if you were to store your published events as described on this wiki page, you could grab the event IDs from simple feed and then use the group_date gem to fetch the events (with the matching ids) from the database while automatically grouping them.

I believe that storing events in a table could address some of these requirements.

Please let me know if this answers your issue sufficiently, and it's ok to close.

[archonic]

Hey, sorry about the super late reply.

Would that work for you, or do you need to go beyond the first page of time-sorted events?

My goal is to add temporal structure and context to an activity feed that would otherwise just be a list. It would be 3 columns, 2 of which are navigation. The left most column would be years, the middle column would be months and the right most column would be the activity feed for the specified month. It would be like pagination but each page is 1 month. I would need to specify a time range on the primary index to do that.

However, if you were to store your published events as described on this wiki page, you could grab the event IDs from simple feed and then use the group_date gem to fetch the events (with the matching ids) from the database while automatically grouping them.

I would still need some kind of secondary index. I suppose I'm after kind of a 2 dimensional activity feed, where events pertaining to one resource can be appended all of that day's activity for that resource. Github's recently rebuilt activity feed does this for commits:

Do you have an idea of how to achieve that with simple-feed, or would it take additional development?

[rubyrider]

For aggregation based feeds redis could be expensive because there could be so many things to keep in the memory. So that would be nice to have another level of abstract layer to implement other engines like ES, Casandra etc. I am doing some studies and interested to implemented the ES version though.

[kigster]

That would be quite amazing to have an ES adapter. If you really wanna do this I could move the adapters into their own gem.

[rubyrider]

Hi @kigster ,
Sorry I was quite busy for quite long time. I am interested to start to see how it works with ES.

[kigster]

Don’t let me stop you :)

There is a shared rspec example that should pass against any new provider to guarantee consistent results. So all you have to do is implement a new provider and make sure it’s passing that spec.

[kigster]

Closing this issue.

If you'd like to do ElasticSearch, @rubyrider, please create a new issue and call it something like ElasticSearch SimpleFeed Provider and label it as a feature.

Thanks!