Keeping cross service calculated/aggregated data up-to-date in realtime app

[kshitizshankar]

Context: I've been trying to find examples/guides that might go over creating realtime charts and dashboards with FeathersJS to figure out how to make aggregated properties update in realtime with the rest of the app.

Consider a Trello board - each board has a bunch of lists and each list has a bunch of cards.
So, a simple setup with 3 services Boards, Lists, and Cards with a DB schema as follows

board: {_id, title}
lists: {_id, boardId, title}
cards: {_id, listId, title}

In my lists, I want to show a count of the number of cards present in that list - I added a fastJoin hook to calculate those counts but how do we keep these counts reactive? Right now, I have to write a bunch of hooks in cards service which detects if the listId is being changed and trigger an update on the old listid and new listid to recalculate the counts and broadcast them to the channels.

Is that how it's supposed to be done? - Feels like I might be doing something wrong here...

Building up from that, say I want to show realtime charts with aggregated values across a bunch of services - I can write a custom service to calculate those values and query that service from the client - BUT how do I "recalculate" them when one of the underlying values change? If I try to do it with the above approach - It just blows up with so much coupling between services that it becomes a nightmare to build and maintain.

What am I missing here?

Thanks!!

[miguelrk]

Hey @kshitizshankar! Somewhat late I guess but I also have a similar use case. Did you happen to solve this? I was thinking about creating a custom dataframe service, which would work sort of like a window above any other feathers service (e.g. readings, which are already stored). The dataframe service could then expose functionality to aggregate for instance. Else I have also thought about doing aggregations entirely client side, above any normal feathers query (e.g. GET /readings). However, doing it on the client might slow down performance and would miss several benefits of doing it on the server. Anyways, I would be grateful if you could share some light regarding this

[kshitizshankar]

@miguelrk Sorry I missed your comment. I haven't looked more into it yet but I am doing it on the front-end for now (which will stop being viable for me soon).

I guess the underlying problem to solve is how to make a reactive aggregate pipeline. I will look deeper into it soon - will add my findings here.

[kshitizshankar]

@miguelrk I am exploring an events-based approach. I realized that the problem can be looked at from a different perspective. Basically, the core thing I was struggling with was - depending on an action that was executed on a service - I want to execute actions on other services.

Currently, with the hooks based approach - I had to create hooks in the cards services itself that would handle what other services need to be updated when cards get updated. This results in too much coupling and becomes harder and harder to manage.

To solve that, I am looking into an events-based approach. Instead of hooks that do cross-service actions, I am listening for events on other services that want to "subscribe" to changes and perform actions.

So for the example, I mentioned, when the listId of a card is changed from L1 to L2 - I can create a change steam custom event that emits the before and after records (or use the Database Change Steam itself) that other services can subscribe to. The lists service can listen for this event from cards service and perform any actions that might be needed. The same thing can be extended to say removed event (if other services need to clean up things if a card is removed successfully)

For the reactive aggregates - The same thing can be extended and the custom service can listen to events from the related services, figure out what needs to be changed and emit the changed values to the client.

I am still exploring this - but so far it has been working quite nicely.

[J3m5]

I really like the way you think about it @kshitizshankar , this should to bee explored further.

[kshitizshankar]

@J3m5 Awesome! Makes me feel a lot better about this approach lol

The only thing that came in the way of implementing this was - since we don't get the full hook context in custom events, I had to "hack" around it by sending additional data that is needed by the publishing logic to figure out the right channels (the user id and/or another identifier). Not sure if it breaks anything from the feathers perspective - will continue testing it out and see if I run into any issues.

[DaddyWarbucks]

I am interested in this as well. I have done some little experiments over at https://github.com/DaddyWarbucks/test-feathers-client-joins. I found the client side joins to have pretty impressive performance, even on REST.

I am not sure if you are using react, but I have been interested in https://github.com/tannerlinsley/react-query lately. I think it will pair really well with a feathers client in general. Furthermore, I think you could write some custom hooks that do all of the "aggregation" for you there at the view level (instead of the service level).

Maybe something like

const listId = 1;
const { loading, data: list } = useFeathers(['lists', listId, { include: ['cards', ...] }])

// Get the list
const { data: list } = useQuery([listId], app.service('lists').get(listId));

// Rather than doing this next bit manually, you would read from the parent hook query `include`
// and loop over those properties to know what joins/aggregations to include.  This would
// be pretty performant with a batchLoader and along with react-query's caching and such
// it should be pretty decent. You would also need to delete this `include` prop from the query
// before going to the actual service because its not a real query, just a way we are informing
// this next logic to aggregate/join our data the same way we might use it to inform fastJoin
// to do the joins at the service level. So we essentially decouple the joining from the service
// level and couple it at the view level...view level fastJoin basically.

// Then get the lists's cards
const { isIdle, data: cards } = useQuery(
  ['cards', list?.id],
  app.service('lists').find({ query: { listId: list.id } }),
  {
    // `list` would be `null` at first (falsy),
    // so the query will not execute until the list exists
    enabled: list
  }
);

// Once all joins/aggregations are completed, returned the final loading state and data

Then you just bust the cache of individual service caches and the aggregated views will rerender.

 app.service('cards').on('created', data => {
  // I am not sure the actual syntax here, but I know you can bust the cards service cache.
  // So by busting this cache, our aggregated view will refresh its cards piece and rerender
  reactQueryCache.bust(['cards'])
})

I haven't test this idea at all. But I have been thinking about it and I stumbled across this issue and thought I'd share.

[kshitizshankar]

@DaddyWarbucks I am not that familiar with React - been using Vue and Feathers Vuex (which has direct integration to feathers API). The problem for me was not regarding joining two or more records - but more about computed properties (aggregated data).

Consider the Gmail app - you have a count that shows the number of unread emails in your inbox. Whenever you mark an email(s) as read or unread, the count updates. To be able to do that, some piece of code needs to know that whenever an email's read field is changed - the count for that inbox needs to be updated. For this - my approach was - to create a separate service called "inbox-stats" - that listens to the "patched, removed" events from "emails" service and calculates and sends the updated stats if needed.

For populating data and relationships - I am using https://github.com/marshallswain/feathers-graph-populate which hooks to the Feathers Vuex. Populating on the client as you described works as well and might even be better depending on the use case (feathersjs-ecosystem/feathers-vuex#397 (comment))

[sebastien-prudhomme]

@kshitizshankar Have some similar need. I'm also using feathers-vuex and feathers-graph-populate. Can you share some code or pseudo-code of your inbox-stats service? I can't figure out how you refresh your data. Thanks.

[J3m5]

@kshitizshankar This feature can be done in a simpler way.
As you said, you need a "read" property on the model, when it's read your toggle it to true.
Then, you need to query the count of mails (in this example) that has the property "read" set to true and probably the id of the user that the mail belongs to.
And use the findGetter to get the reactive count of read mails.
That way you'll get the number of mails that have been read and it will be updated with the events.