Streaming support

[thenonameguy]

Hey @wilkerlucio! 👋

Would there be interest to support streaming responses in Pathom? I.e. a parser which emits "fulfilled" parts of the query in pieces, once they become available. GraphQL folks have it on their roadmaps, but I think it wouldn't be too hard to implement as a core-async channel that emits EDN datastructure's, that could be merged together iteratively.

This could reduce time-to-first data point significantly, which has benefits for both UI and inter-service scenarios.

[wilkerlucio]

Hello @thenonameguy :)

Yes! Streaming is something I get to think about from time to time, the possibilities are very exciting! That said, the implementation requires some time on thinking and experimenting. One big challenge to make this work is to decide when to flush; should it be at each property? each join? when dealing with lists, how much items should be flushed at a time? Then the client also needs to know whatever flush strategy the server dictates, its a lot of work :)

I would love to use this thread so we can discuss these types of decision. I personally don't have the need for this feature at this time, and this means I'm not gonna be working on it directly, but I would be happy to help with any info anyone needs in order to move this forward.

@thenonameguy do you have some practical use case for this, or it is more a wish thing? I ask that because real cases tend to be better to drive a solution than just us imagining use cases :)

[thenonameguy]

Awesome, great to hear that you thought about this.

It's mostly a wish thing currently, I just wanted to explore the cost/benefit ratio of different solutions. The original idea came up to me when I thought about moving parts of our internal architecture to HTTP/2 and making use of multiplexing/server push for non-web-browser contexts. I'll think about this a bit more and check back once I have some proposals.

[bgrabow]

I'm interested in discussing support for subscriptions. I have a system that has many child servers that are interconnected, and the behavior of the servers depends on some centralized configuration data in a parent server. As configuration data changes in the centralized spot, the child servers need to learn about the changes. However, some changes are only relevant to a subset of the child servers, and the configuration data is also graph-like and dynamic so it's not easy to write a single closed-form query of all the data a particular server cares about.

Pathom looks like it would provide the power needed to define the pull-based query I need, however without some form of subscription behavior, each server would need to poll the query to stay updated. Also, since the entire configuration data can be quite heavy, it would be highly inefficient to send the entire query result over the wire. Instead it would be preferable to send diffs between the current configuration state and the last configuration state the child server received.

I know this is only tangentially related to piecewise streaming a single response to a query, but it's at least in the same neighborhood.

@wilkerlucio Any thoughts on at least the subscription portion? The GraphQL folks have opted for supporting subscribing to a particular event key, rather than subscribing to a query result. In the Clojure world, 3DF is emerging as an option for reactive query subscriptions.
https://github.com/sixthnormal/clj-3df
https://www.youtube.com/watch?v=ZgqFlowyfTA
https://www.reddit.com/r/Clojure/comments/b46vl1/reactive_datalog_for_datomic/
Where do you see Pathom fitting in this reactive data flow space?

[wilkerlucio]

Hello Ben, thanks for the question. Currently, Pathom has nothing around subscriptions, but its mostly because I never had that use case, but I'm interested in ideas around it. Here are my thoughts around some options:

Thinking about the idea of GraphQL to listen to events, that seems easy enough to do. There are two main parts to it:

Server part:

1. First, we need to establish some connection between client/server, likely a WebSocket
2. Capture the watchers, so they know which clients are listening to which mutations
3. Write a plugin for Pathom, the plugin can wrap mutations and notify the listeners when its called

Client part:

1. Establish the connection
2. Subscribe to events, when an event comes, trigger some local mutation (in case you are using Fulcro).

I don't have experience with GraphQL subscriptions, so I'm not sure how they handle the message when they come, because I imagine that in some cases you may want to replace data, sometimes add, sometimes remove, so that needs some thinking around those cases, triggering a mutation requires more work, but it's generic enough to handle any case.

Another similar could be around listening to attributes, I can imagine it could work like the Fulcro refresh strategy. The connection part is the same as the previous, what it changes is what the user listens for, the subscription could be something like:

{::subscribe/attributes #{:customer/name :customer/dob}
 ::subscribe/ident [:customer/id 123]}

The ident is necessary (both in this case and the previous), otherwise, any change in the world would trigger that, and I can't see that scaling.

Them, the mutations could have something like:

{::pc/params [...]
 ::pc/output [...]
 ::subscribe/refresh #{:customer/name}
 ::subscribe/refresh-ident :customer/id}

The ::subscribe/refresh-ident can be used to match the ident (looking up that keyword on itself), and the ::subscribe/refresh to match on the attributes.

I guess that summarizes my thoughts on it, but please consider I never had this need, so most of it is speculative and would require proper experimenting to see how well they can work.

To me seems like you can do it without doing any changes to Pathom, most of Pathom is designed to be open (you can always add extra meta-data to resolvers or mutations, and handle those in plugins).

Does this helps to clarify your question around Pathom and subscriptions?

[bgrabow]

Thanks for your thoughts @wilkerlucio. I need to spend some time experimenting with pathom and fulcro before I can comment on your implementation ideas, but it sounds like the foundation is there for some subscription features.

I have tried to find documentation on how Fulcro handles refreshes like this but I haven't found it yet. Can you please point me in the right direction for a section of the documentation or maybe a video segment that describes it?

[alidcast]

This issue should be reopened, given that it's an important feature request that hasn't yet been implemented.

Looks to be one of the few features lacking to give Pathom feature parity with Apollo/Relay Graphql.