Allow for transformation of messages from the upstream server before passing to clients

[thruflo]

I've implemented proxying in Elixir in front of Electric before. For example, in the gatekeeper auth example: https://github.com/electric-sql/electric/blob/main/examples/gatekeeper-auth/api/lib/api_web/controllers/proxy_controller.ex using Req in streaming mode.

I'm now working on a Phoenix.Sync app and have been eyeballing:

1. how best to implement a proxying controller
2. whether we can extract some logic to make this easier / build some proxy apis into Phoenix.Sync

My main question is how to get a "response" from Electric in embedded mode. I.e.: Phoenix.Sync has :embedded mode. When proxying, there isn't necessarily an upstream URL to proxy to.

Is there a way to make a request to the embedded Electric? If not, would exposing an internal shape and then proxying to that internal route be a workaround?

I suspect the answer to this would inform how to approach providing function or even macro APIs.

[thruflo]

Having discussed with @magnetised the best approach is to pass in a transform function that takes a change message and returns an array of 0 to N change messages (same as flat map).

This with sync and sync_render allows the developer to transform and filter the stream without any proxy or serialisation boilerplate.

[Jdyn]

Hello, how big of a lift is this? I can speak to a use case I have where I'd like to transform a path field on each of the elements in the sync_render query, to transform the path into the full image url before reaching the client

[magnetised]

@Jdyn working on it now.

[magnetised]

@Jdyn I have this working in #99 -- could you kick the tires for me?

{:phoenix_sync, github: "electric-sql/phoenix_sync", branch: "53-allow-for-transformation-of-messages-from-the-upstream-server-before-passing-to-clients"}

Based on your original Discord message you should alter your sync_render to look something like this:

sync_render(conn, params, Profile, replica: :full, transform: fn 
   %{"headers" => %{"operation" => "delete"}} = msg  ->
    msg
  msg ->
    Map.update!(msg, "value", &Map.update!(&1, "avatar", construct_full_url(&1))
end)

note the use of replica: :full to ensure that update messages contain the full row.

[Jdyn]

Damn it's working perfectly. Great work