Publish flow button

[johnrees]

Add a 'publish button' in the editor

Problem

We need a way to be able to edit flows without the content changes going live immediately. We could edit them in staging and then migrate the data to production but this adds a lot of complexity with ensuring that the code is going to be compatible with the content.

Potential solution

If there was a publish button, it would allow admins to edit content and these new changes would not be visible in the frontend until someone clicks publish. The downside is this behaves a bit like trying to have multiple people edit and save the same textfile on your desktop, rather than something more flexible like branches in git. It might be enough for now though?

We could add two fields to flows, named published_at:timestamp and published_data:jsonb

Move this dataMerged* function out of the editor.planx.uk codebase and into an express endpoint e.g. flows/:flowId/publish in api.planx.uk

* .dataMerged(flowId) loads a flow, searches for any external portals inside it, then loads those flows too. It does this recursively until it's loaded all of the sub-flows. It then replaces all of the external portal nodes with the actual flows, flattening everything into a single flow object.

How it works

At 2021-06-14T11:16:00, an editor is working on flow xxx-xxx-xxx in the editor

1. they click the publish button
2. this endpoint is called https://api.editor.planx.uk/flows/xxx-xxx-xxx/publish
3. the flattened flow JSON response is stored in published_data and published_at is set to 2021-06-14T11:16:00

The editor can now show "published 0 minutes ago"... or whatever! (not important RN)

The frontend would simply request flows.published_data from hasura, and not need to worry about doing any recursive flattening operations like it does now.

Thoughts

publishing sub flows too

Right now this behaves like a snapshot operation of whatever the current state of a flow is and all of its subflows. I think that needing to open subflows and publish them individually too would become too much a mental burden on the editors. Very happy to discuss if you think I'm wrong here though.

dedicated table

These publish operations should probably be stored in a separate table rather being extra fields in flows e.g. published_flows

Then you'd have the has_one relationship flows.published_flow_id -> published_flows.id, and in hasura you could still call flow.published_data, which would return the associated published_flow's data JSONB field.

Then you'd have the belongs_to relationship published_flows.flow_id -> flows.id, and in hasura you could still call flow.published_flow.data, which would return the associated published_flow's data JSONB field.

It would give you the opportunity to add more fields like publisher_id (user_id), comments (commit notes), flow_version etc.

Explained further in #482 (reply in thread)

storing the published data elsewhere?

this published_data data could become quite large, spitballing here but ~1-5MB? You could remove it and just store the published_at field.

This is probably not necessary now but then you could cache the initial response of the /publish endpoint with a CDN like cloudflare, using a timestamp in the URL.

When publishing you'd call this instead, notice the added timestamp param in the url

https://api.editor.planx.uk/flows/xxx-xxx-xxx/publish/2021-06-14T11:16:00

When loading the flow in the frontend you'd know the flow.id (xxx-xxx-xxx) and flow.published_at (2021-06-14T11:16:00) so you could recreate the same URL and call it, but this time instead of publishing the flow, it would be loading a static JSON response directly from the CDN cache.

or, store s3 URL of json file

Alternatively instead of storing published_data:jsonb you could save the response to a file in s3 and either make the filename deterministic based on the timestamp like the CDN method above

or store the url to that file in the database as something like published_url:text, then when you load the frontend you'd query what flow.published_url is and load that file directly from s3.

[johnrees]

I'm pleasantly surprised that the entire productions flows table is currently 4.7MB in a csv. It should be smaller than that with binary json fields.

[jessicamcinchak]

this overall sounds good, and feasible!!

i'm wondering if it's worth prioritizing a third column on flows table in this potential first iteration - like published_by user - since we're currently used to being able to see who last edited the flow?

the snapshot approach for portals makes a lot of sense to me. i think the alternative assumption that users publish portals/subflows individually would only get more complicated down the line if we introduced more roles & permissions - like, what if you want to publish your local council's flow, but it relies on a subflow that is published in another team that you can't access individually? snapshotting seems like a good way to bypass those questions for now.

in order to snapshot accurately, i think the new columns will need default/initial values, is that assumed whne you mention adding new fields? let's say i'm editing a parent flow, and it has a single portal, and that portal is also being edited by someone else. when i "publish" the parent, it should snapshot the last published version of the portal, right? This way we don't capture any edits-in-progress in the subflow. I think this means that the new fields should be initially populated with the current flow values & now() so that the first user-initiated publish button click queries them. make sense?

i think there's also postgres extensions for jsonb compression that might be quick wins and require minimal extra config, but would have to refresh/research a bit in that area.

[johnrees]

i think there's also postgres extensions for jsonb compression that might be quick wins and require minimal extra config, but would have to refresh/research a bit in that area.

thankfully the JSON isn't toooo big atm I think we can get away without needing to compress it. There are options in future to use something like messagepack but I suspect we might have a better approach for storing and accessing flows before something like that would be necessary.

[johnrees]

in order to snapshot accurately, i think the new columns will need default/initial values, is that assumed whne you mention adding new fields? let's say i'm editing a parent flow, and it has a single portal, and that portal is also being edited by someone else. when i "publish" the parent, it should snapshot the last published version of the portal, right? This way we don't capture any edits-in-progress in the subflow. I think this means that the new fields should be initially populated with the current flow values & now() so that the first user-initiated publish button click queries them. make sense?

I was hoping we might be able to get away with not needing to remember to publish subflows for v1, but perhaps it's important. As there are so few editors right now it might be a shortcut that we can take. I can understand the need for it I'm just a bit scared that there might be a lot of extra thought (and button clicks) required from editors if it's a requirement from the beginning.

[johnrees]

Another reason to store flow_version would be that the publish button could be disabled if you're editing flow: xxx-xxx-xxx

and

flow(id=xxx-xxx-xxx).published_flow.flow_version === flow(id=xxx-xxx-xxx).flow_version

because there are no changes to publish

[johnrees]

More pseudocode for perhaps a more robust way to check if the publish button should be enabled for a flow

publishButton.enabled = flow.operations.last.created_at > flow.published_flow.created_at

[jessicamcinchak]

rollback benefits of a separate table seem very worthwhile!

still wrapping my head around flows & subflows too, but would definitley be nice if publishing all levels isn't necessary in this first go.

[gunar]

We'll need two "preview" urls. One for previewing editor changes and one for end users to see published flows.

[jessicamcinchak]

Rebooting this discusson thread in light of this Trello ticket: https://trello.com/c/lSkxf408/1572-decide-which-functionality-is-still-outstanding-to-be-able-to-enable-the-publish-button-in-production

Recap of how publish works now (disabled on .uk domain):

• editor: click "publish" button, we display a confirmation message with the number of changed nodes on success
• api: one endpoint that does a serverside diff & graphql mutation to create a new record if there have been changes

Proposal for production mvp:

• editor: click "ready to publish" button, we display a list of changed nodes (maybe as direct links, or highlight nodes if not in portals?) and a short text input to describe your changes, click "confirm", we display same confirmation mesage as now
  • the "ready to publish" buton will always be enabled, we'll do the diff after clicking it
  • the confirmation button will only be enbaled if there are altered nodes
• api: same logic as now but split into two endpoints - one to do the diff and return alteredNodes (this maintains ease/pros of serverside diffing as sarah had smartly outlined), and a second endpoint to do the actual graphql mutation when you click "confirm"
• hasura: add text column description to published_flows table for more human-readable changelog

Sandbox for testing against full-sized flow content at #683 // https://683.planx.pizza

Any other ideas or current painpoints we want to smooth out before enabling on prod that this doesn't address yet?

[jessicamcinchak]

talked about this together on dev call, going to move forward with it & push updates to PR #683 for further review/tweaking! ⏩

one additional point that came up to include: save the published flow id at the time of beginning an application in the passport, and in planx debug data in Send component, for future save & return efforts.