Snapshots in Pipelines

[Pezmc]

• Re-order methods in pipeline file
• Extract snapshot creation logic into a service
• Deployment of pipeline stages using snapshots

[hardillb]

As mentioned when discussing:

• What happens with credentials as part of the deploy, they should probably be copied over along with the settings and flows. Credential values can be pulled from Env Vars so if done right should "work"

[hardillb]

Looks like credentials are getting copied if I've read this right

[hardillb]

Notes from testing

• - Ticking the "deploy to device" box when creating a step doesn't appear to set the flag in the db, editing the step does set the flag
• - Not seeing the flow be copied to the second stage
• - Not seeing the flow being sent to the device (after flag fixed by editing stage)
• - Last Deployed time appears to never be updated (even with page refresh)

[Pezmc]

@hardillb / @knolleary Can you start on review of this while I'm finishing the unit test coverage?

[hardillb]

@Pezmc will do

[knolleary]

This is all looking good - although we have identified a blocker with the current implementation that will require more time than available to get this in to 1.9 tomorrow.

When promoting between stages, the snapshot is created at the source, copied to target and then applied - all good. However, the snapshot overwrites the target's env var values - where as it should be merging them.

So the step to create the target's copy of the snapshot needs to merge in the target's env vars at that point in time.

A secondary issue I've just spotted, but needs a bit more diagnosis work; I think the credentials object needs to be re-encrypted for the target instance's key, otherwise it cannot decrypt properly.

[Pezmc]

@knolleary @hardillb Theoretically this is ready for review and to be tested locally, but the tests are hanging for a reason I haven't fully explained yet. Please validate it's behaving as you'd expect locally while I get to the bottom of the hanging.

[knolleary]

Looking good to me locally.

Created a flow using a credential-type env var.

Deployed through the pipeline to another instance, which had access to the env var value.

Confirmed the stored versions (ie encrypted) of the credentials were different - so re-encryption had occurred.

Just want to figure out why the test runs are running away in GH Actions.

[knolleary]

I've just merged main back to this branch in a hope to get the tests behaving.

[Pezmc]

@knolleary Still hanging and I really can't explain why... the tests should all have a 5000ms timeout set on them...

[knolleary]

It feels like the step of the action is just never starting, so not getting as far as have any tests to timeout.

[Pezmc]

@hardillb @knolleary Tests fixed, PR ready for re-review.