Snapshot Download Feature

[MarianRaphael]

Epic

#3617

Description

As a: FlowFuse User

I want: the possibility to download my snapshots locally

So that: I can save them locally or upload them to another FlowFuse system for seamless continuity of my workflow across different environments or for backup purposes.

Which customers would this be availble to

Everyone - CE/Starter/Team/Enterprise

Acceptance Criteria

• Add a "Download Snapshot" button adjacent to the "Download package.json" button, ensuring it is easily accessible and intuitive for users to perform snapshot downloads.
• Owner can perform a download
• Member can perform a download
• Viewer cannot perform a download
• Audit log entry is present and correct
• Downloaded snapshot does NOT contain the secret

Have you provided an initial effort estimate for this issue?

I have provided an initial effort estimate

[Steve-Mcl]

@MarianRaphael @knolleary

The current getSnapshot frontend API only returns

{
  "id": "xxxxx",
  "name": "Auto Snapshot - 2024-04-22 21:09:30",
  "description": "Instance Auto Snapshot taken following a Full deployment",
  "createdAt": "2024-04-22T20:09:30.872Z",
  "updatedAt": "2024-04-22T20:09:30.872Z",
  "modules": {
    "node-red": "3.1.9",
    "@flowfuse/nr-project-nodes": "0.6.1",
    "@flowfuse/nr-file-nodes": "0.0.5",
    "node-red-contrib-mcprotocol": "1.2.1",
    "node-red-contrib-buffer-parser": "3.2.2",
    "@flowfuse/node-red-dashboard": "1.5.1"
  },
  "ownerType": "instance",
  "createdSince": "6 days ago",
  "updatedSince": "6 days ago"
}

I assume since the idea is to be able to upload this to another FF platform there are other elements of the ProjectSnapshots table will need to be included in the snapshot export.

Fields of ProjectSnapshots table:

• Name, description, flows are a given
• What about settings.env (potentially sensitive info in here)
  • members can already see env vars and this feature is scoped to members and above, so cannot immediately think why not.
• What about credentialsSecret? Clearly this will be required but should it be recomputed upon export?
• ProjectId and DeviceId will have little meaning to the other system - I suppose these should be null/excluded (or are there other thoughts/plans around exporting the associated device/project too)?
  • EDIT: I think they should be included in-case SS is uploaded back to source platform. Other platforms can discard these. That though means we may need to include the SOURCE PLATFORM to differentiate.

[MarianRaphael]

What about settings.env

Yes, should be included

should it be recomputed upon export?

Sounds logical, I'll add it here: #3628

ProjectId and DeviceId will have little meaning to the other system

Yes

[Steve-Mcl]

ProjectId and DeviceId will have little meaning to the other system

Yes

Yes include and set null on upload to OTHER FF platform, use on SAME FF platform?
OR
Yes exclude/null the fields always?
OR
????

should it be recomputed upon export?

Sounds logical, I'll add it here: #3628

Shouldnt that be an acceptance criteria of this issue Marian?
This is the Issue for "exporting" a snapshot, #3628 is about importing a snapshot.

[knolleary]

@Steve-Mcl I don't think the export should include device/application/instance ids - they are meaningless when importing.

We already have an established pattern for allowing the user to pick what get includes (env keys only for eg) - thinking of the 'copy project' functionality. My default position would be for this API to be consistent with that, unless there is reason to not be.

I'm not sure about where credentialSecret should sit. We shouldn't export the existing value, so a new secret should be used and the credentials re-encrypted.

I suggest you look at the import API we already have to see what that expects to receive - as this export must be compatible with that.

[Steve-Mcl]

@knolleary ah, I think my interpretation of the 2 stories "Snapshot download feature" and "Enable Snapshot upload functionality" are too literal.

Am I correct in thinking (based on your previous comment) this is less about downloading and uploading snapshots (as snapshots) and more about exporting and importing an instance or device from a point in time snapshot? i.e. this import action will NOT create a usable snapshot rather create the source instance/device?

Is that correct?

@MarianRaphael ^

[knolleary]

Is that correct?

I don't think so.

The goal here is to allow a user to download a snapshot locally. That should be a complete artefact that can then be reuploaded elsewhere.

[Steve-Mcl]

hmmm, now i am confused.

So what is the end goal?
To be able to upload a snapshot (to the snapshots list? of the selected instance/device?) that can be later deployed
OR
To generate an instance/device from a point in time snapshot?

I ask because the only existing import API I am aware of is for importing and creating an instance!?

Please point me in the right direction.

[knolleary]

To generate an instance/device from a point in time snapshot?

I'm not clear what you mean here.

We already have an export snapshot api that, I think, does everything we need - but it is scoped to an instance: https://app.flowfuse.com/api/static/index.html#/Snapshots/post_api_v1_projects__instanceId__snapshots__snapshotId__export

I think we need an API that is not tied to a specific instance to allow export of a snapshot that works in all cases.

The acceptance criteria for this issue is to have a UI driven means to operate that API and get the flows downloaded.

The next step would then be to support uploading that same blob in the UI in some way. That will need some additional thought in terms of API design and scoping.

[Steve-Mcl]

To generate an instance/device from a point in time snapshot?

I'm not clear what you mean here.

The next step would then be to support uploading that same blob in the UI in some way.

I think this is the piece of information I am missing. To help me understand the purpose of downloading a snapshot (and what (if any) other pieces are necessary in addition to the /export API), I would like some vision of how it is to be used.

To clarify:

When the next piece of this puzzle is implemented #3628, where does the exported snapshot go?

• into the snapshots list?
  • of the selected instance/device?
  • somewhere globally?
• or does it create the original "thing" (instance/device) that the snapshot was taken from?
• or does it create "a thing" (instance/device) depending on where it is executed
  • e.g. upload snapshot on the devices page, generates a device
  • e.g. upload snapshot on the application view, generates an instance?

My apologies if i am missing something really obvious, i've read this issue and the related issue several times and it is unclear to me.

@MarianRaphael ^ please advise.

[knolleary]

The import of a snapshot will be into an existing 'thing'. This is not about creating new 'things'.

We already have an API to import a new snapshot for an instance. We will likely want similar APIs for the other types of thing that can 'own' a snapshot. But we need to figure that out properly as part of the Import story. But none of that detail should impact the export side of it.

[Steve-Mcl]

Follow-up to discussion in the Eng meeting:

Current implementation

In my current dev branch, I have added credentialSecret to the API schema and it is return inside the snapshot data (since it is needed for when it is imported)

Being present in the API alongside the the encrypted credentials is not ideal.

Possible approach - send the new secret in the request

Request user to input a new secret (on the UI) and deliver the snapshot without the secret
This could also be a freeform text entry that permits the user to enter own secret.
Additionally (or alternatively), we could auto generated a secret and display it as a one time thing for the user to make note of.
This secret would be POSTed to the /export API, FF would re-encrypt the credentials and the response (the snapshot) would NOT include the secret.

This approach means the user would need to keep a record of the the secret and provide it on the UI (or API call) when importing (as part of #3628)

[Steve-Mcl]

@knolleary @joepavitt

I have moved forward with the proposed approach above - here is what it looks like below.

Please let me know if you have an issue with the approach before I begin writing tests etc.