Create an Addon to Backup CMS-Content

[michaelrall]

Hello,

would it make sense to create an addon (or however an addition would be called) that provides an endpoint where one can download a complete content-backup (database and files) of the CMS?

Whats the current suggestion to create a backup? I think that not everybody want's to mess around with system-backup-tools.

I think I could create such an extension, but some guidance where to start and how to organize the project would be appreciated.

kind regards

[michaelrall]

Maybe on could get one step further and offer an Import/Export Endpoint (restoring a backup is as crucial as taking a backup)?

[jensbrak]

I assume you're talking about some sort of extension to the Manager?

[michaelrall]

Yeah, something that can be activated within the startup an be used headless, so one could take a local backup with one http-call. Can the Manager handle calls that provide some kind of authentication (user/pw or maybe an API-Key) ?

[michaelrall]

Would a import/restore work on a "blank" installation with just putting back the DB and all the uploaded files, or are there more artifacts that are needed ?

[michaelrall]

So the Desaster Recovery Scenario would be (Server exploded):

• Create/Install new Server
• Deploy Application (has to fit the one whose data has been backed up of course)
• Post the Backup-Data (maybe zip-archive) to the import-Endpoint
• CMS ready

[jensbrak]

I see where you're going. I am not so educated on the workings of the manager but I do not think it has that kind of support. And since we're talking about an api function rather than a UI function I guess the manager isn't really needed. Of course, with an api ready for backup the manager itself could call it when the admin clicks a button inside the manager or sets a schedule for it. As an addition to the call you propose.

Not knowing what parts of Piranha Core that might be useful for such a solution, I'd imagine making a separate api would be quite feasible, especially if you use some standard api authentication for the api instead. I am not sure the authentication features of the manager is suitable for an api.

Then again, I have not the knowledge to say what's realistic or even possible in this case. But I understand what you're after and think the idea is great. An api for it would open up both for upgrading the manager with backup (and restore) functionality and a remotely controlled scenario like the one you describe.

Taking it one step further one could imagine the entire manager decoupled with an api... I know there have been discussions about that. I believe the answer to this is that while Piranha is designed to work headless (being a framework rather than a complete CMS) it's NOT designed as a separate api with some optional UI on top of that. If I understand and remember correctly.

[jensbrak]

And yeah, such an api would need to be unaware of what underlying implementation of storage the Piranha installation use. Or not unaware perhaps but equally flexible as Piranha itself. You pick your db implementation for the api pretty mycj as you do for your piranha implementation.

[tidyui]

We have been discussing this kind of feature before, but more to solve the scenario of moving data from one environment to another, for example from development to production.

It would be quite easy to write an API that would export the data into a JSON file and then add an endpoint that imported the same data.

The meta data of media assets are also stored in the database, but the question is how they should be returned, it could be up to the consumer of the JSON to download the physical files perhaps?

As far as authentication goes, we always try to rely on claims for manager pages and APIs. As .NET Core is very modular you could add middleware components that would check any type of authentication you want and then just setup the claims if authorization is successful.

Best regards

[michaelrall]

We have been discussing this kind of feature before, but more to solve the scenario of moving data from one environment to another, for example from development to production.

Yes, that would be another reasonable scenario for import/export.

It would be quite easy to write an API that would export the data into a JSON file and then add an endpoint that imported the same data.

I have to admit in my first thoughts I ignored the fact that the database is not fixed (to sqlite), but exporting to an interchangeable format would even allow to change the DB-System. One would have to keep up with DB-Structure changes or is there a generic approach in Entity-Framework to "export DB"? I could imagine going some generic way to discover all Datatables? Does anyone have some deeper insight into the EF-Code? What about Indentity-Values, are some autogenerated Keys used?

The meta data of media assets are also stored in the database, but the question is how they should be returned, it could be up to the consumer of the JSON to download the physical files perhaps?

Yeah, I too had the problem of too big archives and optional splitting in mind - This reduces the easy of use and you would have to use another script etc. to download the assets (and restoring is tricky too). Maybe one could go a best effort strategy and try to cover (maybe 90-95 %) of the average CMS size, that might easily fit into a zip-archive (~max. 500 MB?)

[michaelrall]

We have been discussing this kind of feature before, but more to solve the scenario of moving data from one environment to another, for example from development to production.

What was the result of that discussion? Would it be useable or have there been thoughts that point against such function?

[tidyui]

I think it would be very useful. I think in many scenarios that you start small locally when you develop, perhaps with a file based storage and a SQLite database. Then when it's time to go to production you want to use another form of storage and a dedicated SQL instance. In these cases it would be very handy to transfer all of the data to the live environment. It's also very useful when you need to test the application in your development environment and you need an easy way to transfer the current production data back to your local environment.

You could also use if for completely different use-cases, for example that you have a production environment running without any manager interface and all the edits are done in a separate staging environment. When you're ready with the updates you want to "publish" the entire site to the production environment.

So there are many really good use-cases for it.

[tidyui]

Here's a related issue #497

[tidyui]

And here's another one :) #455

[michaelrall]

And here's another one :) #455

I like the idea of a extensible importer interface, that would keep the system very open. One could provide a basic import/export provider for the forespoken scenarios.
On the other hand, creating such an interface always struggles the balance between freedom and complexity - if its so open/flexible to import everything one could be better off with just coding a new importer based on an existing importer. So for the first step I would create such an interface, but with focus on piranha objects and artifacts. I think you can't get around mapping and converting objects when importing from another system like WP etc., so there will be always "manual" code neccessary.

[r4g-jon]

It would be great to be able to copy content from one server to another.

Something I've thought about when using the Piranha manager is adding in the option to select another database server when you want to copy a page. This way you could select the database server, then select the site, then select the page. Obviously copying assets is something else to think about.

If Piranha had the option to copy pages/media from staging to production that would make it almost perfect!

[michaelrall]

@r4g-jon I think that would be quite difficult to get it working. Direct access to the database of another server can be tricky and needs to be setup properly. I think the more generic approach with import/export would be way more powerful and less complicated. To automate your deployment process you can think of scripting the export from stageing and import to production.

In a second step we can think about only exporting single artefacts (like pages), so you could "copy/paste" them. But again I think this will be much more complex, because for doing this, we would have to analyze the artifact and it's dependencies to get everything right. So step 1 would be full export/import with complete replacement of the system where the data is imported to.

[tidyui]

@michaelrall @r4g-jon One way to think about it is that the module would have one API for export and one API for import. If you install the module on both servers server A could contact the the export API on server B and import the data.

[michaelrall]

@michaelrall @r4g-jon One way to think about it is that the module would have one API for export and one API for import. If you install the module on both servers server A could contact the the export API on server B and import the data.

Yeah, you could push/pull between piranha instances - that would make sense

[michaelrall]

@tidyui Just as a small update - I think still this would be a great feature, but the "full" solution exceeds the amount of time that I could spend for this at the moment. So we can keep this on our list, but it will take some time...