Automate Umbraco installation

[ssougnez]

Prerequisites

• I have linked this PR to an issue on the tracker at http://issues.umbraco.org
• I have written a descriptive pull-request title

Description

Why?

"Umbraco Cloud" is a great solution to automate "Umbraco" installation. However, this option is not always usable (for example, the client I'm currently working for prefers to keep all his data on premise). Therefore, it would be nice to have a way to fully automate an "Umbraco" installation.

Another reason would be that installation/upgrade are not always done by technical people. For example, in my case, deployments are handled by a team of random people following procedures so we try to automate them as much as possible to reduce the number of manual steps. The "Umbraco" configuration is a huge possibility for these people to mess up with the installation by entering incorrect data.

So far, almost everything can be automated:

• Database creation
• Creation of IIS web app/site
• Copy of "Umbraco" files in the IIS site

However, one crucial step can't be automated:

• Configuring "Umbraco" to create the database schema and the default administrator.

The goal of this PR is to provide an Web API route to install/upgrade "Umbraco" without human interaction. This provides the ability to automate an "Umbraco" installation from A to Z.

How to use it?

The new routes are:

• /install/api/Install: Triggers an "Umbraco" installation
• /install/api/Upgrade: Triggers an "Umbraco" upgrade

The "Install" route needs to be called with a specific payload:

{
    user: {
        name: "Sébastien Sougnez",
        email: "s.sougnez@areaprog.com",
        password: "Umbraco2018",
        subscribeToNewsLetter: false
    },
    database: {
        dbType: 0
    },
    configureMachineKey: true,
    starterKit: "00000000-0000-0000-0000-000000000000"
} 

This would instruct "Umbraco" to:

• Create "Sébastien Sougnez" as an administrator with the specified password/email.
• Use a SQL CE database.
• Configure a machine key in the web.config.
• Not use any starter kit.

Of course, it is also possible to use another type of database, for example:

{
    user: { ... }
    database: {
        dbType: 1,
        server: ".\\SQLExpress",
        databaseName: "UmbracoRocks",
        integratedAuth: true
    },
    configureMachineKey: ...,
    starterKit: ...
}

The route returns a 200 response if everything went correctly or throws an unauthorized exception in case of issue.

The "Upgrade" route also requires a payload such as:

{
    username: "s.sougnez@areaprog.com",
    password: "Umbraco2018"
}

As the user needs to be logged in to be able to start the upgrade.

How does it work?

I didn't reinvent the wheel to do this. After a quick investigation, I understood that the configuration mechanism consists in a specific Web API route called several times to execute all the required steps. of this installation. I guess this is done that way because some of these steps induce an app pool recycling.

Therefore, the idea is to mimic this mechanism but server side. Calling the "Install"/"Upgrade" route initializes a new "Setup", then call the existing routes to execute the steps just as the AngularJs service does. The overlapping recycling mechanism ensure that the call to the new routes won't be terminated until the whole process is done.

[nul800sebastiaan]

@ssougnez Thanks! We'll have to discuss at HQ if we want to implement this change like this, we'll get back to you!

[nul800sebastiaan]

Just an update: Shannon is on holiday but he has ideas about this. Unfortunately for you the most important idea he has is: this is not the way to implement this feature.

1. The install steps are hardcoded here but they need to be retrieved from the server as they change at some point
2. He's not happy with it making requests to itself to deal with app recycles

For now I'll close this but we has a task for him to provide more feedback on the PR when he gets back from holidays!

[nul800sebastiaan]

@ssougnez Sorry for the late reply, I totally forgot to post this response from @Shazwazza :-)

====

This is a bit of a long explanation but here it goes...

The Umbraco installation/upgrade is done via the Umbraco.Web.Install.Controllers.InstallApiController. It contains 2x important methods that are used during installation/upgrade:

InstallSetup GetSetup() - this is the first call made which returns a list of steps required to perform the installation/upgrade. Umbraco will automatically determine if this is a new install or if it's an upgrade and return the appropriate steps. Along with the list of steps to execute the response also includes an installId which must be used in calls to the PostPerformInstall

InstallProgressResultModel PostPerformInstall - this is the method used to perform the action for each of the installation steps. This method will be called many times with the same parameters. For unattended installs such as what happens when you install Umbraco with all default settings (no customizations), this will be called until the response data:complete is true, whenever the response is false, then this endpoint is called again. However ... when the installation is customized, some steps will require user input and the way this is done is to render an view and prompt the user for input. If the response from this endpoint contains a value for view then it is expecting user input. Customized installations could also be run unattended so long as the data for each step is supplied in the PostPerformInstall. This same process is true for Upgrades, however for Upgrades there is generally no steps that require user input (though there have been cases!)

A Curl GET call for GetSetup for a new install can simply be:

curl "http://localhost:15754/install/api/GetSetup" -H "Accept: application/json"

The response might look like:

)]}',
{"installId":"e22529e8-4b9f-4490-8ff7-609c284aa229","steps":[{"model":{"minCharLength":10,"minNonAlphaNumericLength":0},"view":"user","name":"User","description":"","serverOrder":20},{"name":"Permissions","view":"","model":null,"description":"","serverOrder":0},{"view":"database","name":"DatabaseConfigure","model":null,"description":"Setting up a database, so Umbraco has a place to store your website","serverOrder":10},{"view":"machinekey","name":"ConfigureMachineKey","model":null,"description":"Updating some security settings...","serverOrder":2},{"name":"DatabaseInstall","view":"","model":null,"description":"","serverOrder":11},{"name":"DatabaseUpgrade","view":"","model":null,"description":"","serverOrder":12},{"view":"starterKit","name":"StarterKitDownload","model":null,"description":"Adding a simple website to Umbraco, will make it easier for you to get started","serverOrder":30},{"name":"StarterKitInstall","view":"","model":null,"description":"","serverOrder":31},{"name":"StarterKitCleanup","view":"","model":null,"description":"Almost done","serverOrder":32},{"name":"UmbracoVersion","view":"","model":null,"description":"Installation is complete!, get ready to be redirected to your new CMS.","serverOrder":50}]}

A Curl POST call for PostPerformInstall in the case of a non-customized install can be:

curl "http://localhost:15754/install/api/PostPerformInstall" -H "Content-Type: application/json;charset=UTF-8" -H "Accept: application/json, text/plain" --data-binary "^{^\^"installId^\^":^\^"e22529e8-4b9f-4490-8ff7-609c284aa229^\^",^\^"instructions^\^":^{^\^"User^\^":^{^\^"minCharLength^\^":10,^\^"minNonAlphaNumericLength^\^":0,^\^"subscribeToNewsLetter^\^":false,^\^"name^\^":^\^"Shannon^\^",^\^"email^\^":^\^"test^@test.com^\^",^\^"password^\^":^\^"testtesttest^\^"^}^}^}"

The response might look like:

)]}',
{"view":null,"complete":false,"stepCompleted":"Permissions","nextStep":"ConfigureMachineKey","model":null}

And this response will change for each step

So what is preventing us currently from using these endpoints to do unattended installation/upgrades from an external system (such as a script?)

• Authentication/Authorization - The InstallApiController doesn't currently have any flexible AuthN/AuthZ applied to it. It uses HttpInstallAuthorize which only checks 2 things: If it's a new installation, then the endpoint is accessible. If it's an upgrade, then this checks that a valid Umbraco user assigned to the request.
• Angular specific JSON responses - The InstallApiController is attributed with AngularJsonOnlyConfiguration which means that only JSON requests/responses are supported but also that the responses are formatted with a special Angular XSRF prefix which may not be ideal for external callers.

What can we do to make unattended install/upgrades possible?

• We can make Authorization (AuthZ) flexible for this controller. The idea way to do this would be to use Authorization Policies which can be done in our version of ASP.NET using https://www.nuget.org/packages/Microsoft.Owin.Security.Authorization/ (https://github.com/DavidParks8/Owin-Authorization/releases) which we've had success with in other projects. There's a bit of plumbing involved especially to have a public API to modify the policy but we have that code already written in another project internally.
• Authentication (AuthN) - if AuthZ is flexible, then AuthN can be done in any way that a developer wishes to implement that. For example, a token auth server could be installed using ASP.NET Identity for Bearer token authentication for this path. This could then assign certain Claims to the request which a custom AuthZ policy can check for. Otherwise in a more simplistic form, Basic Auth can be implemented for this installer path which can be done by either Owin or an HttpModule. That said, if AuthZ is not flexible, and this controller continued to just use the HttpInstallAuthorize, it would still be possible to work around this if the AuthN assigned a valid UmbracoBackOfficeIdentity to the request.
• Optionally we could investigate disabling the custom XSRF json response that is imposed by AngularJsonOnlyConfiguration somehow - perhaps just based on a request header. It's not detrimental to security to disable this response since this endpoint is an authenticated endpoint and is only active under special circumstances (i.e. in an installation or an upgrade state). Otherwise the client that would be calling these endpoints would need to be configured to deal with this custom prefix response.
• Documentation is required to know what data needs to be passed to the server for unattended installs and for the various customizable steps. This data can be easily discovered by using Chrome Dev tools and looking at the requests for these 2 endpoints when running the Umbraco installer/upgrader.

[ssougnez]

I see. Just for information, I made a blog post about that (https://anotherdevblog.com/2018/07/20/automate-umbraco-deployment-with-powershell/) where I explain how to bypass the installation/upgrade page with Powershell ;-)

I'm using it for a while now and the whole script installs and configures a new site in half a minute.

Cheers

[nul800sebastiaan]

@ssougnez Ah, this takes me back 6 years where I wrote something similar in a batch script! ;-)

[Shazwazza]

Worth noting that Chauffeur has this built in too https://aaronpowell.github.io/Chauffeur/

[ssougnez]

That's weird... I thought that I checked this plugin and decided not to use it because it was paying but apparently it's not ^^ Maybe I mistook it for Courrier... I'll have a look at that, thanks @Shazwazza

[stevetemple]

This, or something similar to this has come up in a Twitter conversation I started: https://twitter.com/sitereactor/status/1059716570367356928

To summarise, there is a use case where upgrades are done in one environment and all the file changes are checked in and this checked in version is released to different environments. In these cases the install could potentially run unattended in some way to complete the data migrations on the environments as it's released to them.

Doing that there is a possibility for something to get missed as the installer steps are dynamically created. But IIRC this hasn't happened previously.

[Shazwazza]

Another side note - though I mentioned that to make this happen in a nicer way would be to make the AuthN/Z flexible: #2789 (comment) ... it would still be perfectly plausable to do an unattended install/upgrade from a command line or other service by authenticating with the AuthenticationController just like the back office does and using the resulting cookie to send up with the requests to the InstallerController.