Single command upgrading

[chillu]

Overview

While the upgrader is a very helpful, it interleaves manual instructions with individual automated commands. Ideally we can guide developers through semi-automated updates through a single command. This is aimed at increasing adoption, but also at making it easier to run a single tool to determine the "upgradeability" of a site. We need to get agencies and developers in a position where they feel somewhat confident quoting for upgrades, and getting a site "mostly running on 4.x" is a great way to achieve that.

Acceptance Criteria

• As a developer looking for upgrade instructions, I get clear guidance on the generic upgrading docs and the individual version upgrading docs (4.0.0 and 4.0.0)
• There is only ONE place where the upgrader instructions are placed: In the 4.0.0 changelog
• The upgrader is clearly marked as the preferred way to start an upgrade process
• I can follow manual steps when choosing not to use the upgrader
• The upgrader clearly spells out what is happening with each step, educating developers during the upgrade. It is meant as a partial replacement for reading the upgrading guide. Each step links to the relevant section in the upgrading guide
• The upgrader command can be run multiple times on the same installation
• The upgrader passes through arguments to individual commands (e.g. --write and --verbose)
• When a single command causes a catchable failure, it can continue to the next command
• Composer upgrade command is expected to run separately, since it's hard to integrate into a single command

Commands

• Rewrite _ss_env to dotenv
• Rewrite index.php and .htaccess
• Fix namespaces
• Move mysite/ to app/
• Migrate files
• Inspect code (this should be last)

Note: Order might need to be adjusted

Notes

• Stretch goal: Aim for upgrader to run automated in most cases, either asking for all required user input upfront, or having a --yes option
• Out of scope: Resume upgrader at particular step
• Out of scope: The upgrader is future proofed to work with new major releases (4.x to 5.x)

Sub task for the all in one command:

• Base command calls each of the other commands ;
• Speed up recompose ;
• Manage transition between recompose and environment;
• Manage transition between environment and add-namespace;
• Manage transition add-namespace and upgrade ;
• Manage transition upgrade and inspect ;
• Manage transition inspect and reorganise;
• Manage transition reorganise and webroot;
• Provide a pretty output.

Pull Request

• Creating an all command #83
• The definitive upgrade guide to SilverStripe 4 silverstripe-framework#8143

• BUG Fix invalid .upgrade.yml silverstripe-admin#539
• BUG Remove duplicate .upgrade.yml keys silverstripe-assets#143
• BUG Remove duplicate key silverstripe-framework#8165

[sunnysideup]

We are going to try to upgrade all our client and OS modules at the same time using as much automation as possible. If the automation is smooth enough, we could even fork all the remaining third-party SS3 modules still out there and create an upgrade-to-4 pull request for them (fully automated).

Right now, I have no idea how "automated" we can get (I may be dreaming), but it would be awesome if we could achieve that. Obviously our goal for this is "works with SS4" NOT "is optimised for SS4". I think it is a mistake the mix up the two. First we should make it "work" and then we can optimise it. Optimisation depends on what we are optimising it for (easy to read code, fast code, adjusted to changed best practice / circumstances / etc..., fully making use of the SS4 coolness, or whatever is most applicable)

I have started some work here:

• https://github.com/sunnysideup/silverstripe-upgrade_to_silverstripe_4/blob/master/docs/en/index.md
• https://github.com/sunnysideup/silverstripe-upgrade_to_silverstripe_4/blob/master/src/MetaUpgrader.php

For the upgrade from 2 to 3 I used:

• https://github.com/sunnysideup/silverstripe-upgrade_silverstripe/blob/master/docs/en/INDEX.md
  It had a couple of strengths:
• mark all the areas that need manual attention
• count the total number of changes required to upgrade (separated by automated / manual) so that you know exactly what is involved to upgrade
• easy to add / remove items that need to be changed, see: https://github.com/sunnysideup/silverstripe-upgrade_silverstripe/blob/master/ReplacementData.php#L90 as an example.

[chillu]

Thanks for sharing @sunnysideup! Our goal for core is a reduced upgrade burden, but since we have pretty high standards on the accuracy of the tools we provide, automated upgrades won't be feasible as part of this effort. So there's some room for additional/optional community-maintained tools - ideally complementing rather than duplicating the existing upgrader work of course :)

mark all the areas that need manual attention

We're trying to do this with the upgrade inspect command - have you tried it?

easy to add / remove items that need to be changed

Wow, that's a pretty comprehensive list! We're already trying to do this through the upgrader command, everything with a replacement key in .upgrade.yml (example). It's probably a bit stricter than your implementation, parsing PHP rather than doing regexes. Gets less coverage, but also less capacity for false positives. Maybe you could check if we missed anything in this list, or if there's any high usage APIs we should add?

[sunnysideup]

Hi Ingo,

Thank you for your notes. The inspect command is broken for me: #70

My list was up to 3.1 and not for 4.0.

Once I do some more testing, I will write a few more notes here.

[sminnee]

As part of this, it would be great if the warnings from the inspect step were placed into a text file or something - upgrade-todo.txt or similar. As a developer you're going to need to refer to this after you close your CLI window.

[sunnysideup]

just wanted to share that I worked on my "multi module" tool a little - see:

https://github.com/sunnysideup/silverstripe-upgrade_to_silverstripe_4/blob/master/src/MetaUpgrader.php

and for usage see:
https://github.com/sunnysideup/silverstripe-upgrade_to_silverstripe_4/blob/master/docs/en/INDEX.md

It still needs a lot of TLC, but I have great hopes ;-)

[maxime-rainville]

I'm about to start working on this. I suggest splitting it in two:

• The doc cleaning ;
• The "one command" to rule them all.

One command to rule them all

Basically, I'll have to go through each command and get a full listing of all the available flags. Some of them might not be relevant, so can probably be omitted.

I don't think we can do this without writing our changes. Some steps require that a previous step had been completed before running. e.g.: inspect can not run before upgrade, webroot can not run unless the site has been upgraded to recipe-core 1.1.

Order of command

Some steps might be optional and won't prevent future steps from being run. (e.g. Upgrading web-root). Other steps will need to be successful before letting the next command run. Of the top of my head this seems like the best order.

1. recompose (Mandatory, stop execution on failure)
2. environment (Mandatory)
3. add-namespace (Mandatory)
4. upgrade (Mandatory, stop execution on failure)
5. inspect (Mandatory, stop execution on failure)
6. reorganise (Optional)
7. webroot (Optional)

I would exclude doctor because it can break things if run repeatably. I would make all the steps mandatory ... at least in the first iteration. That gives more predictability to the script. e.g.: The arguments passed to add-namespace, may very based on whatever you've run reorganise or not.

Providing nice output and feedback

Managing the output here could be interesting. Each command provides warnings, which should be sent to the STDERR. If I just concatenate the output of each command and spit it out at the user, the user will just have a big blurb of text at the end of the process sprinkled with warnings.

If I can capture the warnings and display them all in a neat block at the end of the command, that probably would be better for the user. I might need to refactor the output of each command to make it easier to capture and more consistent. I might need to get each command to expose a ChangeDisplayer so I can see what has been done.

To @sminnee point, it's probably a good idea to automatically save the output to a file. Obviously, there's nothing to stop the user to manually redirect the output to a file with > however I get the feeling a lot of people won't think of doing this before running the command. So I would make it an opt-out.

Making recompose faster

recomposer takes about 10 min to run on a sizable project, irrespective of whatever it's been upgraded or not. It might be worthwhile to add some sort of minimalist sanity checks:

• check that you are running the minimal recipe-core target ;
• runs a composer validate to make sure composer.json and composer.lock are in sync.

This would speed up the process quite a bit, however you wouldn't get the recipe simplification logic and stricter version constraint.

Testing this crazy thing

I'll probably be testing this with a very vanilla project to begin with. However, I'll have to switch to a real world example to tease out edge cases. So I'll be implicitly upgrading that project along the way.

I'll need something small enough that it doesn't take too much time to run the script, but complex enough that it throws some curve-balls at me. You may now suggest a project you want me to fix for you.

Other points

• We don't currently run unit test against the Commands ... we test the underlying components used by each command. The all-in-one command will be relying on each individual command, so it might be worthwhile adding a few high level unit test for each command that test typical scenarios. Although it would potentially add a day to the estimate.
• So far this is mostly geared towards SilverStripe project and not so much towards SilverStripe module. It might be worthwhile to add a command specific for SilverStripe Modules. It wouldn't take very long as it would mostly be a subset of all-in-one command.
• Some of the names of the commands arguably don't make much sense. e.g.: Inspect does more than inspecting things now ... all the commands upgrade something so the upgrade command is kind-of confusing. It might be a good time to rename some of these commands, while it's still manageable to do so.

[maxime-rainville]

I've already put some thought into redrafting the doc, althought it might be bigger than what @chillu had in mind. It's probably best to leave this until I'm done with the all-in-one command as repeatably bashing my head against the wall will increase my knowledge of each step.

Reorganising the doc.

I’m proposing we take the current Upgrading to SilverStripe 4 doc and convert to a multi page upgrade guide.

Each step would become an individual page. The page would be designed to be read in a specific sequence and would follow more or less the same pattern.

The broad purpose of this change would be to make it easier for developers to understand what they need to do when upgrading a site and divide the process up into smaller more understandable task. We still want developers to have a good understanding of what they are getting into and the amount of work that is required.

Problems with the existing doc

• We tell people to go through various change logs and composer.json file to figure out what they need to do.
• It’s not really clear what needs to be done and in one order

What the standard Step/Page layout should look like

• Brief introduction explaining:
  • What the broad nature of the step is
  • Why it is necessary
  • Potential benefits for the developer
• The completely manual way of performing the step
• The automated way of doing it … if available
• How to handle special use case (aka Gotchas)
• Any specific minor version idiosyncrasies
• What state your SilverStripe Project should be in once you’ve completed the step
• Link to next step

Each page should cover at most one upgrader command.

Basically, the manual section explains the detail of what’s going to happens when you run the upgrader command so that you can debug the output afterwards.

Table of content

1. Index page with links to each individual step.
2. Introduction explaining the “manual upgrade” vs “automated upgrade”
3. Pre-requisite

• Install composer
• Backup your files and DB (aka: don’t do this on a live site)
• PHP 5.6 and above
• Not using homebrewed forks of core module

4. Step 1 - Upgrade your dependencies (recompose)

• Switch to recipe-core and recipe-cms
• Update constraints to get latest version of everything
• Run composer update / Resolve any conflicts

5. Step 2 - Update your environment (environment)
6. Step 3 - Add Namespace (add-namespace)
7. Step 4 - Refactor your existing code base (upgrade)
8. Step 5 - Apply API changes (inspect)
9. Step 6 - Rename mysite (reorganise)
10. Step 7 - Switch to public web-root

• Move folder and files around
• Update web server config

11. Step 8 - Running your upgraded site for the first time

• Run a dev build
• Any other script that needs running.

12. Strategies for Go Live.
13. Note for module maintainers.

[sunnysideup]

I will focus on building a tiny tool to upgrade modules that will be complementary to the effort detailed above so I am stoked if this goes ahead. Thank you.

[maxime-rainville]

@chillu On the point of renaming the commands, the Symfony console command class allows us to group command allows us to categories command with a : syntax. e.g.: code:upgrade

This means when you look at the help for your command you get something like:

We're adding new command at a furious rate right now, so it would make sense to start categorising them.

[chillu]

Great write up, that really helps me as a PO! Sorry for taking so long to respond.

I don't think we can do this without writing our changes

That's fine, as long as the upgrader clearly spells out that it's going to change the codebase as part of the process

I would make all the steps mandatory ... at least in the first iteration

I don't want to force people to change their webroot, or rename from mysite to app. How much extra effort is it to ask for confirmation before running those steps?

If I can capture the warnings and display them all in a neat block at the end of the command, that probably would be better for the user.

Only do this if it doesn't involve multi-hour refactoring. We can use colour and ascii headlines to highlight "sections" of output.

it's probably a good idea to automatically save the output to a file

Let's stick to stdout, this is gold plating.

recomposer takes about 10 min to run on a sizable project, irrespective of whatever it's been upgraded or not. It might be worthwhile to add some sort of minimalist sanity checks

Good idea, but sounds like something we can put into docs for now. How often do you think people will run this command on an already updated composer.json?

We don't currently run unit test against the Commands ... we test the underlying components used by each command. The all-in-one command will be relying on each individual command, so it might be worthwhile adding a few high level unit test for each command that test typical scenarios.

I don't think that's worth a day of effort. How much would it be if you stub out the commands instead, and just test the "unit" of the all-in-one command? Basic stuff like checking it doesn't run a command if you've opted out of it, nothing fancy.

[chillu]

I’m proposing we take the current Upgrading to SilverStripe 4 doc and convert to a multi page upgrade guide.

I like single page upgrade guides because they become easily searchable through your browser. Talking to partners and developers, that's a real use case. Yes, it looks more scary (longer) than individual pages for steps, but there's no point hiding the complexity behind a bunch of links.

Otherwise I agree with your "standard step/page layout", I'd just change that to a section on a single page instead. And we should recommend the automated way before explaining the manual way.

How are you proposing we consolidate the upgrading.md, 4.0.0.md and 4.1.0.md guides? I think we should trim down upgrading.md, leave the bulk in 4.0.0.md, and move over the public webroot and app folder instructions from 4.1.0.md into 4.0.0.md marked as "4.1 only". The 4.1.0.md document should link to those sections. It's more important to have one document you can read coherently from start to finish, than to have information fragmented but in the "right" place. We'd still have a 4.2.0.md etc as usual, but 4.1.0.md is a bit different since it contains opt-in API changes we want people to action as part of the major release upgrades.

I guess those guides serve a few different audiences:

• Devs getting a quick overview of technical changes (without requiring details on how to action them)
• Decision makers getting a quick overview of changes (tech and UX)
• Authors getting a quick overview of changes (UX)
• Devs preparing to upgrade (and estimating, assessing risk for their project)
• Devs going through a major upgrade (e.g. 3.6.2 to 4.1.0)
• Devs going through a minor upgrade (e.g. 4.1.0 to 4.2.0)
• Devs fixing specific issues during an upgrade (e.g. searching for an API change)
• Devs generally figuring out the upgrade process of a SilverStripe site (stuff like "back up your database")

Your doc restructure is pretty geared to "Devs going through a major upgrade". Do you think we can create one doc which serves these different use cases? Maybe we can do a step-by-step guide and link off to details on the individual steps as anchors on the same page?

Proposed structure:

1. Overview for Developers (with deep links into sections further down the page)
2. Overview for Authors
3. Upgrade Process (manual vs. automated. Should incorporate advice from manual upgrades, but trim down its content)
4. Step-by-step upgrade (favouring automated upgrades and providing overview, linking to detailed sections on API changes further down the page with manual steps)
5. Important API Changes (in depth descriptions, flagging any changes which are automatically rewritten. liberally link to permanent docs where possible. keep topics together, e.g. database API changes)
6. List of all API changes (bullet points)

It might be worthwhile to move the full changelog at the bottom to a different document to make this page look a bit less scary?

Also, I'm not sure where additional tips easily missed fits at the moment. It feels like some of those should be sections in the "Important API Changes", others should be in "Overview for Developers"? At the moment, we have so many "overviews" that there's no overview.

[maxime-rainville]

I would make all the steps mandatory ... at least in the first iteration

I don't want to force people to change their webroot, or rename from mysite to app. How much extra effort is it to ask for confirmation before running those steps?

It's not really asking the user that will take time. It's that if I need to apply changes to the code base later on, I'll need some mechanism to know if I need to operate in mysite/code or app/src. It adds a bit more complexity to the process. But it should not be a big deal.

recomposer takes about 10 min to run on a sizable project, irrespective of whatever it's been upgraded or not. It might be worthwhile to add some sort of minimalist sanity checks

Good idea, but sounds like something we can put into docs for now. How often do you think people will run this command on an already updated composer.json?

Quite a bit actually. Most steps require a previous step to have completed successfully. So the execution will need to be halted on many occasion to give a chance to the user to fix stuff.

How much would it be if you stub out the commands instead, and just test the "unit" of the all-in-one command? Basic stuff like checking it doesn't run a command if you've opted out of it, nothing fancy.

That's a good idea. I'll give that crack.

[maxime-rainville]

Single page vs multi-page

I will strongly dissent with the single page idea:

• I think it does more than show complexity, it compounds it ;
• it hurts readability ;
• it will make it harder to edit the document.

We add a progress marker in he guide (e.g.: step 1 of 20), that should give the reader a good indication of where they are in the volume of work involved.

Audience

I really saw this has a developer focus document, but I can see the value in having some section for other stakeholders.

I would definitely would not to mix content author and decision maker doc with the developer doc. If I’m a content author who wants to learn more about the difference between SS3 and SS4, I’ll probably be pretty intimidated if I land on a massive document that talks about API changes, PHP version, running commands, etc.

Guides vs change logs

We pretty much have the exact opposite view on this one. I would want to shift most of the content on the guide. 4.0.0.md and 4.1.0.md should just be a boring list of changes since the last release with little to no explanation about how to implement those changes.

If there’s a specific minor version introduce a significant change that require some decent amount of work to upgrade, than that should be a designated step in the guide.

It’s not all frowny faces and discontent

The API reference section is a great idea. I agree that additional tips easily missed should mostly go in the important API change section.

[chillu]

Do we want recipes to install their installer files when running recompose?

You mean stuff like moving Page.php? Shouldn't that already work gracefully on existing installation? Otherwise, those files are considered necessary for a project, right? So it shouldn't cause duplicates. If there's something wrong with that logic in the existing recompose command, please raise it as a separate bug.

Should add-namespace detect folder changes and incorporate those in the updated namespace?

No, separate feature ticket please.

What do we want the default namespace to be?
add-namespace doesn't support single-level namespace. This may or may not be an issue.
Would be cool if add-namespace auto added a reference to composer's autoload.psr-4.

How is this related to the single command use case? Sounds like enhancements to me. I think overall, if the add-namespace command isn't up to scratch to add namespaces to your project code, we should exclude it from the single command run for now. It's an optional step, presumably your project still works even if you don't apply a PSR-4 autoloader.

Would be cool if reorganise updated autoload.psr-4 and SilverStripe\Core\Manifest\ModuleManifest::project

Yes, I think we should have it change the project config if we rename from mysite to app. How much work is that, assuming we're doing it in YAML, and remove any $project leftovers in _config.php?

[chillu]

I would definitely would not to mix content author and decision maker doc with the developer doc

The problem is that devs can be both decision makers and upgrade implementors. Those are quite different use cases. A decision maker would want to scroll through easily digestible (technical) information quickly, in order to make a judgement call if they recommend an upgrade implementation to their boss/client in the first place. An implementor is more interested in the step-by-step. And then there's devs who are halfway through upgrading their codebase, but are stuck and need to search for a specific problem (e.g. .env handling). If we hide that on Step 7 of 15 in another page, it's harder to find. Maybe that can be solved with inter-linking...

We pretty much have the exact opposite view on this one. I would want to shift most of the content on the guide. 4.0.0.md and 4.1.0.md should just be a boring list of changes since the last release with little to no explanation about how to implement those changes.

I'm OK with that. Just worried that as we release 4.2, 4.3 etc, we'll need to communicate those upgrade concerns (optional deprecated stuff, new features) in the individual changelogs again - it'd become too messy to tack them onto upgrading.md. So 4.0.0.md and 4.1.0.md become special cases, where this info is contained in upgrading.md, but once we announce 4.2 we'd want to point to 4.2.0.md as the canonical source of information, right?

Overall, I'm still not sure what use case different pages serve that can't be served better in a single doc with anchors. I think we need another opinion - @unclecheese @flamerohr does either of you want to weigh in before we proceed?

[chillu]

I've run a quick poll on Twitter for a related question: https://twitter.com/chillu/status/999393971817275392. The upgrade guide is a bit different again, but overall there's a clear preference for longer single page docs.

[flamerohr]

I think SinglePage is good for a guide/readme - only if it's anchored and linked properly with proper section headings.

MultiPage is a good mechanism for enforcing proper sections to be written out, something that can be done on a SinglePage.
The sections have the same issue - they need to be named properly, otherwise it falls directly into the issue which @chillu is concerned about - hidden information.
E.g. I had a lot of grievances when bootstrap decided to multi-page their docs.

At least with SinglePage, even if the section is named incorrectly you can search on the page without needing to jump to a search engine.

As a sidenote: I do agree with keeping change logs simple and keep information of implementation somewhere else, a simple changelog list helps the decision making without getting lost in the implementation details.

[sunnysideup]

@maxime-rainville , I have now created a really fragile working model of a module upgrader, that will hopefully work with your all-in-one upgrader:

This file may give you a bit of an idea on how it works:
https://github.com/sunnysideup/silverstripe-upgrade_to_silverstripe_4/blob/master/example-index.php

[maxime-rainville]

@sunnysideup

The way I did the CommandRunner could in theory allow it to be use for upgrading modules with slightly different defaults.

That might be a bit more relevant to #85. Let's maybe take that discussion there so we don't get things mixed up.

[sunnysideup]

sounds good.