Attempt Drupal 8.4.x support in Drush 8.x via the global Drush

[greg-1-anderson]

Despite the fact that Composer dependency hell is a hard problem, everyone loves the convenience of using a global Drush installation. Drush 9 and Drupal Console only support site-local installations, but Drush 8 has continued to support the global installation method.

This story is greatly complicated by Drupal 8.4.x, which has upgraded to a new major version of Symfony (3.x) during a minor release cycle. While SemVer explicitly allows this, it poses a particular problem with Drush. With most SemVer-respecting software, upgrading to match the new requirements of dependencies is a one-time task. Once an app has been upgraded, it no longer needs to remain compatible with older versions of its dependencies.

Not so with Drush. Drush 8.x remains compatible with Drush 6, 7 and 8. If we do not address the Symfony 3 issue, then Drush 8 will be forced to drop support for Drupal 8.4.x. This poses some serious problems:

• It complicates the version support story, as traditionally, each major version of Drush has supported specific major versions of Drupal.
• There would be no way to use Drush to update from Drupal 8.3.x to 8.4.0.
• Anyone using Drupal 8.4.0 and later would have to use Drush 9, which means that they would have to use Composer to manage their Drupal site, or do all updates by hand.

This PR lays down the groundwork to attempt to provide support for both Drupal 8.3.x and earlier, and Drupal 8.4.x and later, in the same global installation of Drush at the same time.

Strategy:

• Drush will perform its early bootstrap using only code provided by Drush; no dependencies will be used at first.
• Once Drush has located the desired Composer root for the desired site (currently defined as the folder that contains both composer.json and composer.lock), then it will select an appropriate compatible autoload file and include it. If no composer root is found (Drupal 7 sites not managed by Composer, and Drupal 6 sites), then Drush will load its current autoload file.
• The later bootstrap stages (bootstrap root and later) then run as they always have.

In its current form, this PR succeeds at delaying the loading of the autoload file until immediately before the bootstrap root phase, after it has found the Composer root. At this point, no attempt is made to identify whether the dependencies in the target Drupal site are compatible or not, nor has any attempt been made to provide classes Drush needs that are compatible with either Symfony 2 or Symfony 3. This is the easier problem, though, and I am sure it is possible. (n.b. The Consolidation components are already compatible with both Symfony 2 and Symfony 3, and abstract some of the differences. These versions of Symfony are already very similar, though.)

There are two problems remaining in the early bootstrap phase, though, as there are two external dependencies that Drush needs from the very beginning:

• webmozart/path-util
• psr/log

In the current version of this PR, Drush is manually including the two files it needs from path-util. This is an interim implementation. For logging, there is a commented-out procedural version of the logger that simply printf's the log messages until the PSR-2 logger becomes available. It is commented out because the interim implementation does not respect verbose / debug modes.

There are a number of ways we can solve the problem of the dependencies needed early in the bootstrap process. If the tests can be made to pass on the code written so far (or at least reduce the failures to only those issues related to unimplemented items mentioned above), then I think this will demonstrate that this is a viable proof-of-concept.

Note that the strategies suggested here are not particularly maintainable. The assumption is that this code will not be brought forward to Drush 9 (UPDATE: This strategy was implemented in Drush 9.0), and that the Drupal community will move to exclusively managing their Drupal sites with Composer prior to any future Drupal change that is unsupportable in Drush 8 (e.g. Symfony 4). Presumably, all of that happens by / with Drupal 9.

[greg-1-anderson]

Since webmozart/path-util is small, I imported a copy of it into our repository and re-namespaced it to avoid conflicts should any dependency loaded later require the original package.

There are still code paths that fail trying to use classes that are not yet available, so this PR is not provably viable yet. There are already some passing tests, though.

[alisonjo315]

Hi! Is this PR still critical for continued support of pm-* commands in drush (in the context of Drupal 8)? (It was referenced a while ago as one of the most critical issues related this functionality, but I wasn't sure if that's still the case.) Thanks!

[alisonjo315]

( @greg-1-anderson Not positive, but I think this is a typo?)

Not so with Drush. Drush 8.x remains compatible with Drush 6, 7 and 8.
("...with Drupal 6, 7 and 8" -- right?)

[weitzman]

Drush 8.15+ can live inside a composer project that also has drupal 8.4.

If you are not using composer, then Drush 8 happens to work with Drupal 8.4 but thats not supported by the Drush maintainers

[weitzman]

I'm closing this PR as it is no longer needed.

[alisonjo315]

Understood re: "global / site local drush" situation, and I understood that right now it "happens to work" / is not supported by the maintainers.

My intended question (here and with my comment over on #3041) was more along the lines of...

• Drush 8 with Drupal 8.4 is a little precarious (i.e. "happens to work"), so it seems reasonable to ask, do we need to worry about Drush 8 with Drupal 8.5+, or, do you think it will continue to "happen to work" for the foreseeable future, of course based only on how things seem to be progressing on the Drupal core side of things?
  • (I'm not objecting to the "happens to work" status, I'm quite grateful for it; I want to know if there are things y'all are already aware of coming soon that will endanger the "happens to work" status -- like, a while back, @greg-1-anderson linked to some Drush issues/PRs that he suggested would be important to contribute to if you're worried about Drush 8 + Drupal 8 compatability -- so I'm wondering if any such issues exist right now, so I can follow along now instead of waiting 'til it becomes a problem for me ;-) )
• Similarly, pm-* commands with Drupal 8 + Drush are on shaky ground -- Greg explained a few months ago that the maintainer simply didn't have the time to keep up with them, given all the changes happening on the Drupal core side of things, so they're not in Drush 9 rn.
  • We were pointed to this PR as one example of work to follow/contribute to related to future support of pm-* commands -- I understand if that's not the case anymore, but that's what I'm asking about.

I hope that all makes more sense, but please lmk if not. Thank you!

[greg-1-anderson]

I still feel that this PR is needed, and would make the stability of the global Drush 8 with Drupal 8.4.0 / 8.5.x / 8.6.x improved. The techniques originally pioneered in this experiment are now implemented as the Drush 9 preflight / bootstrap. I'd like to re-open this and continue working here.

Factors for consideration:

• There is an initiative to make Composer work well in the Drupal admin interface. This won't be in 8.5.x, and might be in 8.6.x, but there's also the chance that we'll need to wait until 8.7.x for this.

• Until Composer is solidly supported in core, we'll still have a lot of users who continue to use pm-* and the Drupal admin interface exclusively to manage their modules.

• As long as there are still a lot of users avoiding the use of Composer, widespread use of Composer as a dependency manager for modules will be slowed.

• We therefore probably have 12-18+ months of pm-* users.

To address this, I think that two things should be done:

• Finish this PR, so Drush 8 has enough legs to continue being used as a global Drush.

• Support the pm-* commands as a standalone project (either a global Drush 9 extension or maybe just a Symfony Console / Robo application that has all of the pm-* commands).

Note that Drush 9 should work fairly well as a global Drush, so if we had pm-* support that worked compatibly with Drush 9, and we had most of the key Drush module extensions ported over, then that would be a viable support strategy as well. However, I think we also need this PR to support that transition, as I'm not sure that the existing Drush 8 will easily survive the 8.5.x transition without this.

[gitressa]

@greg-1-anderson: Like @alisonjo2786 I am very grateful that Drush 8 still works with Drupal 8, and I am very encouraged by the plans to carry on supporting Drush 8 with Drupal 8 12-18+ months. Your thoughts about pm-* support in Drush 9 also sounds very promising, and I am very much ready to test patches aiding the future support of Drush 8 for Drupal 8.