New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Drush up should update contrib profiles as well #99

Closed
seanr opened this Issue Sep 12, 2013 · 17 comments

Comments

Projects
None yet
6 participants
@seanr
Copy link

seanr commented Sep 12, 2013

Oirginally here: https://drupal.org/node/1009298

Currently drush up only handles modules and themes, meaning any install profiles being used in a multisite install need to be manually downloaded with drush dl. Ideally, drush up would update any contrib profiles the same time it updates modules and themes so future sites added to a multisite use the updated install profile. I believe this would make even more sense for Drupal 7 since install profiles will now contain update functions.

This was Moshe's response:

The drupal community has not decided what it means for modules to be packaged under /profiles/[profilename]. Specifically, are those modules supposed to be upgraded by the user or are they under the management of the distribution? It might be that each profile should declare its maintenance strategy.

I'd like to propose that drush update the entire profile if a distribution has a new version. Otherwise, maybe it could prompt about updating individual modules within the profile. Selection no would then just continue with updating modules outside the profile.

@shawnholt

This comment has been minimized.

Copy link

shawnholt commented Oct 28, 2013

This is a big issue - another previous thread with people offering $$$ - https://drupal.org/node/1888344

@kreynen

This comment has been minimized.

Copy link

kreynen commented Oct 29, 2013

While the rest of the community may still be catching up, I think developers packaging distributions have already decided what packaging modules in a distribution means.

Modules, themes, libraries in /profiles/[NAME]/ should NEVER be updated independently of the distribution. You shouldn't "hack" a distribution any more than you should hack core

The distribution has a version number and the version indicates what version of ALL the modules, patches, and libraries required for that specific version of the distribution to work. Users can upgrade a specific module from the version included in the distribution by overriding by adding it to sites/all/modules, but they should never update modules in /profiles/[NAME]/ any more than they would update modules to core's /modules... like running the contact module from 7.23 with core 7.21. I think everyone would agree that mixing the versions of modules from different core releases would be a VERY bad idea. It's only slightly less bad for distributions.

We've been developing tools like https://drupal.org/project/profile_status_check to give site builders a visual overview of what's been overridden and what is still running on the "core" distribution making it easier to add an override then remove it when the distribution catches up.

I believe the distributions are also already declaring their approach to maintenance by creating windows to push an update to their distribution in hook_update_status_alter...

http://drupalcode.org/project/commerce_kickstart.git/blob/refs/heads/7.x-2.x:/commerce_kickstart.profile#l163

This approach of suppressing updates to modules in the distribution until developers have a chance to test the update is also supported by how upstream distribution updates are handled on platforms like Pantheon and Acquia Cloud. See any update in the Drupal administration UI? Is there an update for the distribution to handle that?

This shouldn't be treated any different that sub-modules in a large module project. You'd never advocate updating views_ui if views wasn't being updated. Same pattern. Different scale. The code packaged in a distribution is designed to work together and should always be updated together.

@seanr

This comment has been minimized.

Copy link

seanr commented Oct 29, 2013

Where this becomes an issue is if you use a distribution merely as a starting point for something more complex (i.e. the distribution gets you 90% of what you need in basic setup, but you want to add a lot of extra functionality around that). We've run into bugs in versions of modules that were packaged in distributions which later versions have fixed. Some distributions can also get quite stale, compounding that problem. Furthermore, if developers on a project get switched around, the new one may come along and see a bunch of pending updates and decide to run them, not realizing that drush will choke on it (and then it can be a bit of a pain to revert or even figure out what happened to begin with).

If you want to prohibit drush up from running on distributions, at the very least, you should add a check so that it cannot be used in that case (IIRC, Drupal stores the profile it was installed from in a variable which you could check prior to running the actual update).

@shawnholt

This comment has been minimized.

Copy link

shawnholt commented Oct 29, 2013

Well I would say it is either a profile or not (ie standard) Why not use
something like this: https://drupal.org/project/profile_switcher when you
are no longer on the distro.

On Tue, Oct 29, 2013 at 4:36 PM, Sean Robertson notifications@github.comwrote:

Where this becomes an issue is if you use a distribution merely as a
starting point for something more complex (i.e. the distribution gets you
90% of what you need in basic setup, but you want to add a lot of extra
functionality around that). We've run into bugs in versions of modules that
were packaged in distributions which later versions have fixed. Some
distributions can also get quite stale, compounding that problem.
Furthermore, if developers on a project get switched around, the new one
may come along and see a bunch of pending updates and decide to run them,
not realizing that drush will choke on it (and then it can be a bit of a
pain to revert or even figure out what happened to begin with).

If you want to prohibit drush up from running on distributions, at the
very least, you should add a check so that it cannot be used in that case
(IIRC, Drupal stores the profile it was installed from in a variable which
you could check prior to running the actual update).


Reply to this email directly or view it on GitHubhttps://github.com//issues/99#issuecomment-27340986
.

Shawn Holt
me@shawnholt.com
Google Voice: (973) 850-9773
LinkedIn.com/in/shawnholt

@seanr

This comment has been minimized.

Copy link

seanr commented Oct 29, 2013

That doesn't solve the problem of having the original profile's modules in the profile folder, which is what chokes drush.

@kreynen

This comment has been minimized.

Copy link

kreynen commented Oct 29, 2013

I think we're getting OT. I'm only referring to how drush should handle updating modules and themes included in the directory of a site's current .profile.

When using modules, themes, and libraries from a distribution in sites/all, drush should treat those like any other site regardless of the .profile.

All distributions packaged on Drupal.org include a non-core download that makes it easy to place just the modules, themes, and libraries in sites/all/modules. Go to Project Page, click "View all releases" and select the -no-core.tar.gz or -no-core.zip. That's a much better approach if you are only planning to use a distribution as a starting point. You can even add the .profile, .info, and .install to /profiles. Then you can use the distribution's profile install and drush to update the modules.

Normally when you install from a profile and go to Admin > Reports > Status Report, you'll see which profile the site is using. The first line will say Drupal and the version of core you are using. If you used a .profile other than Standard to configure the site, the next line says Install profile and the profile/distribution's version.

ONLY modules in the /profiles/[PROFILE NAME] directory of the active profile can be used. Just "saying" you are or aren't using a distribution/profile doesn't isn't enough... it actually has to be configured. If you add a profile to /profiles to a site that was started with a standard install, there is a manual process for activating the profile by editing a var with drush or PHP and updating the system table... or you could use https://drupal.org/project/profile_switcher to switch to an existing site to enable or disable a distribution.

Let's say you have a staff wiki you want to turn into an OpenAtrium 2 sites. Download the OA profile with modules, themes, and libraries to /profiles. Enable profile_switcher. Switch the profile. Just like module dependencies, anything required by http://drupalcode.org/project/openatrium.git/blob/refs/heads/7.x-2.x:/openatrium.profile will be activated. That works because OA isn't doing much in http://drupalcode.org/project/openatrium.git/blob/refs/heads/7.x-2.x:/openatrium.install. That wouldn't work as well for http://drupalcode.org/project/commerce_kickstart.git/blob/refs/heads/7.x-2.x:/commerce_kickstart.install

This also works for getting off of a distribution. Say you installed OpenPublish and didn't want to be one of the 1000+ sites that were running versions of core and modules with known security issues, you can copy the versions of the modules from profiles/openpublish/modules and switch the profile from openpublish to standard and start updating like you normally would.

As far as the developers getting switched to a project and just updating everything... when is that OK?!?

I don't think you are going to find anyone actively managing a starter kit or product distribution that wants drush to update the modules in profiles/[PROFILE NAME]/modules.

@shawnholt

This comment has been minimized.

Copy link

shawnholt commented Oct 29, 2013

Right, so if I want to update a module that is part of a profile and retain
the profile in tact, it should add a new module to the site/all/module
directory. If I want to update the profile, it should update the profile
directory. What if the profile update now supersceeds the sites/all/module
version? Just delete it?
On Oct 29, 2013 6:35 PM, "kreynen" notifications@github.com wrote:

I think we're getting OT. I'm only referring to how drush should handle
updating modules and themes included in the directory of a site's current
.profile.

When using modules, themes, and libraries from a distribution in
sites/all, drush should treat those like any other site regardless of the
.profile.

All distributions packaged on Drupal.org include a non-core download that
makes it easy to place just the modules, themes, and libraries in
sites/all/modules. Go to Project Page, click "View all releases" and select
the -no-core.tar.gz or -no-core.zip. That's a much better approach if you
are only planning to use a distribution as a starting point. You can even
add the .profile, .info, and .install to /profiles. Then you can use the
distribution's profile install and drush to update the modules.

Normally when you install from a profile and go to Admin > Reports >
Status Report, you'll see which profile the site is using. The first line
will say Drupal and the version of core you are using. If you used a
.profile other than Standard to configure the site, the next line says
Install profile and the profile/distribution's version.

ONLY modules in the /profiles/[PROFILE NAME] directory of the active
profile can be used. Just "saying" you are or aren't using a
distribution/profile doesn't isn't enough... it actually has to be
configured. If you add a profile to /profiles to a site that was started
with a standard install, there is a manual process for activating the
profile by editing a var with drush or PHP and updating the system table...
or you could use https://drupal.org/project/profile_switcher to switch to
an existing site to enable or disable a distribution.

Let's say you have a staff wiki you want to turn into an OpenAtrium 2
sites. Download the OA profile with modules, themes, and libraries to
/profiles. Enable profile_switcher. Switch the profile. Just like module
dependencies, anything required by
http://drupalcode.org/project/openatrium.git/blob/refs/heads/7.x-2.x:/openatrium.profilewill be activated. That works because OA isn't doing much in
http://drupalcode.org/project/openatrium.git/blob/refs/heads/7.x-2.x:/openatrium.install.
That wouldn't work as well for
http://drupalcode.org/project/commerce_kickstart.git/blob/refs/heads/7.x-2.x:/commerce_kickstart.install

This also works for getting off of a distribution. Say you installed
OpenPublish and didn't want to be one of the 1000+ sites that were running
versions of core and modules with known security issues, you can copy the
versions of the modules from profiles/openpublish/modules and switch the
profile from openpublish to standard and start updating like you normally
would.

As far as the developers getting switched to a project and just updating
everything... when is that OK?!?

I don't think you are going to find anyone actively managing a starter kit
or product distribution that wants drush to update the modules in
profiles/[PROFILE NAME]/modules.


Reply to this email directly or view it on GitHubhttps://github.com//issues/99#issuecomment-27350238
.

@kreynen

This comment has been minimized.

Copy link

kreynen commented Oct 30, 2013

The version of the module in sites/all/modules always trumps the profile version regardless of whether it's newer or older. This order of priority is a function of core. The issue of having a older version of a module in sites/all that's trumping what's in profile/ is why I wrote https://drupal.org/project/profile_status_check

Because the starter kit distributions we developer are designed to remain flexible and allow developers/site builders to add modules and override existing modules, it became increasing common to find that an issue with a site was caused by an override. The sitebuilder would add the latest version of something like Media to their site and it would solve a problem they had without causing any other issues. Eventually the distribution would also update to that version, which was still fine. The problems start when the distribution updates Media again. Then the outdated module in sites/all is trumping the newer version in the profile. Because other parts of the distribution expect the newer version of Media, the site starts experiencing errors.

Before profile_status_check, the only way to see this is to look at the modules in the different directories in sites/all/modules or the system table. Moving modules between /profile and /sites/all works for most modules and allows site builders to offer feedback on potential updates before the updates are rolled the next distribution, but modules that use Drupal's registry can generate fatal errors when moved. Drush rr is helpful in some cases, but in others it requires manually clearing the cache tables and potentially even updating rows in the registry table (https://drupal.org/node/1974964).

@seanr

This comment has been minimized.

Copy link

seanr commented Oct 30, 2013

As far as the developers getting switched to a project and just updating everything... when is that OK?!?

It's not, but do you have any idea how often I've seen crap like that happen? The problem is when you've got a large group of people of varying skill sets and even a small amount of turnover with a large column of projects over time, you can lose control because the continuity isn't there. This is particularly a problem when a project done a year or two ago resurfaces for additional features or bug fixes.

@kreynen

This comment has been minimized.

Copy link

kreynen commented Oct 30, 2013

@seanr Wouldn't handling updates to modules in a profile by only updating the entire profile unless the module has been added to sites/all only help with the rogue, update-happy, developers you work with? Only allow senior level developers to make updates to your profiles. Let the jr. level devs build on that?

@seanr

This comment has been minimized.

Copy link

seanr commented Oct 30, 2013

In theory, but that only helps if you know it was done that way in the first place. Even I have gotten caught by that before on a site I myself previously worked on, but it'd been so long I'd forgotten it was set up that way (I didn't do the initial setup in that case).

Basically, my point here is it would seem to me like Drush should error out here. Since we say we should never allow updates in this fashion, why does Drush allow it? Or, is there a way for drush to update an entire profile as a contiguous unit rather than seeing its modules as separate items? This is really what I was trying to get at with my original report.

@kreynen

This comment has been minimized.

Copy link

kreynen commented Oct 30, 2013

I'm thoroughly confused. In your OP you included a quote from @weitzman that "community has not decided what it means for modules to be packaged under /profiles/[profilename]". My original response was that the developers building profiles have decided what that means as well objecting to your suggestion that "Otherwise, maybe it could prompt about updating individual modules within the profile."

Drush should only update the entire profile/distribution... which seems possible since it's listed in the system table and and already checked by update status. Rather than give users the option to update the module inside the /profiles, I suggest overriding outdated modules in the distribution by placing an additional copy in sites/all/modules/contrib.

Does that make sense or do you still believe drush should be updating modules inside /profiles/[profilename]?

@weitzman

This comment has been minimized.

Copy link
Member

weitzman commented Oct 31, 2013

OK, lets explore a bit the @kreynen proposal where Drush can only update a profile as a whole. There are two key pieces that I can see.

  1. Folks who want to update a part of a distro have to download the module and put it in sites dir. drush pm-download would work for that (with a custom --destination) but updatecode would not (because projects inside the profile dir are ignored).
  2. Updating profile as a whole. I see that we already have some pieces in place. For example, we know how to use profiles with updatestatus (e.g. dr pm-releases openatrium) and we can download their projects (dr dl openatrium --variant=projects). Not sure how we should put it all together though. Maybe @jonhattan has some ideas.
@nedjo

This comment has been minimized.

Copy link

nedjo commented May 7, 2014

lets explore a bit the @kreynen proposal where Drush can only update a profile as a whole

Yes, what distro users tend to expect is that they can use drush to update the distro like they would with any other code. There are complex workarounds like http://www.acquia.com/blog/maintaining-your-installed-drupal-distro, but that's clearly way too much for most users....

Part of our challenge is that a profile may be packaged in one of two ways on d.org, depending on the presence or absence of a drupal-org.make file:

  • If absent, as a profile to be added to profiles.
  • If present, as a fully packaged distro including Drupal core.

So supporting profiles in drush up would require first distinguishing between the two. Would this require new info from the d.org update feed? Or would we examine the extracted archive and determine that way?

The profile on its own is of course the simplest: replace profiles/profilename with the download, much like we replace e.g. sites/all/modules/contrib/modulename with a module.

For the distro case, a normally constructed distro includes two basic pieces:

  • A (potentially patched) version of Drupal core.
  • The profile code in profiles/profilename.

So maybe this is best handled as a combination of a regular profile update and a Drupal core update? Steps might include:

  • Extract distro archive.
  • Replace existing profiles/profilename with new version from extracted archive.
  • Treat the remaining as a Drupal core update.
@weitzman

This comment has been minimized.

Copy link
Member

weitzman commented May 7, 2014

@nedjo plan sounds reasonable to me. i won't be working on it personally but am happy to review PRs.

@kreynen

This comment has been minimized.

Copy link

kreynen commented May 8, 2014

@nedjo If you are going to be working on this, I'd like to contribute. I've been looking at Drush and https://drupal.org/project/drupalorg_drush to see how difficult it would be to package distributions w/ inheritable install profiles on Drupal.org (https://drupal.org/node/2067229). IMHO, solid base distributions that are designed more like base themes and packaged with profiles that use them as a parent is the next logical step in the evolution of distributions/install profiles.

There are several great base/starter distributions now, but the options to share work built on theme are either to nest the .make of the parent into the next distribution or only use sites/all. There are already examples of nesting the majority of Commerce, Panopoly, Red Hen, and CiviCRM in another distribution. This makes starting the new distribution much easier since you are focusing on customizations on top of the work that was already done, but makes maintaining the distributions more complicated since you have to constantly compare the new .makes, .profile, and .install to what the parent distribution has changed with each release.

@nedjo may have already experienced this with the Red Hen elements you are using in Open Outreach. One way around this is to develop a profile designed to be a "base" with all functionality in a module included in the profile and leave the .profile and .install essentially empty, but that really limits what can be done with a base distribution "out of the box" as well as fails to track it's usage. Without being able to demonstrate some compelling functionality on install and allow site builders to use a base distribution the same way they would a normal Drupal install by adding modules and themes to sites/all, it's unlikely a base would gain any traction.

There is also a third type of distribution that will likely cause problems. It's not something I'd recommend supporting, but just be aware that they exist. Distributions like https://drupal.org/project/tb_sirate_starter have simply committed duplicate versions of all the modules and Drupal core. I've been trying to get those distributions fixed or removed (or at least unpublished) for > 6 months. They are clearly violations of Drupal.org's Git policy, the third party code policy, and in many cases included code that isn't GPLv2 compatible. In addition, this approach prevents the distribution security warnings now displayed on the project pages from working. Despite all of that, there doesn't seem to be anyone involved w/ running Drupal.org who is willing to remove them.

The reason I bring up inheritable profiles is this makes it less clear which profile's core should be used. Off the top of my head, I'd think using core from the parent of the active profile would make sense... not sure if this could support another level for something like Red Hen (base) > Open Outreach (child of Red Hen) > Community Media Outreach (child of Open Outreach) or Panopoly (base) > Commons (child of Panopoly) > ELMS Commons (child of Commons). That starts to make my head hurt.

@greg-1-anderson

This comment has been minimized.

Copy link
Member

greg-1-anderson commented Jul 16, 2018

pm-* commands are in maintenance only. Closing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment