CiviCRM in profiles is *not* found, but installed, although without CiviCRM cron

[sluc23]

CiviCRM is being installed when is located in
sites/all/modules
sites/all/modules/contrib

But when is located in profiles, it is found but not installed
profiles/<profile_name>/modules
profiles/<profile_name>/modules/contrib

I'm not sure if it's a provision_civicrm issue, but was working in the previous branch with the backport I commited

[omega8cc]

Ouch, it worked in the first stage of our over 2-days long debugging. Not sure why it no longer works, but I suspect we have manage to re-add some regression in our hardened Drush version.=

[omega8cc]

We have determined that it was the upstream merge-in we did, which effectively removed this feature from provision_civicrm. Not nice.=

[omega8cc]

Further investigation revealed that the problem must be elsewhere. We have tested this with vanilla Drush 6 and vanilla provision_civicrm and it still doesn't work, because provision_civicrm depends on the packages identified by Provision on platform verify, and it is Provision what doesn't detect civicrm, unless it is in sites/all/modules, which suggests a problem with Provision itself, ugh..

[omega8cc]

We have added some workarounds in provision_civicrm during debugging, so now CiviCRM is installed, even if not found and not registered on platform verify task, if it exists somewhere else than in sites/all/modules

The only downside currently is that the site will not be present in Aegir as a 'real' CiviCRM site, so it will not run CiviCRM cron for it. But otherwise it works just fine, can be cloned etc.

[omega8cc]

This is a work in progress, but for now we must assume that CiviCRM is fully supported only when located in the sites/all/modules tree. We will get back to it later, as time permits. Any feedback and debugging is welcome.

[sluc23]

Ok, I will take a look if I can help you out with this.
For us is a blocker issue because we started to use CiviCRM in profiles/ in our current platforms, so until is fixed we won't be able to migrate to BOA 2.3.x

[omega8cc]

Another surprise was that the even the old provision_civicrm boa-2.2.9-dev (which worked fine with civicrm in profiles and Drush 4 on BOA-2.2.9), still works with BOA-2.3.1 under modified Drush 6 and civicrm sites are properly installed no matter if the module is in sites/all/modules or in the profile space. Just no extra cron in Aegir is set up on sites with civicrm in the profiles tree -- that is the only difference, so maybe we could develop a workaround for this.

This further suggests that the problem could be related to Provision itself, which simply doesn't detect civicrm properly when located in the profiles tree, while in BOA-2.2.9 it still worked fine.

[omega8cc]

We are reverting BOA to use provision_civicrm boa-2.2.9-dev and will try to fix the missing extra cron issue.

[omega8cc]

Ah, it should be as simple as disabling hosting_civicrm and enabling the old good hosting_civicrm_cron again.

[omega8cc]

Actually, such revert doesn't fix the missing extra cron feature auto-magically, but everything else works.

[omega8cc]

Posted this also on https://www.drupal.org/node/2339543

[omega8cc]

It is even more weird. Even if CiviCRM is not listed in the packages list on the platform when it exists in the profiles tree, it is listed in the packages list for the site created on that platform.

[omega8cc]

What we have tested here without much thinking really, was simply moving civicrm into profile space, but without actually adding it to profiles/$name/$name.info so how could we expect Provision to find it in the platform before the Drupal is bootstrapped? That was just a silly mistake. Please test current BOA head with civicrm in the profile space, but listed in profiles/$name/$name.info

[omega8cc]

Well, at least this is how it looks to me, but in fact, Provision should detect and register all modules physically present in the profile tree, no matter if added to profiles/$name/$name.info or not, Hmm.. more testing is needed, I guess.

[omega8cc]

Ongoing debugging is reported now at: https://www.drupal.org/node/2339543

[omega8cc]

It is a showstopper for BOA-2.3.2 we need to release asap.

[sluc23]

Hi, thanks for the hard debug work.
I got a little bit lost in which component of the stack the problem is now.

In our case, we have no problem to stay in legacy mode using 2.2.9 until this is fixed, so you can move forwards with another relevant issues to release BOA 2.3.2 (I would remark in the CHANGELOG the CiviCRM compatibility status on that release to adivse CiviCRM users)

[omega8cc]

I think that most BOA users use the classic sites/all/modules tree for CiviCRM, so they wouldn't even notice any issue, but if anyone already switched to use it in profiles, that could be a very serious problem, because there is no downgrade back to 2.2.9 possible, so we really need to get it right now, before we will run upgrades on hosted and managed systems.=

[omega8cc]

It is even worse, because it no longer works for D6, even if CiviCRM is in sites/all//modules

[mlutfy]

The Drupal 6 issue is resolved in CiviCRM 4.4.8 & 4.5.1 (unreleased). c.f. d.o#2339543 and CRM-15331.

I did a quick test using an install profile (https://www.drupal.org/project/civicrm_starterkit) and it seemed OK. However, I had to patch that install profile to remove the civicrm dependancy (in civicrm_starterkit.info) so that it does not enable CiviCRM too early in the process, otherwise it outputs an error regarding the "system" db table not existing.

"CiviCRM: civicrm is in /var/aegir/platforms/civicrm_starterkit/profiles/cm_starterkit_moderate/modules/civicrm"

[omega8cc]

Thank you for this update. Good to know it is something which needs a fix on the CiviCRM side for D6 support.

[omega8cc]

We have released BOA-2.3.2 with some caveats added and auto-protection from upgrading when anyone is using unsupported directory structure: https://github.com/omega8cc/boa/blob/2.3.x-dev/CHANGELOG.txt#L13

[LunkRat]

I have BOA 2.3.5 with a Drupal 7.33.1 custom platform with CiviCRM 4.5.3 installed to sites/all/modules. CiviCRM cron will not run. I am looking into this issue because the changelog for BOA 2.3.3 states "CiviCRM support for Drupal 7 works great when added in sites/all/modules". Are there steps I can take to troubleshoot and resolve this? Thank you for working to support CiviCRM platforms.

[mlutfy]

Patch needing review here:

https://www.drupal.org/node/2387173#comment-9436571
http://cgit.drupalcode.org/hosting_civicrm/commit/?id=0bbe64a

[sluc23]

https://www.drupal.org/node/2364871

I've tested and it works fine to install CiviCRM on /profiles for CiviCRM +4.4.8 (older CiviCRM versions
has a drush related bug)
Any plan to include this provision_civicrm in next BOA release?
if it's included the warning when civicrm is present in /profiles must be removed

[omega8cc]

We have added latest hosting_civicrm 6.x-2.x and provision_civicrm 6.x-2.x in BOA head and BOA-2.4.x branch, so it will be included both in BOA-2.3.9 and BOA-2.4.0 expected very soon.

Thank you for your work on this!

[omega8cc]

Feel free to reopen if there will be any further action required.

[sluc23]

thanks @omega8cc . I will focus provision_civicrm testing on BOA-2.4.x branch.
Since we are still using legacy BOA -2.2.x and 2.4.x is close in the horizon, would be a double effort to test it in 2.3.9 as well.

I'll keep this thread updated if I find any issue

[omega8cc]

OK, thanks. Please note that there will be no 2.3.9 release anyway, and the next release will be 2.4.0

[omega8cc]

We have tested this again in BOA head with latest civicrm-4.5.6 which supports Drush 7, in sites/all and in profiles, and everything seems to work just fine. Older versions need this patch to make them Drush 7 compatible: cf06f92

[omega8cc]

By the way, integration extension needs this patch (included in BOA): omega8cc/provision_civicrm@87003dd

[sluc23]

AEgir tasks work ok (Install, Verify, Clone, Migrate) tested in /sites/all & /profiles with CiviCRM 4.5.6 and CiviCRM 4.4.12 (patched for drush7)

I cannot get it working civicrm cron in /profiles. Regardless in AEgir site UI it says that the CiviCRM cron was executed, in CiviCRM side nothing is triggered.
CiviCRM Cron works fin in /sites/all

I'll keep investigating this issue... almost there :)

[sluc23]

After a long debugging I found that hosting_civicrm_cron.module, line 55 is returning an error when invoking drush civicrm-api in sites with civicrm in /profiles:

$result = provision_backend_invoke($site_name, "civicrm-api", array('--user=1', 'Job.execute', 'auth=0'));

returns: {"DRUSH_COMMAND_NOT_FOUND":["The drush command 'civicrm-api Job.execute auth=0' could not be found. Run drush cache-clear drush to clear the commandfile cache if you have installed new extensions."]}

Basically the user o1 (the one that executes civicrm_cron) doesn't find the drush civicrm commands in web sites with civicrm in /profiles.
I did the test, logged in as o1 :

o1@aegir:~/static/civicrm_in_sites_all$ drush cvapi
Array ( [is_error] => 1 [error_message] => API (, ) does not exist (join the API team and implement it!) ) --> (Found it!!)

o1@aegir:~/static/civicrm_in_profiles$ drush cvapi
The drush command 'cvapi' could not be found. Rundrush cache-clear drushto clear the commandfile cache if you have installed new extensions. [error] A Drupal installation directory could not be found [error]

Funny part is the user o1.ftp does find the drush civicrm commands in web sites with civicrm in /profiles, I'm not sure why o1 doesn't

[omega8cc]

This must be a result of our custom Drush hardening.

We do this to limit/close various security holes in the stock Aegir/Provision/Drush.

Can this cron be run via web request instead of Drush command?

[omega8cc]

Please create an empty control file: /data/conf/vanilla_drush.inc Then test if this makes any difference on BOA-2.4.0

[sluc23]

yes, adding the file /data/conf/vanilla_drush.inc did the trick 👍
Now the civicrm cron is executed in /sites and /profiles..