Pantheon ddev pull fails after download, during import. #670

ultimike · 2018-02-22T00:35:05Z

What happened (or feature request):

I tried to do a ddev pull, got the following error:

~/Sites/artist [master*]$ ddev pull
You're about to delete the current database and files and replace with a fresh import. Would you like to continue (y/N): y
Importing database... 
Could not perform import: Failed to run docker-compose [-f /Users/michael/Sites/artist/.ddev/docker-compose.yaml exec -T db bash -c cat /db/*.sql | mysql], err='exit status 1', stdout='ERROR 1227 (42000) at line 6894: Access denied; you need (at least one of) the SUPER privilege(s) for this operation
', stderr=''

I had just done a ddev start on an existing codebase that I hadn't used DDEV for previously.

What you expected to happen:

The pull to be complete.

How to reproduce this:

I could share the project with someone, I suppose...

Version: Please include the output of `ddev version`, `docker version` and the project's .ddev/config.yaml.

~/Sites/artist [master*]$ ddev version
domain	ddev.local                     
cli   	v0.13.1                        
web   	drud/nginx-php-fpm-local:v0.9.3
db    	drud/mariadb-local:v0.7.1      
dba   	drud/phpmyadmin:v0.2.0         
router	drud/ddev-router:v0.5.0        
commit	v0.13.1

~/Sites/artist [master*]$ docker version
Client:
 Version:	17.12.0-ce
 API version:	1.35
 Go version:	go1.9.2
 Git commit:	c97c6d6
 Built:	Wed Dec 27 20:03:51 2017
 OS/Arch:	darwin/amd64

Server:
 Engine:
  Version:	17.12.0-ce
  API version:	1.35 (minimum version 1.12)
  Go version:	go1.9.2
  Git commit:	c97c6d6
  Built:	Wed Dec 27 20:12:29 2017
  OS/Arch:	linux/amd64
  Experimental:	true

~/Sites/artist [master*]$ cat .ddev/config.yaml 
APIVersion: "1"
name: artist
type: drupal7
docroot: ""
php_version: "7.1"
webimage: drud/nginx-php-fpm-local:v0.9.3
dbimage: drud/mariadb-local:v0.7.1
dbaimage: drud/phpmyadmin:v0.2.0
router_http_port: "80"
router_https_port: "8443"
provider: pantheon

# Certain ddev commands can be extended to run tasks before or after the ddev
# command is executed.
# See https://ddev.readthedocs.io/en/latest/users/extending-commands/ for more
# information on the commands that can be extended and the tasks you can define
# for them.
# hooks:
#   post-import-db:
#     - exec: "drush cc all"

Anything else do we need to know:

Nothing comes to mind.

Please use a complexity rating of 1-5 (5 is high) for a feature request. (High complexity implies more PR planning)

The text was updated successfully, but these errors were encountered:

rfay · 2018-02-22T00:43:53Z

Wow, that looks like it's trying to use a different database than "db" in the "db" container. Please check to see that the settings.php is in fact loading settings.local.php, where we overload it, and that the settings.local.php has our contents in it - it should say #ddev-generated in there. But I'm thinking that all happens after the db is loaded.

One thing to try is to rm -r .ddev, and then do the ddev config pantheon again. In case there's something wrong with the docker-compose.yml. I don't think that's the case at all.

I'm happy to take a look if it's appropriate to share with my pantheon account, randy at randyfay.com

ultimike · 2018-02-22T01:04:41Z

Well - I think you're on the right track because the settings.php doesn't have anything about the settings.local.php, nor is there a settings.local.php (did I mention this is a Drupal 7 site?)

Still make sense to remove .ddev and re-config it?

-mike

rfay · 2018-02-22T02:12:28Z

It doesn't do any harm to do re-config. Doesn't do any harm to study it from the settings.php end either.

I'm pretty baffled by that error message though. Almost sounds like you have a pantheon dump that is specifying a particular database name.

One thing I'd try is doing a new archive/dump/backup of both files and db on Pantheon's end.

ultimike · 2018-02-22T17:13:28Z

At what point should the settings.local.php appear? During the ddev config pantheon step? If so, it's not happening...

Once it was complete, I did a ddev start, then a ddev pull, with the same result as above.

I checked, and it looks like the DB was imported, so I manually created a settings.local.php file and now everything appears to be working.

One caveat that may explain the weirdness: this site has some SQL views that get created via a custom drush command. I have had issues importing this database previously (I didn't think about that until today).

It might make sense to just close this issue, as it is entirely possible the site's DB SQL views are to blame.

rfay · 2018-02-22T17:19:55Z

It actually appears after the import of the db, and I don't think you're getting that far.

sql views sounds like a great culprit! Again, if it's possible and you're willing to share the db or the pantheon site, I'll try it.

I wonder if we need some extra permissions for the views? Is it possible the views refer to a different db?

Thanks for all your work on this.

ultimike · 2018-02-22T18:41:19Z

My client isn't keen on sharing the codebase - sorry.

The SQL Views have always been a bit of a pain to deal with - including some weird permission stuff. Would it explain the settings.local.php not being generated as well?

rfay · 2018-02-22T18:43:46Z

Yes, the failure to load means that ddev never gets to the point where it generates the settings.local.php.

It's a lot to ask, but if you know how to put together a random database that generates this, I'd love to figure out how to solve this. But it does seem like you've got it here.

We can close this when it works for you, but my inexperience with sql views makes me insecure about the reason for this failure. Glad it's working for you though!

ultimike · 2018-02-22T18:55:52Z

I wish I could say that I had to the time to put together a quick DB for you, but I'm pretty swamped for the next few weeks... :(

I can tell you that we have a drush command to create the SQL views that looks something like this:

function mymodule_drush_command() {
  $items = array();

  // The 'create-sql-views' command
  $items['create-sql-views'] = array(
    'description' => "Re-create SQL Views.",
  );
  return $items;
}
function drush_mymodule_create_sql_views() {
  _mymodule_create_sql_views();
}

function _mymodule_create_sql_views() {
  $sql = "CREATE OR REPLACE VIEW {this_is_the_sql_view_name} AS select blah blah blah;";
  db_query($sql);
};

Hopefully that's enough to get you going.

rfay · 2018-02-22T18:56:41Z

Thanks!

rfay · 2018-03-14T16:10:33Z

My bet is that this will be solved in #693

rfay · 2018-04-16T17:45:53Z

Closing this one unless we hear otherwise, assuming #693 solves it. Note that this was a D7 site though, which by default would not have the include of settings.local.php, which could cause this kind of breakage. Happy to reopen!

texvetadmin · 2018-05-01T14:22:52Z

Howdy! I just recreated this issue. It's (almost) exactly the same (The line that the error occurs on differs from the original poster's issue.):

TexVet-Leistiko-iMac:texvet jonathanleistiko$ ddev pull
You're about to delete the current database and files and replace with a fresh import. Would you like to continue (y/N): y
Importing database... 
ERROR 1227 (42000) at line 2639: Access denied; you need (at least one of) the SUPER privilege(s) for this operation 
Could not perform import: Failed to run docker-compose [-f /Users/jonathanleistiko/Documents/local_dev/ddev_texvet_local/texvet/.ddev/docker-compose.yaml exec -T db bash -c cat /db/*.sql | mysql -udb -pdb db], err='exit status 1', stdout='', stderr='ERROR 1227 (42000) at line 2639: Access denied; you need (at least one of) the SUPER privilege(s) for this operation' 
TexVet-Leistiko-iMac:texvet jonathanleistiko$

Docker is running. I'm certain that the DB's name is "pantheon" – Pantheon's Connection Info > Database > DB Name says so.

My organization is willing and able to share access to our Pantheon site/database/etc. In the meantime, I'm working through the notes above (and suggestions in other articles) to see if I can get the local copy of the site working correctly.

Extra info that may or may not be relevant: We did not set up the Pantheon site ourselves. Instead we hired Kalamuna to migrate an existing site from Blackmesh to Pantheon. Part of that included getting CiviCRM to work correctly. That involved setting up some symbolic links or somesuch.

Version:

Mac OS 10.13.4

TexVet-Leistiko-iMac:texvet jonathanleistiko$ ddev version
domain ddev.local
cli v0.17.0
web drud/nginx-php-fpm-local:v1.2.2
db drud/mariadb-local:v0.9.0
dba drud/phpmyadmin:v0.2.0
router drud/ddev-router:v0.5.0
commit v0.17.0

TexVet-Leistiko-iMac:texvet jonathanleistiko$ docker version
Client:
Version: 18.03.1-ce
API version: 1.37
Go version: go1.9.5
Git commit: 9ee9f40
Built: Thu Apr 26 07:13:02 2018
OS/Arch: darwin/amd64
Experimental: false
Orchestrator: swarm

Server:
Engine:
Version: 18.03.1-ce
API version: 1.37 (minimum version 1.12)
Go version: go1.9.5
Git commit: 9ee9f40
Built: Thu Apr 26 07:22:38 2018
OS/Arch: linux/amd64
Experimental: true

Complexity Rating: 2?

rfay · 2018-05-01T16:22:33Z

@texvetadmin thanks for the report. If there's any possibility of sharing the pantheon site with me for debugging it would be marvelous. randy (at) randyfay.com. Does yours have any SQL views in it?

texvetadmin · 2018-05-01T16:31:10Z

I just shared our Pantheon site with you.
I hate to admit this, but I'm not sure what you mean when you ask if it has any SQL views in it. We're using the Views module and we have many views that make SQL queries, so I'm inclined to say "Yes, it has SQL views.," but I may be misunderstanding what you're asking.

rfay · 2018-05-01T18:45:25Z

Thanks for that! I'm able to recreate the issue. The problem line is

/*!50003 CREATE*/ /*!50017 DEFINER=pantheon@%*/ /*!50003 TRIGGER civicrm_activity_before_insert before insert ON civicrm_activity FOR EACH ROW BEGIN

Since we GRANT ALL to the db user, I'm studying to try to figure out what we're missing here.

rfay · 2018-05-01T20:09:52Z

Please try this @texvetadmin:

ddev start
ddev ssh and mysql -uroot -proot and GRANT SUPER ON *.* TO 'db'@'localhost' IDENTIFIED BY 'db'; and FLUSH PRIVILEGES;
Exit back out to host
ddev pull

I see that you have a huge db and files there. If you want to just import the db you can ddev import-db --src ~/.ddev/pantheon/texvet/texvet_dev_<latest>_*sql.gz

If this works for you (works for me) we'll consider that as just a routine change.

I do note these complaints on the file import, but neither macOS tar nor gnu-tar does any better on the tarball provided by pantheon:

Could not perform import: failed to extract provided archive: Failed to create file /Users/rfay/workspace/texvet/sites/default/files/civicrm/templates_c/en_US/%
/0A1/0A1AE998%%string:2101+S+IH-35,+Suite+216,+Austin,+TX+78741+from+2:30+to+4.

The+Power+Partners+are+hosting+a+job+club+where+you+will+learn+useful+tips+for+your+job+search.+You+can+also+talk+with+our+featured+employers+and+Veteran+Service+Organization+providers.+Feel+free+to+bring+your+resume+for+review..php, err: open /Users/rfay/workspace/texvet/sites/default/files/civicrm/templates_c/en_US/%
/0A1/0A1AE998%%string:2101+S+IH-35,+Suite+216,+Austin,+TX+78741+from+2:30+to+4.

The+Power+Partners+are+hosting+a+job+club+where+you+will+learn+useful+tips+for+your+job+search.+You+can+also+talk+with+our+featured+employers+and+Veteran+Service+Organization+providers.+Feel+free+to+bring+your+resume+for+review..php: file name too long

texvetadmin · 2018-05-01T21:12:06Z

I'm giving your recommended commands a go now. I'm up to "ddev pull"
After "thinking" for a good chunk of time, it's on "Generating settings.local.php file for database connection." I'll let you know how it goes.
I sincerely appreciate your assistance. Thank you very much!

texvetadmin · 2018-05-02T13:07:23Z

Short answer: It worked. Thank you!

Longer answer: It failed. Boo! But it's totally an issue specific to the files. There's a file with a file name that's too long. I have to track it down and delete it.

Importing files... 
Could not perform import: failed to extract provided archive: Failed to create file /Users/jonathanleistiko/Documents/local_dev/ddev_texvet_local/texvet/sites/default/files/civicrm/templates_c/en_US/%
/0A1/0A1AE998%%string:2101+S+IH-35,+Suite+216,+Austin,+TX+78741+from+2:30+to+4.

The+Power+Partners+are+hosting+a+job+club+where+you+will+learn+useful+tips+for+your+job+search.+You+can+also+talk+with+our+featured+employers+and+Veteran+Service+Organization+providers.+Feel+free+to+bring+your+resume+for+review..php, err: open /Users/jonathanleistiko/Documents/local_dev/ddev_texvet_local/texvet/sites/default/files/civicrm/templates_c/en_US/%
/0A1/0A1AE998%%string:2101+S+IH-35,+Suite+216,+Austin,+TX+78741+from+2:30+to+4.

The+Power+Partners+are+hosting+a+job+club+where+you+will+learn+useful+tips+for+your+job+search.+You+can+also+talk+with+our+featured+employers+and+Veteran+Service+Organization+providers.+Feel+free+to+bring+your+resume+for+review..php: file name too long 
TexVet-Leistiko-iMac:texvet jonathanleistiko$

I have little doubt that once I track that down (and other ridiculous files like that one), it'll work just fine.

rfay · 2018-05-02T13:29:45Z

Opened drud/mariadb-local#55 to solve this generally.

BTW @texvetadmin I don't think those long filenames actually cause a failure that would be in your way; take a look and see if most everything you need isn't there in sites/default/files. Mostly with big sites like that having every single one of the files is a waste. People have requested a ddev pull --database-only in #405, I imagine we'll get that going.

rfay · 2018-05-24T13:47:22Z

Closing this as a duplicate of [https://github.com//issues/852](Use mysql root user for imports #852) (which was formerly https://github.com/drud/mariadb-local/issues/55). I'm pretty sure that when we start importing using the root-privileged mysql user this will be solved.

…ixes #853, fixes #810, fixes #670 (#891)

rfay · 2018-05-31T12:56:24Z

@texvetadmin @ultimike I think both of your problems should be fixed in today's v0.19.0 release, and hope you'll be able to try it out and confirm. @texvetadmin thanks to your generosity in sharing your pantheon project I was already able to give a quick run on yours and it worked out fine. Thanks for that!

rickmanelius added this to the v1.4.0 milestone Feb 23, 2018

rickmanelius added the actionable label Feb 23, 2018

rfay closed this as completed Apr 16, 2018

rfay reopened this May 1, 2018

rfay mentioned this issue May 14, 2018

Use mariadb root user for db imports for drud/ddev#670 #852

Closed

dclear modified the milestones: v1.4.0, v1.3.0 May 17, 2018

rfay closed this as completed May 24, 2018

rfay mentioned this issue May 30, 2018

Make mariadb root user work, drop tables before import, fixes #852, fixes #853, fixes #810, fixes #670 #891

Merged

rfay added a commit that referenced this issue May 30, 2018

Make mariadb root user work, drop tables before import, fixes #852, f…

881ed26

…ixes #853, fixes #810, fixes #670 (#891)

dclear assigned rfay May 30, 2018

dclear removed the actionable label May 30, 2018

therobyouknow mentioned this issue Sep 28, 2021

DX: ddev import of db fails with error: "ERROR 1071 (42000) at line 59: Specified key was too long; max key length is 767 bytes" NaturalHistoryMuseum/scratchpads2#6413

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Pantheon ddev pull fails after download, during import. #670

Pantheon ddev pull fails after download, during import. #670

ultimike commented Feb 22, 2018

rfay commented Feb 22, 2018

ultimike commented Feb 22, 2018

rfay commented Feb 22, 2018

ultimike commented Feb 22, 2018

rfay commented Feb 22, 2018

ultimike commented Feb 22, 2018

rfay commented Feb 22, 2018

ultimike commented Feb 22, 2018

rfay commented Feb 22, 2018

rfay commented Mar 14, 2018

rfay commented Apr 16, 2018

texvetadmin commented May 1, 2018

rfay commented May 1, 2018

texvetadmin commented May 1, 2018

rfay commented May 1, 2018

rfay commented May 1, 2018 •

edited

Loading

texvetadmin commented May 1, 2018

texvetadmin commented May 2, 2018 •

edited

Loading

rfay commented May 2, 2018

rfay commented May 24, 2018

rfay commented May 31, 2018

Pantheon ddev pull fails after download, during import. #670

Pantheon ddev pull fails after download, during import. #670

Comments

ultimike commented Feb 22, 2018

What happened (or feature request):

What you expected to happen:

How to reproduce this:

Version: Please include the output of ddev version, docker version and the project's .ddev/config.yaml.

Anything else do we need to know:

Related source links or issues:

Please use a complexity rating of 1-5 (5 is high) for a feature request. (High complexity implies more PR planning)

rfay commented Feb 22, 2018

ultimike commented Feb 22, 2018

rfay commented Feb 22, 2018

ultimike commented Feb 22, 2018

rfay commented Feb 22, 2018

ultimike commented Feb 22, 2018

rfay commented Feb 22, 2018

ultimike commented Feb 22, 2018

rfay commented Feb 22, 2018

rfay commented Mar 14, 2018

rfay commented Apr 16, 2018

texvetadmin commented May 1, 2018

Version:

Complexity Rating: 2?

rfay commented May 1, 2018

texvetadmin commented May 1, 2018

rfay commented May 1, 2018

rfay commented May 1, 2018 • edited Loading

texvetadmin commented May 1, 2018

texvetadmin commented May 2, 2018 • edited Loading

rfay commented May 2, 2018

rfay commented May 24, 2018

rfay commented May 31, 2018

Version: Please include the output of `ddev version`, `docker version` and the project's .ddev/config.yaml.

rfay commented May 1, 2018 •

edited

Loading

texvetadmin commented May 2, 2018 •

edited

Loading