MVP Integration w/Pantheon

[rickmanelius]

What happened (or feature request):

Feature Request

What you expected to happen:

Extend ddev to allow:

• Connection to Pantheon through its terminus API to access files and database archives.
• Download and import database and/or file archives.
• Collection and storage of Pantheon specific configuration information.

Wishlist of related items (moved to or referenced in other tickets)

• URL search/replace for WordPress (relies on "Create a Strategy for Pre/Post Hook Functionality")
• Use Pantheon specific webserver and database images.

Acceptance Criteria

• Pantheon plugin exists (noting this simply to indicate that, ideally, provider plugins are not simply mixed into the existing command files and are separate extensions).
• Pantheon specific container variants exist and (at a minimum) match webserver/mysql/php application selection and versions.
• ddev config pantheon results in an additional set of information placed in .ddev/config.yml namespaced by application type and the provider plugin name to include information needed to communicate with the Pantheon terminus API.
• ddev import pantheon available with the ability to specify which environment to pull the database and/or files from.

Wishlist related items (moved to other tickets referenced above)

• ddev config pantheon results in the docker-compose.yml files utilizing the pantheon specific images instead of the stock images.
• ddev import pantheon for a WordPress captures the production URL and performs a wp-cli search replace to the local database host.

Testing Criteria

• Adding the pantheon argument results in a docker compose file with Pantheon containers specified.
• Adding the pantheon argument results in additional data (terminus API credentials) to get stored in configuration.
• We can pull database and file archives directly from a Pantheon hosted site and confirm that we get a working site (this might get rate limited from Pantheon).
• The WP search/replace results in the homepage source containing no instances of the previous URL/host references.

Related source links or issues:

• ddev Roadmap
• As a developer, I want integration with 3rd party providers.
• Create Strategy for Pre/Post Hooks

[rickmanelius]

@beeradb Please scan, but I think this gets us an actionable ticket for a Pantheon MVP.

[beeradb]

The contents of this look good, but I would probably cut a few things out of phase 1 to make it more bite-sized. Those things specifically are:

1. custom pantheon containers. For a basic site, I don’t see any reason off the top of my head that our current containers won’t work. I agree we need to match environments as much as possible in the long run, but it’s a lot of additional work for (imho) relatively little gain right now.
2. (I feel like I’m killing your kitten) Post import stuff. This really gets into the hook system I think which would be nice to think through and architect rather than doing a 1 off for this story.

fwiw, I don’t think this is necessarily even pushing anything back, I think it’s mostly trying to make this huge story achievable in steps. as written, this is a ton of code and would probably keep a dev busy for a couple weeks. The smaller the steps we can make it, the more reviewable and achievable it will be.

I also feel like even just the first bullet in the story could be It’s own story for starters, Connection to Pantheon through its terminus API to access files and database archives. It might be wise to start by writing a go package that does all that for us as a first step.

[rickmanelius]

Quick response base on a convo in Slack.

• Happy to make this more minimal by moving the custom Pantheon containers and the WP search/replace to separate issues (the latter already has an issue referenced in the issue summary and could potentially be completed before this gets started, making this moot).

If that's the case, I can extract the other points to a wishlist and reference them in existing/new tickets.

@rfay @tannerjfco would love a quick sanity check in the next day or so, otherwise I would love to see this actionable by EOW given this could be an issue that takes some time to hammer through while the earlier milestones get completed.

[rickmanelius]

Updated the acceptance criteria. This is actionable IMHO.

[rickmanelius]

Need a package/test that precedes this. Specifically

• Auth tokens for Pantheon.
• Ability to download archives.

[rickmanelius]

Removing from active sprint while we are focusing on the precursor issue #229.

[beeradb]

Here's an initial stab at the CLI commands for an initial Pantheon integration.

Global config with interactive prompts

Valid config

I think we're going to need some notion of global config, because we don't want API tokens to be stored anywhere they might actually be committed. I was also hesitant to commit to multi-word (i.e. ddev config pantheon) commands
both here and elsewhere.

$ ddev pantheon-config
Please visit https://pantheon.io/docs/machine-tokens/ for instructions on creating a Pantheon machine token.

Machine Token: aaa
Authentication successful! Configuration has been saved to ~/.ddev/pantheon-config

Invalid Config

$ ddev pantheon-config
Please visit https://pantheon.io/docs/machine-tokens/ for instructions on creating a Pantheon machine token.

Machine Token: bbbb
Could not authenticate. Please ensure you have a valid machine token. Configuration has not been saved.

Global config with arguments

Valid Config

We only have a single value to config, so making it an argument feels natural. That being said, I'm not sure this scales to other implementations. It's hard to design for implementations we haven't seen though :)

$ ddev pantheon-config aaa
Authentication successful! Configuration has been saved to ~/.ddev/pantheon-config

Invalid config

$ ddev pantheon-config bbbb
Could not authenticate. Please ensure you have a valid machine token. Configuration has not been saved.

Site-Specific config

Global pantheon config is missing or incorrect

In this case, 'pantheon' is an argument to tell it which import source to use. I imagine import-config will be single command which takes the provider as a plugin, and then calls a method on the provider
config struct to take over any user interaction from there.

Invalid argument detected

$ ddev import-config some-unknown-service
Invalid option. 'some-unknown-service` is not a valid import type.

$ ddev import-config pantheon
No valid pantheon configuration exists. Please run `ddev pantheon-config` first.

Global pantheon config is missing or incorrect

I would have loved to do something more interactive with the site selection here, but it's hard to build a usable system in a CLI that would work for both 1 site and 100. I think in the end it's probably best to just make the site name input
free form and instruct them how to find it. I do think we could check to see if a site that matches their ddev local install exists and default to that. i.e. if your site is named "drudio" in both ddev and pantheon then attempt to prompt/default to that value.

I did not include it in these mock-ups, but I think it probably makes sense to ask if they'd like to run an import directly after configuring. Perhaps just a "you may now run ddev import" message would suffice.

$ ddev import-config pantheon

Configuring Pantheon imports.

Site name: some-invalid-site
'some-invalid-site' could not be found. Please use 'terminus site:list' or visit your Pantheon dashboard find available sites.

Site name: drudio
Site found! Finding available environments.

- live
- dev
- staging
- bbowman

Please choose an environment: dev

You've chosen to import from the dev environment on the drudio site. Your configuration has been written to ~/Projects/drudio/.ddev/import.yaml. You may now run `ddev import`.

$ cat ~/Projects/drudio/.ddev/import.yaml
---
platform: pantheon
site-name: drudio
environment-name: dev

Running import

Running import with missing/invalid configurations

Missing configuration.

$ ddev import
No import config found.

Site not found.

$ cat ~/Projects/drudio/.ddev/import.yaml
---
importer: pantheon
site-name: some-invalid-site
environment-name: dev

$ ddev import
Site 'some-invalid-site' was not found.

Environment not found.

$ cat ~/Projects/drudio/.ddev/import.yaml
---
importer: pantheon
site-name: drudio
environment-name: some-invalid-env

$ ddev import
Environment 'some-invalid-env' was not found for site drudio.

Running import with valid configuration

Need to ensure the language on this command is suitably scary. It is definitely a destructive action.

We also need to make decisions on whether to keep the backups we pull down around or not. I think it probably makes sense to keep them locally and then prompt during import about whether to check for new backups.

$ ddev import

Importing will pull the latest files and database backup from Pantheon and import it into your running site. 

This action will overwrite any files and database modifications you have made locally and is not reversible. 

Continue (y/N)?: y

Pulling down latest files and database backups...
Importing...

Import complete! You can visit your site at http://drudio.ddev.local

[rfay]

I spent some time with Elysium today and had an excellent experience. Everything I tried worked as advertised, and casual review of the code shows solid, tested, quality code.

Comments:

• Elysium is extraordinarily fragile and risky unless we can get a working agreement with Pantheon. We could build an ongoing API test, but even then our customers would be broken before we detected and fixed our integration. We also have a hard time knowing what the actual varieties of responses are. This is a really, really important risk.
• The implementation has very nice test coverage, at least for this point in its life.
• Elysium does not yet contemplate a "push", like kbox does, it is only ready to grab downloads.
• Just for one minute It's worth considering whether we could do what we need just using terminus and either shell integration or something of the type.
• I do hope we can give the repo a name that reflects its usage. I know that's plain-jane behavior.

[rickmanelius]

Feedback

This is specifically related to the current state of the comment here #216 (comment).

Global Configurations

• Overall: I feel global configurations are an inevitability and I'm happy to see us moving in that direction.
• Security: Agreed that anything that shouldn't be stored in a repo belongs in the global config.
• Naming Convention: Given that other provider plugins and other config files may get added to this global area, should be adopt a config.PROVIDER.yml pattern? That would set them apart/namespace them.
• I'm assuming there is a 1:1 ratio between a Pantheon login and a machine token. What are the odds that someone would have multiple accounts and tokens? This is probably an edge case to ignore. However, I know I've had issues with this on Acquia w/multiple logins.

Multi-word Versus Hyphenated Words

(Ironically, "multi-word" is hyphenated whereas "hyphenated words" is not!)

• My overall preference is multi-word. If we hyphenate, and think about a future where we have 5-20 plugins, the command list going to become unwieldy. And because each plugin essentially extends and/or overrides the base functionality, it would seem to me that a ddev COMMAND PROVIDER ARGUMENT --flags approach would be the way to go, although I realize things get tricky with arguments without flags.
• If we stick with hyphenated, my preference would be to switch the order: ddev config-pantheon so they are namespaced/sorted together.

Site Specific Configurations

• Is it worth having two separate commands (one for global and one for site specific)? Given that global would likely only need to get set once (unless we need to deal with multiple tokens). What about noting that there is no token set and going through that first before proceeding with a site specific config? I know that overloading a single function can violate certain clean code principles, so this is more of an open question than anything.
• import-config or config-import? The former seems to imply we're downloading from Pantheon (e.g. we're importing a configuration) whereas the latter seems to indicate that we're configuring our imports (e.g. what should happen when I then run an import command?). Going back to the hyphenated versus multi-word approach, wouldn't this fall under the umbrella of "config" and therefore more generally solved by ddev config pantheon?
• Chicken/Egg. For Pantheon, I can imagine that the typical scenario is that the site is created first on Pantheon, then imported/developed locally. However, it's equally true that one has started to develop a site well before a hosting provider has been chosen and is then looking to modify the config to target that provider. It might be the case that one simply wants to develop locally against the pantheon stack prior to having anything to pull down at all. Need to think this through further...
• Import.yml. I love this idea, similar to the generation of a drush.inc file so that you don't need to re-specify your import configs. The question to ask is whether or not we should notify an end-user that we're going to use the default configs versus prompt to use the local plugin (which is kinda the default if we were to use ddev import). So either local is the default and to pull from Panthen we should specify ddev import pantheon or we need to provide the user with the way to toggle which particular import they are about to target.

[beeradb]

Responses to @rfay in #216 (comment)

Elysium is extraordinarily fragile and risky unless we can get a working agreement with Pantheon. We could build an ongoing API test, but even then our customers would be broken before we detected and fixed our integration. We also have a hard time knowing what the actual varieties of responses are. This is a really, really important risk.

This is absolutely correct. I believe there are factors in play that help mitigate this, but regardless, the risk is still there.

• Other tools (such as kalabox) are implementing the API.
• Terminus is also implementing the API. Any changes to the API would likely be made in terminus and pushed out before changes.
• Most of what we're doing are simple GET requests on REST endpoints. There's always the potential for those endpoints to change, but I'd be a lot more scared if we were doing POST/PUT's containing undocumented fields. To that end, one of my bigger concerns is the POST we're doing to get backup URLs which has very odd HTTP semantics.

That being said, even with the factors I listed there is absolutely a risk. Guarantees would be fantastic.

I do hope we can give the repo a name that reflects its usage. I know that's plain-jane behavior.

Agreed. Elysium was just chosen as a project name so we didn't bike shed. I'd much rather have it be descriptive and meaningful. I've reached out to Josh at Pantheon and asked him about any objections to the name.

[beeradb]

responding to @rickmanelius in #216 (comment)

Naming Convention: Given that other provider plugins and other config files may get added to this global area, should be adopt a config.PROVIDER.yml pattern? That would set them apart/namespace them.

My thinking was that a single config file with a "provider" key would likely be enough. I think if there's a use case for multiple import sources then having multiple configs might make sense, although we'd have to consider how to handle it in the ddev import command. So, I guess I'm open to it, but it'd be great to think through what the use cases are.

Is it worth having two separate commands (one for global and one for site specific)? Given that global would likely only need to get set once (unless we need to deal with multiple tokens). What about noting that there is no token set and going through that first before proceeding with a site specific config? I know that overloading a single function can violate certain clean code principles, so this is more of an open question than anything.

This would possibly work for initially setting the value, but what if it needed to be modified? It feels cumbersome to go through this on every site config.

import-config or config-import? The former seems to imply we're downloading from Pantheon (e.g. we're importing a configuration) whereas the latter seems to indicate that we're configuring our imports (e.g. what should happen when I then run an import command?). Going back to the hyphenated versus multi-word approach, wouldn't this fall under the umbrella of "config" and therefore more generally solved by ddev config pantheon?

ddev config is already owned by site config. I'm personally hesitant to overload it with import related configurations too. I'd be more open to the import-config command for setting providers, but I think mixing the semantics into a single ddev import command would probably get confusing.

Import.yml. I love this idea, similar to the generation of a drush.inc file so that you don't need to re-specify your import configs. The question to ask is whether or not we should notify an end-user that we're going to use the default configs versus prompt to use the local plugin (which is kinda the default if we were to use ddev import). So either local is the default and to pull from Panthen we should specify ddev import pantheon or we need to provide the user with the way to toggle which particular import they are about to target.

Currently there's no a ddev import, just a ddev import-files and ddev import-db. This is definitely a new semantic for a unified single import. It's worth considering if/how this fits into the current import paradigm.

[rfay]

@rickmanelius requested "technical/maintenance considerations that would make a provider plugin either in the main ddev codebase or to extend the ddev codebase untenable. Another factor to consider: how would not including this affect a GUI, which at this point assumes a CLI equivalent/parity":

First, nothing makes a provider plugin for Pantheon/Terminus untenable. I think Elysium proves it can be done well.

The maintenance issues to weigh are:

• We're talking about a fair bit of code here. Each bit costs. Then each bit goes on costing forever in maintenance.
• This is an integration with an external API which is subject to change and which we have only peripheral contact with. (In other words, we have done only a few test cases on a very few users, so don't have a complete picture of what might be in there.) Any time you add integrations to external APIs (which are wonderful of course) you commit yourself to maintaining them forever, and the maintenance and surprises never cease. And this is an undocumented API at this point. We might want to collaborate with them on an external spec if they'd maintain it?
• If I were the target dev and the command line were the mode, I'd be perfectly happy with using a variety of techniques to get artifacts to load, and I'd be happy without a push. We could document techniques to use terminus for download and then ddev to import. We could wrap those techniques with shell scripts or golang functions. With a technique like this the worst dependencies of the API would be sorted pretty well. However I'm not the target dev, and we just have to know who is.
• I'm a believer in GUIs that don't just wrap CLIs, but integrate directly with functionality. Kalabox seems to just wrap, and I think that's one of the reasons it's so unacceptable in its error reporting and overall capabilities. So if a GUI is going to really present a real interface and it's going to do a pull from Pantheon, it's my bet that my suggested download hacks above are unacceptable.

Oh and by the way @beeradb is certainly right that the risk is significantly mitigated by the fact that the terminus tool is maintained in public and if we follow it we should know what's going on.

On configuration style: It would be ever-so-cool if we could use the Pantheon config that terminus and kalabox already share. I'm not sure that's possible. That would make terminus, kalabox, and ddev interop quite nice. But I sure did like it when I rolled out terminus and it already understood everything based on what I'd done with Kalabox.

[beeradb]

I started reworking everything from above, but ended up backing out of that as I really feel the combination of ddev config and ddev import-config into a single command is actually the crux of it. The rest feels fairly cosmetic to me.

Global config and ddev import

The only change to these items has been changing ddev pantheon-config to ddev config-pantheon. I've omitted them here for brevity, as otherwise the original in #216 (comment) is still accurate for ddev import and ddev config-pantheon.

Site-Specific config

Global pantheon config is missing or incorrect

In these examples, I've combined pantheon config with the standard config.

Invalid argument detected

$ ddev config some-unknown-service
Invalid option. Valid services are: pantheon

Global pantheon config is missing or incorrect

$ ddev config pantheon
No valid pantheon configuration exists. Please run `ddev config-pantheon` first.

Working config example

This shows a combined ddev config and ddev import-config from the previous mock-ups. Overall, I feel like this streamlines and simplifies the configuration, but I will note it forces the ddev site name to match the pantheon site name. This has some drawbacks, chief amongst them some difficulty in getting
multiple versions of the same site running locally, I think it's worth the trade-off to simplify and streamline the config.

This also maintains separate YAML files (config.yaml and import.yaml) which have their own name keys. Advanced users could maintain the site name separation by manually configuring.

$ ddev config pantheon

Project name: some-project-name
Could not find a pantheon project named 'some-project-name'. 

Project name: drudio
Pantheon project drudio was found successfully.

The docroot is the directory from which your site is served. This is a relative path from your application root (~/Projects/drudio)
You may leave this value blank if your site files are in the application root
Docroot Location: docroot
Found a drupal8 codebase at ~/Projects/drudio/docroot

Configure import environment

- live
- dev
- staging
- bbowman

Please choose an environment: dev

You've chosen to import from the dev environment on the drudio site. Your configuration has been written to ~/Projects/drudio/.ddev/import.yaml. You may now run `ddev import`.

$ cat ~/Projects/drudio/.ddev/import.yaml
---
platform: pantheon
site-name: drudio
environment-name: dev

[rickmanelius]

Overall, this is trending in the right direction. I know this has the ability to get bike-shedded, and I would be willing to accept the last proposal versus arguing over details that may not matter and we can change/refactor based on user feedback. And while I know the linking to private Slack channel's is not ideal in public issue queues, there were some useful summaries worth referencing here https://drud.slack.com/archives/C45DF87JT/p1496871068545215.

That said, here are a few things to consider based on the last proposal #216 (comment).

• namespacing: Would config-pantheon appear in the command list for ddev? While I'm in agreement as per the summary posted in our Slack that we should bias towards releasing faster instead of finding a perfect way to grow the available plugins, I do want to keep the top level command namespace manageable. We can always change later, but just noting this preference for the long term even it doesn't affect this proposal.
• ddev import: I still there are a few hidden details to consider. Notably, import-db and import-files are interactive and focused on local imports. If we ultimately use import as a shortcut for import-files then import-db, then those interactive prompts would essentially stack. At that point, what would be the appropriate default for ddev import? Should it default to the local import? Or should it default to a provider plugin (if applicable)? Given that connection with a hosting provider is optional (as are plugins), I would vote that ddev import would default to local file imports and that to trigger an import from a provider, ddev import provider would be the way. Essentially, adopting the ddev import <provider> where is optional allows a consistent set of import expectations and the ability to target local/provider. It could also be extended to ddev import provider environment in order to override the default target environment.
• "config-pantheon" versus "config pantheon". The former is a global configuration setter and the latter is a site specific configuration. Given the single character difference, I could see this being a trip up that will hit a lot of people that aren't paying attention. It's also not easy to remember which one would be used for each case. I don't necessarily have a great solution for this, but would ddev config pantheon global be more accurate? It's not something that would have to be used often and it would allow a user to target this whether or not there were in a working directory. It might also resolve the namespace issue noted above and also set a pattern that's resuable. Or ddev config global provider intead of ddev config provider global if we want to help the user deliniate between site and global specific configs?

Again, I don't think any of these changes/comments necessitate holding back from moving on creating the first plugin. I'm surfacing these as things to consider from a usability/extendability/support perspective. Of course, user feedback would help us determine which are valid concerns and what items are not important.

[rickmanelius]

Last minute note before our meeting.

Perhaps a way to solve the site versus global configuration is to move to delineate that into two separate functions. Specifically:

• ddev config (defaults to site specific configuration)
• ddev config provider (site specific configuration for provider)
• ddev config-global (defaults to ddev global configurations/preferences)
• ddev config-global provider (global configurations/preferences for that plugin)

[rickmanelius]

@rfay @beeradb @tannerjfco and I had a conference call to discuss. The TL;DR is that we have consensus / buy-in to proceed knowing that we're still going to learn things along the way and will likely need to adjust. That said, here are some notes from the conversation.

Config

Overall we like the direction of separating the site-specific and global configuration (even if we don't have an immediate need for global ddev configuration). From a semantic perspective, it's probably best to change the command used to specific hosting provider configurations: "auth" and "connection" were two proposed terms. Summarizing:

• ddev config (core site/app specific configuration)
• ddev config provider (site specific configuration for provider)
• ddev a-global-config-command (global ddev configs/preferences; exact function name TBD)
• ddev provider-connect-command provider (provider configs/preferenes; exact function name TBD)

Import

For now, we're going to keep import specific to the provider plugin import and not overload this function. Those importing locally should use import-db and import-files for now.

Other future considerations:

• Should import-db and import-files be combined into import so that a person could specify files and db by flags and complete in one shot?
• Should we provide an ability to use a local cached copy of an archive (this will likely be a requirement for larger site archives)?
• Should be allow overriding of the default environment to target? Specify only files/db versus both?

This is now actionable.

[rickmanelius]

One admin item. The issue summary should be updated to reflect the conversation summary as well as other changes from @beeradb's last mocked out functions. Additionally, the testing criteria needs to be updated to reflect the updated naming conventions/command name organization.

[rickmanelius]

@beeradb I'm going to leave this issue be for now unless you think I can be of service/value on end-user testing or documentation. That said, ready to dive back in as soon as you give me the all clear!

[rickmanelius]

Technically the PR is closed, but I believed @beeradb wanted this opened as a reminder to address/file any follow-up issues.

[beeradb]

Closing this as #345 is merged and followups have been created.