Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Platform integration: Download all databases, Allow a primary relationship for the main 'db', fixes #4415, fixes platformsh/ddev-platformsh#59 #4426

Merged
merged 9 commits into from Dec 12, 2022
Merged

Platform integration: Download all databases, Allow a primary relationship for the main 'db', fixes #4415, fixes platformsh/ddev-platformsh#59 #4426

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
9d7ac4a
Allow for primary relationship in platform.sh pull/push
rfay Dec 2, 2022
f3f4ca0
on ddev pull, allow importing a set of databases, not just one
rfay Dec 7, 2022
d6580c9
Sort out difference between relationship and db name
rfay Dec 8, 2022
f0fae87
Make files import work
rfay Dec 8, 2022
f122105
Test*Pull/Push working
rfay Dec 8, 2022
6feb45b
Handle PLATFORM_PRIMARY_RELATIONSHIP again
rfay Dec 8, 2022
48b1abd
Update docs
rfay Dec 8, 2022
1e1eefa
Pacify linter
rfay Dec 8, 2022
18d3600
Fix type in failure [skip ci]
rfay Dec 9, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
14 changes: 8 additions & 6 deletions cmd/ddev/cmd/pull.go
View file
Open in desktop
Expand Up @@ -62,13 +62,15 @@ func appPull(providerType string, app *ddevapp.DdevApp, skipConfirmation bool, s
}

// Add or override the command-line provided environment variables
envVars := strings.Split(env, ",")
for _, v := range envVars {
split := strings.Split(v, "=")
if len(split) != 2 {
util.Failed("unable to parse environment variable setting: %v", v)
if env != "" {
envVars := strings.Split(env, ",")
for _, v := range envVars {
split := strings.Split(v, "=")
if len(split) != 2 {
util.Failed("unable to parse command-line environment variable setting: '%v'", v)
}
provider.EnvironmentVariables[split[0]] = split[1]
}
provider.EnvironmentVariables[split[0]] = split[1]
}

if err := app.Pull(provider, skipDbArg, skipFilesArg, skipImportArg); err != nil {
Expand Down
16 changes: 9 additions & 7 deletions cmd/ddev/cmd/push.go
View file
Open in desktop
Expand Up @@ -60,14 +60,16 @@ func apppush(providerType string, app *ddevapp.DdevApp, skipConfirmation bool, s
util.Failed("Failed to get provider: %v", err)
}

// Add or override the command-line provided environment variables
envVars := strings.Split(env, ",")
for _, v := range envVars {
split := strings.Split(v, "=")
if len(split) != 2 {
util.Failed("unable to parse environment variable setting: %v", v)
if env != "" {
// Add or override the command-line provided environment variables
envVars := strings.Split(env, ",")
for _, v := range envVars {
split := strings.Split(v, "=")
if len(split) != 2 {
util.Failed("unable to parse environment variable setting: %v", v)
}
provider.EnvironmentVariables[split[0]] = split[1]
}
provider.EnvironmentVariables[split[0]] = split[1]
}

if err := app.Push(provider, skipDbArg, skipFilesArg); err != nil {
Expand Down
14 changes: 14 additions & 0 deletions docs/content/users/providers/platform.md
View file
Open in desktop
Expand Up @@ -42,6 +42,20 @@ web_environment:
4. Run `ddev pull platform`. After you agree to the prompt, the current upstream database and files will be downloaded.
5. Optionally use `ddev push platform` to push local files and database to Platform.sh. The [`ddev push`](../basics/commands.md#push) command can potentially damage your production site, so we don’t recommend using it.

If you have more than one database on your Platform.sh project, you'll need to choose which one you want to use
Copy link
Sponsor Collaborator

@mattstein mattstein Dec 13, 2022 •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@rfay I understand only one database will be handled by the integration, but what would you recommend if I had additional databases I also wanted to pull into DDEV?

Create my own modified version of platform.yaml and modify db_pull_command?

I ask because I’m looking at the docs, and designating a primary naturally begs the question of what I should do with my other ones. Even if it’s just a hint, it seems like it’d be worth addressing.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In this version, all databases will be handled by the ddev pull platform, so line 45 is just incorrect. Thanks for catching that.

Copy link
Sponsor Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Am I understanding correctly that all databases will be pulled, but that I still need to designate a PLATFORM_PRIMARY_RELATIONSHIP so DDEV knows what to use for its (default) 'db' database?

Copy link
Sponsor Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ultimately asking while I attempt to improve the wording over here.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mattstein this is correct.

  • All databases will be downloaded
  • The remote database that is flagged as primary will be imported into the db database of DDEV (the default one)
  • All other remote databases will be imported into ddev databases with the same as in the remote

I believe I've never used the word database that much in such a short message

Copy link
Sponsor Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@bserem Thanks for clarifying the database behavior for this database-related documentation adjustment regarding databases. 😄

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not all projects will need for a "db" database to be populated. For example, as the ddev-platformsh add-on matures, it should be able to handle databases of any number that have the same names they do on Platform.sh, and the "db" database not be used at all.

Copy link
Sponsor Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So this is more of an if you have multiple databases and want to designate a primary to use with 'db', yes? If so I’ll update #4469 accordingly.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That is correct. Most people who have ever used DDEV will want that. There may be contexts in the future where that's not needed. The same may be true for Acquia or Pantheon or other hosts, where the settings are wired to particular database names and people are not using DDEV's automatically generated settings, like settings.ddev.php or Craft CMS's .env configuration.

Copy link
Sponsor Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Revised in this commit, which is hopefully more accurate! 😅

as the 'db' primary database on DDEV, and that will be the one pulled or pushed.
Do this by setting PLATFORM_PRIMARY_RELATIONSHIP, for example,

```
ddev config --web-environment-add=PLATFORM_PRIMARY_RELATIONSHIP=main`
```

or run `ddev pull platform` with the `--environment` flag, for example,

```
ddev pull platform --environment="PLATFORM_PRIMARY_RELATIONSHIP=main"
```

## Usage

* `ddev pull platform` will connect to Platform.sh to download database and files. To skip downloading and importing either file or database assets, use the `--skip-files` and `--skip-db` flags.
Expand Down
44 changes: 40 additions & 4 deletions pkg/ddevapp/dotddev_assets/providers/platform.yaml
View file
Open in desktop
Expand Up @@ -23,6 +23,12 @@
# 5. Run `ddev pull platform`. After you agree to the prompt, the current upstream database and files will be downloaded.
# 6. Optionally use `ddev push platform` to push local files and database to platform.sh. Note that `ddev push` is a command that can potentially damage your production site, so this is not recommended.

# If you have more than one database on your Platform.sh proect,
# you will likely to choose which one you want to use
# as the primary database ('db').
# Do this by setting PLATFORM_PRIMARY_RELATIONSHIP, for example, `ddev config --web-environment-add=PLATFORM_PRIMARY_RELATIONSHIP=main`
# or run `ddev pull platform` with the environment variable, for example
# `ddev pull platform -y --environment=PLATFORM_PRIMARY_RELATIONSHIP=main`
# If you need to change this `platform.yaml` recipe, you can change it to suit your needs, but remember to remove the "ddev-generated" line from the top.

# Debugging: Use `ddev exec platform` to see what platform.sh knows about
Expand All @@ -31,19 +37,44 @@
auth_command:
command: |
set -eu -o pipefail
if [ -z "${PLATFORMSH_CLI_TOKEN:-}" ]; then echo "Please make sure you have set PLATFORMSH_CLI_TOKEN in ~/.ddev/global_config.yaml" && exit 1; fi
if [ -z "${PLATFORMSH_CLI_TOKEN:-}" ]; then echo "Please make sure you have set PLATFORMSH_CLI_TOKEN." && exit 1; fi
if [ -z "${PLATFORM_PROJECT:-}" ]; then echo "Please make sure you have set PLATFORM_PROJECT." && exit 1; fi
if [ -z "${PLATFORM_ENVIRONMENT:-}" ]; then echo "Please make sure you have set PLATFORM_ENVIRONMENT." && exit 1; fi

db_pull_command:
command: |
#set -x # You can enable bash debugging output by uncommenting
# set -x # You can enable bash debugging output by uncommenting
set -eu -o pipefail
export PLATFORMSH_CLI_NO_INTERACTION=1
ls /var/www/html/.ddev >/dev/null # This just refreshes stale NFS if possible
platform db:dump --yes --gzip --file=/var/www/html/.ddev/.downloads/db.sql.gz --project="${PLATFORM_PROJECT}" --environment="${PLATFORM_ENVIRONMENT}"
# /tmp/db_relationships.yaml is the full yaml output of the database relationships
db_relationships_file=/tmp/db_relationships.yaml
PLATFORM_RELATIONSHIPS="" platform relationships -y -e "${PLATFORM_ENVIRONMENT}" | yq 'with_entries(select(.[][].type == "mariadb:*" or .[][].type == "*mysql:*" or .[][].type == "postgresql:*")) ' >${db_relationships_file}
db_relationships=($(yq ' keys | .[] ' ${db_relationships_file}))
db_names=($(yq '.[][].path' ${db_relationships_file}))
db_count=${#db_relationships[@]}
# echo "db_relationships=${db_relationships} sizeof db_relationships=${#db_relationships[@]} db_names=${db_names} db_count=${db_count} PLATFORM_PRIMARY_RELATIONSHIP=${PLATFORM_PRIMARY_RELATIONSHIP}"
# If we have only one database, import it into local database named 'db'
if [ ${#db_names[@]} -eq 1 ]; then db_names[0]="db"; fi

for (( i=0; i<${#db_relationships[@]}; i++ )); do
db_name=${db_names[$i]}
rel=${db_relationships[$i]}
# if PLATFORM_PRIMARY_RELATIONSHIP is set, then when doing that one, import it into local database 'db'
if [ "${rel}" = "${PLATFORM_PRIMARY_RELATIONSHIP:-notset}" ] ; then
echo "PLATFORM_PRIMARY_RELATIONSHIP=${PLATFORM_PRIMARY_RELATIONSHIP:-} so using it as database 'db' instead of the upstream '${db_name}'"
db_name="db"
fi

platform db:dump --yes --relationship=${rel} --gzip --file=/var/www/html/.ddev/.downloads/${db_name}.sql.gz --project="${PLATFORM_PROJECT:-setme}" --environment="${PLATFORM_ENVIRONMENT:-setme}"
done
echo "Downloaded db dumps for databases '${db_names[@]}'"

files_import_command:
command: |
#set -x # You can enable bash debugging output by uncommenting
set -eu -o pipefail
export PLATFORMSH_CLI_NO_INTERACTION=1
# Use $PLATFORM_MOUNTS if it exists to get list of mounts to download, otherwise just web/sites/default/files (drupal)
declare -a mounts=(${PLATFORM_MOUNTS:-/web/sites/default/files})
platform mount:download --all --yes --quiet --project="${PLATFORM_PROJECT}" --environment="${PLATFORM_ENVIRONMENT}" --target=/var/www/html
Expand All @@ -54,16 +85,21 @@ db_push_command:
command: |
# set -x # You can enable bash debugging output by uncommenting
set -eu -o pipefail
export PLATFORMSH_CLI_NO_INTERACTION=1
ls /var/www/html/.ddev >/dev/null # This just refreshes stale NFS if possible
pushd /var/www/html/.ddev/.downloads >/dev/null;
gzip -dc db.sql.gz | platform db:sql --project="${PLATFORM_PROJECT}" --environment="${PLATFORM_ENVIRONMENT}"
if [ "${PLATFORM_PRIMARY_RELATIONSHIP:-}" != "" ] ; then
rel="--relationship ${PLATFORM_PRIMARY_RELATIONSHIP}"
fi
gzip -dc db.sql.gz | platform db:sql --project="${PLATFORM_PROJECT}" ${rel:-} --environment="${PLATFORM_ENVIRONMENT}"

# push is a dangerous command. If not absolutely needed it's better to delete these lines.
# TODO: This is a naive, Drupal-centric push, which needs adjustment for the mount to be pushed.
files_push_command:
command: |
# set -x # You can enable bash debugging output by uncommenting
set -eu -o pipefail
export PLATFORMSH_CLI_NO_INTERACTION=1
ls "${DDEV_FILES_DIR}" >/dev/null # This just refreshes stale NFS if possible
platform mount:upload --yes --quiet --project="${PLATFORM_PROJECT}" --environment="${PLATFORM_ENVIRONMENT}" --source="${DDEV_FILES_DIR}" --mount=web/sites/default/files

105 changes: 65 additions & 40 deletions pkg/ddevapp/provider.go
View file
Open in desktop
Expand Up @@ -3,7 +3,9 @@ package ddevapp
import (
"github.com/drud/ddev/pkg/output"
"os"
"path"
"path/filepath"
"strings"

"fmt"

Expand Down Expand Up @@ -88,7 +90,7 @@ func (app *DdevApp) Pull(provider *Provider, skipDbArg bool, skipFilesArg bool,
if skipDbArg {
output.UserOut.Println("Skipping database pull.")
} else {
output.UserOut.Println("Obtaining database...")
output.UserOut.Println("Obtaining databases...")
fileLocation, importPath, err := provider.GetBackup("database")
if err != nil {
return err
Expand All @@ -105,9 +107,8 @@ func (app *DdevApp) Pull(provider *Provider, skipDbArg bool, skipFilesArg bool,
if err != nil {
return err
}
output.UserOut.Println("Importing database...")
output.UserOut.Printf("Importing databases %v\n", fileLocation)
err = provider.importDatabaseBackup(fileLocation, importPath)

if err != nil {
return err
}
Expand All @@ -118,7 +119,7 @@ func (app *DdevApp) Pull(provider *Provider, skipDbArg bool, skipFilesArg bool,
output.UserOut.Println("Skipping files pull.")
} else {
output.UserOut.Println("Obtaining files...")
fileLocation, importPath, err := provider.GetBackup("files")
files, _, err := provider.GetBackup("files")
if err != nil {
return err
}
Expand All @@ -132,7 +133,11 @@ func (app *DdevApp) Pull(provider *Provider, skipDbArg bool, skipFilesArg bool,
output.UserOut.Println("Skipping files import.")
} else {
output.UserOut.Println("Importing files...")
err = provider.importFilesBackup(fileLocation, importPath)
f := ""
if files != nil && len(files) > 0 {
f = files[0]
}
err = provider.doFilesImport(f, "")
if err != nil {
return err
}
Expand Down Expand Up @@ -202,34 +207,37 @@ func (app *DdevApp) Push(provider *Provider, skipDbArg bool, skipFilesArg bool)
return nil
}

// GetBackup will create and download a backup
// GetBackup will create and download a set of backups
// Valid values for backupType are "database" or "files".
// returns fileURL, importPath, error
func (p *Provider) GetBackup(backupType string) (string, string, error) {
// returns []fileURL, []importPath, error
func (p *Provider) GetBackup(backupType string) ([]string, []string, error) {
var err error
var filePath string
var fileNames []string
if backupType != "database" && backupType != "files" {
return "", "", fmt.Errorf("could not get backup: %s is not a valid backup type", backupType)
return nil, nil, fmt.Errorf("could not get backup: %s is not a valid backup type", backupType)
}

// Set the import path blank to use the root of the archive by default.
importPath := ""

p.prepDownloadDir()

switch backupType {
case "database":
filePath, err = p.getDatabaseBackup()
fileNames, err = p.getDatabaseBackups()
case "files":
filePath, err = p.getFilesBackup()
fileNames, err = p.doFilesPullCommand()
default:
return "", "", fmt.Errorf("could not get backup: %s is not a valid backup type", backupType)
return nil, nil, fmt.Errorf("could not get backup: %s is not a valid backup type", backupType)
}
if err != nil {
return "", "", err
return nil, nil, err
}

return filePath, importPath, nil
importPaths := make([]string, len(fileNames))
// We don't use importPaths for the providers
for i := range fileNames {
importPaths[i] = ""
}

return fileNames, importPaths, nil
}

// UploadDB is used by Push to push the database to hosting provider
Expand All @@ -238,7 +246,7 @@ func (p *Provider) UploadDB() error {
_ = os.Mkdir(p.getDownloadDir(), 0755)

if p.DBPushCommand.Command == "" {
util.Warning("No DBPushCommand is defined for provider %s", p.ProviderType)
util.Warning("No DBPushCommand is defined for provider '%s'", p.ProviderType)
return nil
}

Expand Down Expand Up @@ -268,7 +276,7 @@ func (p *Provider) UploadFiles() error {
_ = os.Mkdir(p.getDownloadDir(), 0755)

if p.FilesPushCommand.Command == "" {
util.Warning("No FilesPushCommand is defined for provider %s", p.ProviderType)
util.Warning("No FilesPushCommand is defined for provider '%s'", p.ProviderType)
return nil
}

Expand Down Expand Up @@ -297,15 +305,14 @@ func (p *Provider) getDownloadDir() string {
return destDir
}

func (p *Provider) getFilesBackup() (filename string, error error) {

func (p *Provider) doFilesPullCommand() (filename []string, error error) {
destDir := filepath.Join(p.getDownloadDir(), "files")
_ = os.RemoveAll(destDir)
_ = os.MkdirAll(destDir, 0755)

if p.FilesPullCommand.Command == "" {
util.Warning("No FilesPullCommand is defined for provider %s", p.ProviderType)
return "", nil
util.Warning("No FilesPullCommand is defined for provider '%s'", p.ProviderType)
return nil, nil
}
s := p.FilesPullCommand.Service
if s == "" {
Expand All @@ -314,21 +321,21 @@ func (p *Provider) getFilesBackup() (filename string, error error) {

err := p.app.ExecOnHostOrService(s, p.injectedEnvironment()+"; "+p.FilesPullCommand.Command)
if err != nil {
return "", fmt.Errorf("Failed to exec %s on %s: %v", p.FilesPullCommand.Command, s, err)
return nil, fmt.Errorf("Failed to exec %s on %s: %v", p.FilesPullCommand.Command, s, err)
}

return filepath.Join(p.getDownloadDir(), "files"), nil
return []string{filepath.Join(p.getDownloadDir(), "files")}, nil
}

// getDatabaseBackup retrieves database using `generic backup database`, then
// getDatabaseBackups retrieves database using `generic backup database`, then
// describe until it appears, then download it.
func (p *Provider) getDatabaseBackup() (filename string, error error) {
func (p *Provider) getDatabaseBackups() (filename []string, error error) {
_ = os.RemoveAll(p.getDownloadDir())
_ = os.Mkdir(p.getDownloadDir(), 0755)

if p.DBPullCommand.Command == "" {
util.Warning("No DBPullCommand is defined for provider")
return "", nil
util.Warning("No DBPullCommand is defined for provider '%s'", p.ProviderType)
return nil, nil
}

s := p.DBPullCommand.Service
Expand All @@ -337,22 +344,39 @@ func (p *Provider) getDatabaseBackup() (filename string, error error) {
}
err := p.app.MutagenSyncFlush()
if err != nil {
return "", err
return nil, err
}
err = p.app.ExecOnHostOrService(s, p.injectedEnvironment()+"; "+p.DBPullCommand.Command)
if err != nil {
return "", fmt.Errorf("Failed to exec %s on %s: %v", p.DBPullCommand.Command, s, err)
return nil, fmt.Errorf("Failed to exec %s on %s: %v", p.DBPullCommand.Command, s, err)
}
return filepath.Join(p.getDownloadDir(), "db.sql.gz"), nil
err = p.app.MutagenSyncFlush()
if err != nil {
return nil, err
}

sqlTarballs, err := fileutil.ListFilesInDirFullPath(p.getDownloadDir())
if err != nil || sqlTarballs == nil {
return nil, fmt.Errorf("failed to find downloaded files in %s: %v", p.getDownloadDir(), err)
}
return sqlTarballs, nil
}

// importDatabaseBackup will import a downloaded database
// importDatabaseBackup will import a slice of downloaded databases
// If a custom importer is provided, that will be used, otherwise
// the default is app.ImportDB()
func (p *Provider) importDatabaseBackup(fileLocation string, importPath string) error {
func (p *Provider) importDatabaseBackup(fileLocation []string, importPath []string) error {
var err error
if p.DBImportCommand.Command == "" {
err = p.app.ImportDB(fileLocation, importPath, true, false, "db")
for i, loc := range fileLocation {
// The database name used will be basename of the file.
// For example. `db.sql.gz` will go into the database named 'db'
// xxx.sql will go into database named 'xxx';
b := path.Base(loc)
n := strings.Split(b, ".")
dbName := n[0]
err = p.app.ImportDB(loc, importPath[i], true, false, dbName)
}
} else {
s := p.DBImportCommand.Service
if s == "" {
Expand All @@ -364,10 +388,11 @@ func (p *Provider) importDatabaseBackup(fileLocation string, importPath string)
return err
}

// importFilesBackup will import a downloaded files tarball or directory
// If a custom importer is provided, that will be used, otherwise
// doFilesImport will import previously downloaded files tarball or directory
// If a custom importer (FileImportCommand) is provided, that will be used, otherwise
// the default is app.ImportFiles()
func (p *Provider) importFilesBackup(fileLocation string, importPath string) error {
// FilesImportCommand may also optionally take on the job of downloading the files.
func (p *Provider) doFilesImport(fileLocation string, importPath string) error {
var err error
if p.FilesImportCommand.Command == "" {
err = p.app.ImportFiles(fileLocation, importPath)
Expand All @@ -376,7 +401,7 @@ func (p *Provider) importFilesBackup(fileLocation string, importPath string) err
if s == "" {
s = "web"
}
output.UserOut.Printf("Importing files via custom files_import_command")
output.UserOut.Printf("Importing files via custom files_import_command...")
err = p.app.ExecOnHostOrService(s, p.injectedEnvironment()+"; "+p.FilesImportCommand.Command)
}
return err
Expand Down
2 changes: 1 addition & 1 deletion pkg/ddevapp/providerPlatform_test.go
View file
Open in desktop
Expand Up @@ -90,7 +90,7 @@ func TestPlatformPull(t *testing.T) {
err = app.Start()
require.NoError(t, err)
err = app.Pull(provider, false, false, false)
assert.NoError(err)
require.NoError(t, err)

assert.FileExists(filepath.Join(app.GetHostUploadDirFullPath(), "victoria-sponge-umami.jpg"))
out, err := exec.RunHostCommand("bash", "-c", fmt.Sprintf(`echo 'select COUNT(*) from users_field_data where mail="margaret.hopper@example.com";' | %s mysql -N`, DdevBin))
Expand Down