PLATFORM_RELATIONSHIPS environment variable and mapping .platform.app.yaml and services.yaml

[lolautruche]

Services are configured in .platform/services.yaml and referenced in .platform.app.yaml.
Let's reflect them in DDEV, in config.platformsh.yaml.

A lot of services are available in Platform.sh, but we want to reflect the ones that are already available out-of-the-box in DDEV.

Configuring them includes the exposure of the $PLATFORM_RELATIONSHIPS environment variable, which contains a base64 encoded JSON blob.
We need to ensure that we expose the correct information and credentials about the services we configure.

Example of decoded blob:

{
  "cache": [
    {
      "service": "memcached",
      "ip": "169.254.30.132",
      "hostname": "aulc4mtgtpdll5phtb3eioygey.memcached.service._.fr-3.platformsh.site",
      "cluster": "ohppb3vrezyog-master-7rqtwti",
      "host": "cache.internal",
      "rel": "memcached",
      "scheme": "memcached",
      "type": "memcached:1.6",
      "port": 11211
    }
  ],
  "database": [
    {
      "username": "user",
      "scheme": "mysql",
      "service": "mysqldb",
      "fragment": null,
      "ip": "169.254.118.116",
      "hostname": "gzanmenz3vkkimeol5i54acinm.mysqldb.service._.fr-3.platformsh.site",
      "public": false,
      "cluster": "ohppb3vrezyog-master-7rqtwti",
      "host": "database.internal",
      "rel": "mysql",
      "query": {
        "is_master": true
      },
      "path": "main",
      "password": "",
      "type": "mysql:10.2",
      "port": 3306,
      "host_mapped": false
    }
  ]
}

hostname, port, path (DB name), username, password, etc... should get the corresponding for the DDEV database, e.g. (values presented below are examples only, they may differ from real values from DDEV):

{
  "cache": [
    {
      "service": "memcached",
      "ip": "17.20.0.5",
      "hostname": "memcached",
      "cluster": "ddev-memcached-container",
      "host": "memcached",
      "rel": "memcached",
      "scheme": "memcached",
      "type": "memcached:1.6",
      "port": 11211
    }
  ],
  "database": [
    {
      "username": "db",
      "scheme": "mysql",
      "service": "mysqldb",
      "fragment": null,
      "ip": "172.20.0.4",
      "hostname": "db",
      "public": false,
      "cluster": "ddev-db-container",
      "host": "db",
      "rel": "mysql",
      "query": {
        "is_master": true
      },
      "path": "db",
      "password": "db",
      "type": "mariadb:10.4",
      "port": 3306,
      "host_mapped": false
    }
  ]
}

The JSON value should then be encoded in base64:

ewogICJkYXRhYmFzZSI6IFsKICAgIHsKICAgICAgInVzZXJuYW1lIjogImRiIiwKICAgICAgInNjaGVtZSI6ICJteXNxbCIsCiAgICAgICJzZXJ2aWNlIjogImRiIiwKICAgICAgImZyYWdtZW50IjogbnVsbCwKICAgICAgImlwIjogIjE3Mi4yMC4wLjQiLAogICAgICAiaG9zdG5hbWUiOiAiZGRldi1teWFwcC1kYiIsCiAgICAgICJwdWJsaWMiOiBmYWxzZSwKICAgICAgImNsdXN0ZXIiOiAiIiwKICAgICAgImhvc3QiOiAiZGRldi1teWFwcC1kYiIsCiAgICAgICJyZWwiOiAibXlzcWwiLAogICAgICAicXVlcnkiOiB7CiAgICAgICAgImlzX21hc3RlciI6IHRydWUKICAgICAgfSwKICAgICAgInBhdGgiOiAiZGIiLAogICAgICAicGFzc3dvcmQiOiAiZGIiLAogICAgICAidHlwZSI6ICJtYXJpYWRiOjEwLjQiLAogICAgICAicG9ydCI6IDMzMDYsCiAgICAgICJob3N0X21hcHBlZCI6IGZhbHNlCiAgICB9CiAgXQp9

The information exposed is up to the service (each service has a dedicated page, e.g. MariaDB).

Services to configure and to expose in $PLATFORM_RELATIONSHIPS

• MariaDB / MySQL
• Postgres
• Redis
• Memcached
• RabbitMQ

Properties in the Relationship JSON object

Each service has its own set of properties, but most are common.
All documented properties must be present, even if they are not used in DDEV.

Meaningful Properties

Var name	Value
username	Username for the service (set as null if not needed)
password	Password for the service, when applicable (null otherwise)
port	Port of the service
scheme	The scheme that will be used when applicable (e.g., mysql, redis, pgsql, amqp, http, ...)
path	When a database, it's the database name. May be an URI path for HTTP services like Solr. Set to null when not needed
service	Service identifier with its version (e.g., postgresql12)
ip	IP of the service
hostmame	Hostname for the service
type	Reflected type defined in services.yaml

Properties that must be present, but can be left with default values

Var name	Value
fragment	Set to null
public	Set to false
cluster	Can be anything, e.g. the container name
rel	Same as scheme
query	When a database, set with {"is_master": true} Otherwise, set to {}
host_mapped	Set to false

[lolautruche]

Services to be considered:

• Varnish
• Elasticsearch
• OpenSearch
• Solr
• Memcached
• RabbitMQ

[lolautruche]

What are the databases that we currently support in the add-on? Postgres only or is MySQL/MariaDB also supported?

[rfay]

DDEV supports MariaDB 5.5-10.8, MySQL 5.5-8.0, and Postgres 9-14. That's a superset of what Platform supports, so every database type and version Platform supports should work. I've manually tested a variety of MariaDB/MySQL/Postgres but haven't tested all permutations.

[lolautruche]

Thanks @rfay .

I'm also describing here what $PLATFORM_RELATIONSHIPS env var contains. We need to expose it for the services we support, as it may be used in various situations, such as:

• Application services configuration
  • blog.blackfire.io example
  • Laravel template example, which uses the Laravel bridge, which leverages the config reader
• Environment variables mapping using a .environment file, which gets sourced when FPM is started, before cron commands are run, and when logging into the container over SSH (example with Akeneo)
• Build/Deploy hooks

[lolautruche]

Added more details about the relationship object for common services.

[rfay]

I was hoping for examples of the variable PLATFORM_RELATIONSHIPS. BTW, your blog.blackfire.io example is in a private repo, the link doesn't work.

If you could update with the PLATFORM_RELATIONSHIPS env variable that you expect or have, that would be great.

[lolautruche]

@rfay I updated the description 🙂 . You have an example with the database.
Regarding blog.blackfire.io, I know but I can't add you (not enough seats)

[lolautruche]

Exposing $PLATFORM_RELATIONSHIPS would ensure that any app pulled from Platform.sh would work without any configuration change. Users leverage this env var in different ways, as I explained in my previous comment. As an example, they may configure their database by extracting the credentials our of $PLATFORM_RELATIONSHIPS and exposing the env vars that their app is expecting. This is what is done in the Akeneo template, since Akeneo expects the $APP_DATABASE_* variables to be populated:

# .environment
export APP_DATABASE_HOST=$(echo $PLATFORM_RELATIONSHIPS | base64 --decode | jq -r ".database[0].host")
export APP_DATABASE_PORT=$(echo $PLATFORM_RELATIONSHIPS | base64 --decode | jq -r ".database[0].port")
export APP_DATABASE_NAME=$(echo $PLATFORM_RELATIONSHIPS | base64 --decode | jq -r ".database[0].path")
export APP_DATABASE_USER=$(echo $PLATFORM_RELATIONSHIPS | base64 --decode | jq -r ".database[0].username")
export APP_DATABASE_PASSWORD=$(echo $PLATFORM_RELATIONSHIPS | base64 --decode | jq -r ".database[0].password")

The Laravel bridge, which is recommended to use along with a Laravel app (and used in the Laravel template), has the same purpose : populating the expected environment variables so that the Laravel app is properly configured:

// $config is a Platformsh\ConfigReader\Config instance, from the ConfigReader library    
$credentials = $config->credentials($relationshipName);

setEnvVar('DB_CONNECTION', $credentials['scheme']);
setEnvVar('DB_HOST', $credentials['host']);
setEnvVar('DB_PORT', $credentials['port']);
setEnvVar('DB_DATABASE', $credentials['path']);
setEnvVar('DB_USERNAME', $credentials['username']);
setEnvVar('DB_PASSWORD', $credentials['password']);

The difference from the previous example is that the Laravel bridge uses the Config Reader (a PHP library provided by Platform.sh) to read the $PLATFORM_RELATIONSHIPS variable.

Those are only 2 examples, but would apply to any project pulled from PSH, that users would expect to work (almost) right away using DDEV. And for it to work, it needs $PLATFORM_RELATIONSHIPS to be exposed 😉 .

[lolautruche]

Exposing $PLATFORM_RELATIONSHIPS would ensure that any app pulled from Platform.sh would work without any configuration change

Their apps wouldn't connect to PSH services, but local ones provided by DDEV. All the details to connect to these local services (host, credentials, etc) would be provided by the $PLATFORM_RELATIONSHIPS variable

[rfay]

Thanks for explaining this.

There are a some implications here.

• The normal usage of DDEV with a known project type generates settings for that project type. For example, for Drupal9, ddev will generate a settings.ddev.php and make settings.php include it. We would have to turn that off with disable_settings_management: true
• Currently DDEV tries to do all this configuration work without starting a project. That means it doesn't have a predictable environment to work in, doesn't have jq or base64 to use directly. Sprig, which is already built into the golang template features here, has base64 encoding capability with b64enc, http://masterminds.github.io/sprig/encoding.html. However, the data structure we have to create and then encode is probably too complex for the install.yaml.

Can we construct a master PLATFORM_RELATIONSHIPS that is essentially static? Giving directions on how to contact each service regardless of whether they've decided to use that service?

[lolautruche]

Thanks for these precisions!

• Settings management: I think we probably should disable settings management as you suggest, but let's discuss it today.
• I'm not sure this can be handled without starting the project, since it needs the services to be started I guess. Regarding your suggestion of constructing a $PLATFORM_RELATIONSHIPS that is essentially static, I don't know. If it's possible yes, but is it? Anyway, we could expose everything that's available for sure.

[rfay]

Creating json with just jq, https://spin.atomicobject.com/2021/06/08/jq-creating-updating-json/

[rfay]

@flovntp points out in slack:

i don't know if it helps but to get (dynamic) credentials into the .environment file, this is how i did :
https://github.com/flovntp/platformsh-symfony-template/blob/symfony-6.1-php8.1-webapp/.environment#L5

export DATABASE_HOST=$(echo $PLATFORM_RELATIONSHIPS | base64 --decode | jq -r ".database[0].host")
export DATABASE_PORT=$(echo $PLATFORM_RELATIONSHIPS | base64 --decode | jq -r ".database[0].port")
export DATABASE_NAME=$(echo $PLATFORM_RELATIONSHIPS | base64 --decode | jq -r ".database[0].path")
export DATABASE_USER=$(echo $PLATFORM_RELATIONSHIPS | base64 --decode | jq -r ".database[0].username")
export DATABASE_PASSWORD=$(echo $PLATFORM_RELATIONSHIPS | base64 --decode | jq -r ".database[0].password")
export DATABASE_URL="postgresql://${DATABASE_USER}:${DATABASE_PASSWORD}@${DATABASE_HOST}:${DATABASE_PORT}/${DATABASE_NAME}?serverVersion=14&charset=utf8"

[rfay]

With

• Implement PLATFORM_RELATIONSHIPS, fixes #9 #48
  blog.blackfire.io works out of the box, nothing to do.