Deploying staging and production: the issue of handling DATABASE_URL in secrets

[dvodvo]

The docs state Path to secrets, defaults to .kamal/secrets. Kamal will look for <secrets_path>-common and <secrets_path> (or <secrets_path>.<destination> when using destinations): This leaves a lot of room for varying interpretations. Given one wants to deploy staging and production, one would comply with 2 paragraphs later
require_destination: true
and create
.kamal/secrets.staging
.kamal/secrets.production
But what goes into the base file? the shared secrets across all environments?
What then distinguishes it from -common ?
Does one run one? or the other? or both?

Now extend the logic across multiple servers - say sharding on a tenancy basis where, practically, each server should have its standalone database. One might have

1. deploy.production_first.yml
2. deploy.production_second.yml
3. deploy.staging_first.yml
4. deploy.staging_second.yml
   each file has a distinct host, but refers to a DATABASE_URL that is proper to the environment.
   secrets have no reason to change across servers, so should only require the variants for the environment.

But at this point, having defined require_destination: true, 'staging_first' simply does not match up to 'staging',
there is not guarantee the file name will be interpreted accordingly.
and there is no documentation I can find which allows on to specify the destination within the deploy.x.yml file

[cookesan]

Kamal chooses the destination-specific secrets file from the deploy destination, not from the service name.

The secrets loader reads the common secrets file and then the destination file: secrets-common plus secrets.<destination>. So if you deploy with -d staging, the staging value belongs in the staging destination secrets file; if you deploy production, it belongs in the production destination secrets file.

The practical layout is:

.kamal/secrets-common
.kamal/secrets.staging
.kamal/secrets.production

Put shared values in secrets-common, and put DATABASE_URL in the destination file where that deployment should read it. Do not put both staging and production database URLs under the same DATABASE_URL name in one secrets file and expect Kamal to choose by service.

[dvodvo]

right. we're almost synchronous here. Still, what distinguishes .kamal/secrets-common from .kamal/secrets? what happens to the latter?
TBH, it would be much more straightforward to have a layout, based on my above, more complex example:

.kamal/secrets
.kamal/secrets.staging_one
.kamal/secrets.staging_two
.kamal/secrets.production_one
.kamal/secrets.production_two

[dvodvo]

I may be confusing the documentation with other valuable references found out there. If I understand correctly the above scenario can be realised with .kamal/secrets.staging_first, .kamal/secrets.staging_second , .kamal/secrets.production_first , .kamal/secrets.production_second. Subsequently to deploy any group of servers, one would devise a script to do so. I still wonder what distinguishes '-common' from the regular [ahem] common file...

As a consequence I would re-write the initial paragraphs of https://kamal-deploy.org/docs/configuration/environment-variables/#secrets My thinking is that it is the docs/configuration section and thus should be explicit

Kamal uses dotenv to automatically load environment variables set in the .kamal/secrets file. 

If you are handling different or multiple environments, secrets may change by destination. Destinations are defined by the existence of files with distinct name `.kamal/secrets.<DESTINATION>`.
If you are using destinations, common secrets across all destinations can be set in .kamal/secrets-common, while
destination-only secrets will instead be read from .kamal/secrets.<DESTINATION> if it exists. Secrets will be read from .kamal/secrets.<DESTINATION> first or .kamal/secrets-common if it is not found. [ ADD something here about the plain secrets file ]

[dvodvo]

another, somewhat vague matter is require_destination: true .
I assume it would be placed in deploy.yml with no destination.
But that remains an assumption; expliciteness would be wise.

[cookesan]

The exact load order is secrets-common first, then one override file.

With the default secrets_path, Kamal looks for:

.kamal/secrets-common
.kamal/secrets

when no destination is supplied, and:

.kamal/secrets-common
.kamal/secrets.<destination>

when you run with -d <destination>. Values from the second file override values from secrets-common.

So .kamal/secrets is the no-destination override file. If your app sets require_destination: true in the base deploy.yml and you always deploy with -d staging_one, -d production_one, etc., then deploys will read .kamal/secrets-common plus .kamal/secrets.staging_one or .kamal/secrets.production_one; plain .kamal/secrets is not the shared file in that flow. Shared values belong in .kamal/secrets-common.

Your proposed destination-specific layout is the right shape; I would keep require_destination: true in the base deploy.yml so an accidental no-destination deploy fails early.