Problem using sops with passphrase-protected age key file

[skuethe]

Hey there,

I am currently trying out sops with the freshly released version 1.0.0 of age.
Creating passphrase-protected age key files seems rather easy:
https://github.com/FiloSottile/age#passphrase-protected-key-files

However, when I try to use these with sops I get the following error:

# sops values.enc.yaml
Failed to get the data key required to decrypt the SOPS file.

Group 0: FAILED
  age18v8fr5zpnq84u9pque8ey7sdng3j79qnc7jk2p38r3wsza9dgufszjufnn: FAILED
    - | error at line 1: malformed secret key: separator '1' at
      | invalid position: pos=20, len=21

Recovery failed because no master key was able to decrypt the file. In
order for SOPS to recover the file, at least one key has to be successful,
but none were.

Am I missing something or is this currently not supported by sops?
I have not yet found any sops issues or documentation on this topic.

[skuethe]

So, can anyone help on this?

[pogobanane]

since 086c11d (3.7.2 #1006 ), there exists a env-var which should enable an interactive password prompt via SOPS_AGE_KEY=$(age -d ~/.config/sops/age/keys.age) sops secrets.yml

[jmpolom]

since 086c11d (3.7.2 #1006 ), there exists a env-var which should enable an interactive password prompt via SOPS_AGE_KEY=$(age -d ~/.config/sops/age/keys.age) sops secrets.yml

This is definitely a work around as the documentation for it isn't included in the sops command help output.

Are passphrase encrypted AGE secret keys designated an anti-pattern for sops usage? The documentation doesn't say much about how AGE should be used with sops, and this env var exists so I'm thinking the answer is no here.

If not, I believe the user experience should be improved here by automatically prompting for the key decryption passphrase much like age itself does. I believe consistency in behavior here is what most users familiar with age would expect.

[artm]

An alternative approach: decrypt to a file on tempfs (won't be written to disk, will disappear on restart), schedule the file to be removed some time in the future. The following works on linux with systemd:

# decrypt to an ephemeral file (will be asked to enter the passphrase)
age -d -o /run/shm/keys.txt ~/.config/sops/age/keys

# schedule the file deletion in 1 hour from now
systemd-run --user -u age-clean --on-calendar "@$(date -d "+1hour" +%s)" rm /run/shm/keys.txt

The command's are tedious but can be integrated in some kind of workflow and sops needs to be configured to look for private keys in /run/shm/keys.txt (e.g. by making the default path - ~/.config/sops/age/keys a symlink to the ephemeral path /run/shm/keys.txt).

The benefit of this approach is that it can be used with the editor-sops integration (as long as you keep recreating the ephemeral decrypted key file).

Disclaimer: I have just thought of this right now and haven't really tested it yet.

[DerRockWolf]

since 086c11d (3.7.2 #1006 ), there exists a env-var which should enable an interactive password prompt via SOPS_AGE_KEY=$(age -d ~/.config/sops/age/keys.age) sops secrets.yml

For me this isn't working anymore since sops 3.8.0. Regardless if the SOPS_AGE_KEY env var is exported or put directly in front of the binary.

Failed to get the data key required to decrypt the SOPS file.

Group 0: FAILED
  age1w8egqmyze2hrf2vcyjkxylny9qg8nw8rp9ux9kl9y95rrruthdmqxlzwpp: FAILED
    - | failed to load age identities: failed to parse
      | '/home/USER/.config/sops/age/keys.txt' age identities: error
      | at line 1: malformed secret key: separator '1' at invalid
      | position: pos=20, len=21

Recovery failed because no master key was able to decrypt the file. In order for SOPS to recover the file, at least one key has to be successful, but none were.

---

Update:

This seems to happen because setting SOPS_AGE_KEY doesn't skip loading the other keysources.
Sops always looks into ~/.config/sops/age/keys.txt and if this file exists and is encrypted the decryption obviously fails.
So just moving/renaming the keys.txt file is enough to workaround this 🥳

[sidhanti-ctl]

When using the latest version of SOPS v3.8.1, I too notice the issue where sops fails to read the SOPS_AGE_KEY environment variable. SOPS version 3.7.3 does not exhibit this problem. The issue persists even when you remove the key files (~/.config/sops/age/keys.txt or ~/Library/Application\ Support/sops/age/keys.txt).

➜  SOPS_AGE_KEY=$(age -d ~/.config/sops/age/age-key.txt)

➜  ~/apps/sops/sops-v3.7.3.darwin.arm64 -d secret.enc.json|cat
───────┬─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
       │ STDIN
───────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
   1   │ {
   2   │     "password": "42"
   3   │ }
───────┴─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

➜  ~/apps/sops/sops-v3.8.1.darwin.arm64 -d secret.enc.json
Failed to get the data key required to decrypt the SOPS file.

Group 0: FAILED
  age1zwzgl7q9l6hzhl4f2gqgcxc83gfgj3t932fw4wgrgluvxf6g7s0q6345vf: FAILED
    - | failed to load age identities: failed to open
      | SOPS_AGE_KEY_FILE file: open : no such file or directory

  age1spwwvmnwyhe7urhckwwjejumcy45sch4htmrjur2g4ektetphulqysj7hm: FAILED
    - | failed to load age identities: failed to open
      | SOPS_AGE_KEY_FILE file: open : no such file or directory

Recovery failed because no master key was able to decrypt the file. In
order for SOPS to recover the file, at least one key has to be successful,
but none were.

➜

[DerRockWolf]

When using the latest version of SOPS v3.8.1, I too notice the issue where sops fails to read the SOPS_AGE_KEY environment variable. SOPS version 3.7.3 does not exhibit this problem. The issue persists even when you remove the key files (~/.config/sops/age/keys.txt or ~/Library/Application\ Support/sops/age/keys.txt).

The issue you are experiencing is not related to sops.
If you want sops to use SOPS_AGE_KEY you need to set it in your environment using export or put SOPS_AGE_KEY directly in front of the binary name!

[sidhanti-ctl]

Still getting the same error when exporting or adding the environment variable before the binary:

➜  export SOPS_AGE_KEY=$(age -d ~/.config/sops/age/age-key.txt)

➜  ~/apps/sops/sops-v3.7.3.darwin.arm64 -d secret.enc.json|cat
───────┬─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
       │ STDIN
───────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
   1   │ {
   2   │     "password": "42"
   3   │ }
───────┴─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

➜  ~/apps/sops/sops-v3.8.1.darwin.arm64 -d secret.enc.json
Failed to get the data key required to decrypt the SOPS file.

Group 0: FAILED
  age1zwzgl7q9l6hzhl4f2gqgcxc83gfgj3t932fw4wgrgluvxf6g7s0q6345vf: FAILED
    - | failed to load age identities: failed to open
      | SOPS_AGE_KEY_FILE file: open : no such file or directory

  age1spwwvmnwyhe7urhckwwjejumcy45sch4htmrjur2g4ektetphulqysj7hm: FAILED
    - | failed to load age identities: failed to open
      | SOPS_AGE_KEY_FILE file: open : no such file or directory

Recovery failed because no master key was able to decrypt the file. In
order for SOPS to recover the file, at least one key has to be successful,
but none were.

➜  SOPS_AGE_KEY=$(age -d ~/.config/sops/age/age-key.txt) ~/apps/sops/sops-v3.8.1.darwin.arm64 -d secret.enc.json
Failed to get the data key required to decrypt the SOPS file.

Group 0: FAILED
  age1zwzgl7q9l6hzhl4f2gqgcxc83gfgj3t932fw4wgrgluvxf6g7s0q6345vf: FAILED
    - | failed to load age identities: failed to open
      | SOPS_AGE_KEY_FILE file: open : no such file or directory

  age1spwwvmnwyhe7urhckwwjejumcy45sch4htmrjur2g4ektetphulqysj7hm: FAILED
    - | failed to load age identities: failed to open
      | SOPS_AGE_KEY_FILE file: open : no such file or directory

Recovery failed because no master key was able to decrypt the file. In
order for SOPS to recover the file, at least one key has to be successful,
but none were.

➜

[DerRockWolf]

Did you set the SOPS_AGE_KEY_FILE env var? Try unsetting it.

For me everything is working as expected, which means something is wrong on your end.

[sidhanti-ctl]

The SOPS_AGE_KEY_FILE env var is not set. If I use/set the SOPS_AGE_KEY_FILE env var with the decrypted (i.e. non password-protected) key file then the decryption works fine in both versions.

[DerRockWolf]

Are you 100% sure that the SOPS_AGE_KEY_FILE isn't set because the error failed to open SOPS_AGE_KEY_FILE can only be returned if SOPS_AGE_KEY_FILE is set to a non empty string value.

sops/age/keysource.go

Lines 248 to 255 in b6d3c97

	if ageKeyFile, ok := os.LookupEnv(SopsAgeKeyFileEnv); ok {
	f, err := os.Open(ageKeyFile)
	if err != nil {
	return nil, fmt.Errorf("failed to open %s file: %w", SopsAgeKeyFileEnv, err)
	}
	defer f.Close()
	readers[SopsAgeKeyFileEnv] = f
	}

[sidhanti-ctl]

As sure as I can be :)

➜  unset SOPS_AGE_KEY ; unset SOPS_AGE_KEY_FILE

➜  export SOPS_AGE_KEY_FILE= ; export SOPS_AGE_KEY= ;

➜  echo $SOPS_AGE_KEY_FILE $SOPS_AGE_KEY


➜  export SOPS_AGE_KEY=$(age -d ~/.config/sops/age/age-key.txt)

➜  ~/apps/sops/sops-v3.7.3.darwin.arm64 -d secret.enc.json|cat
───────┬─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
       │ STDIN
───────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
   1   │ {
   2   │     "password": "42"
   3   │ }
───────┴─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

➜  SOPS_AGE_KEY=$(age -d ~/.config/sops/age/age-key.txt) ~/apps/sops/sops-v3.8.1.darwin.arm64 -d secret.enc.json
Failed to get the data key required to decrypt the SOPS file.

Group 0: FAILED
  age1zwzgl7q9l6hzhl4f2gqgcxc83gfgj3t932fw4wgrgluvxf6g7s0q6345vf: FAILED
    - | failed to load age identities: failed to open
      | SOPS_AGE_KEY_FILE file: open : no such file or directory

  age1spwwvmnwyhe7urhckwwjejumcy45sch4htmrjur2g4ektetphulqysj7hm: FAILED
    - | failed to load age identities: failed to open
      | SOPS_AGE_KEY_FILE file: open : no such file or directory

Recovery failed because no master key was able to decrypt the file. In
order for SOPS to recover the file, at least one key has to be successful,
but none were.

➜  export SOPS_AGE_KEY_FILE=~/.config/sops/age/key.txt

➜  ~/apps/sops/sops-v3.7.3.darwin.arm64 -d secret.enc.json|cat
───────┬─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
       │ STDIN
───────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
   1   │ {
   2   │     "password": "42"
   3   │ }
───────┴─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

➜  ~/apps/sops/sops-v3.8.1.darwin.arm64 -d secret.enc.json|cat
───────┬─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
       │ STDIN
───────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
   1   │ {
   2   │     "password": "42"
   3   │ }
───────┴─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

➜

[DerRockWolf]

With export SOPS_AGE_KEY_FILE= you are setting the envvar to an empty value but it needs to be unset.
To verify if a envvar is unset you can do something like this env | grep SOPS_AGE_KEY_FILE.

[sidhanti-ctl]

You were correct, it was my bad between unset vs export. Once I only unset the env var that worked. Thanks for helping with that!