Audit storage not being mounted for HA mode

[jasonodonnell]

Bug where only standalone mode is allowed to mount audit storage. Audit storage can be used in HA mode and this restriction should be lifted.

[laurentiuspurba]

Hi @jasonodonnell , I would like to submit PR regarding this issue. I did read CONTRIBUTING.md and I also did sign the HashiCorp SLA. Please let me know how to proceed. Thanks!

[laurentiuspurba]

Hi @jasonodonnell, if audit device was issued with vault audit enable file file_path=/var/log/vault/audit.log, should I set mountPath: /var/log/vault and have chown vault:vault /var/log/vault in _helpers.tpl?

[jasonodonnell]

Hi @laurentiuspurba, please see https://github.com/hashicorp/vault-helm/pull/79/files for the patch so far. The mount path is /vault/audit for audit storage when it's enabled.

I'm working on some additional changes here and will push them to that PR when ready.

[jasonodonnell]

@laurentiuspurba, the workflow for this requires no chown now. Assuming auditStorage has been enabled via helm, after Vault is initialized and unsealed you would simply:

kubectl exec -ti vault-0 -- vault audit enable file file_path=/vault/audit/audit.log

[laurentiuspurba]

@jasonodonnell, I will try that command.

[laurentiuspurba]

Hi @jasonodonnell, This is my use case that I am working right now.

I have main vault cluster with consul backend, and vault audit enabled to file_path=/var/log/vault. Then I did vault operator migrate which migrated all data from consul backend to gcs bucket. The data was migrated successfully.

Then I spun up a new vault cluster with gcs bucket as the backend. This new cluster had auto-unseal and ha enabled.

The pod was up and running, but the log showed the following error

[ERROR] core: failed to create audit entry: path=file/ error="sanity check failed; unable to open "/var/log/vault/audit.log" for writing: mkdir /var/log/vault: permission denied"

The vault status is the following:

▶ kubectl exec -it laurentvault-0 -- vault status
Key                      Value
---                      -----
Recovery Seal Type       shamir
Initialized              true
Sealed                   false
Total Recovery Shares    11
Threshold                2
Version                  1.0.3
Cluster Name             vault-cluster-98b9997a
Cluster ID               11100668-31e2-d781-3ba3-a696ff93d7c8
HA Enabled               true
HA Cluster               n/a
HA Mode                  standby
Active Node Address      <none>

While trying to unseal it, the process failed and I saw this in the log

[INFO]  rollback: starting rollback manager
[ERROR] core: failed to create audit entry: path=file/ error="sanity check failed; unable to open "/var/log/vault/audit.log" for writing: mkdir /var/log/vault: permission denied"
[INFO]  core: pre-seal teardown starting
[INFO]  core: cluster listeners not running
[WARN]  expiration: context cancled while restoring leases, stopping lease loading
[INFO]  rollback: stopping rollback manager
[INFO]  core: pre-seal teardown complete
[ERROR] core: post-unseal setup failed: error="failed to setup audit table"
[WARN]  core: vault is sealed
[WARN]  failed to unseal core: error="unseal with stored key failed: failed to setup audit table"
[INFO]  core: stored unseal keys supported, attempting fetch
[INFO]  core: vault is unsealed
[INFO]  core: post-unseal setup starting
[INFO]  core: loaded wrapping token key
[ERROR] core: failed to create audit entry: path=file/ error="sanity check failed; unable to open "/var/log/vault/audit.log" for writing: mkdir /var/log/vault: permission denied"

My question is, can I use thisvault-helm chart to spin up vault cluster with backend storage is the result of vault operator migrate with vault audit enabled to file_path=/var/log/vault?

If that is not possible, most probably this is what I need to do:

• I need to disable vault audit in my main vault,
• Issue vault operator migrate to migrate data to gcs bucket backend
• Spin up a new vault cluster with this newly created backend, wiht auditStorage set to true
• Then enable audit device by issuing command vault audit enable file file_path=/vault/audit/audit.log

I'll appreciate your comments on this.

Thank you,
Laurentius

[jasonodonnell]

@laurentiuspurba Unfortunately I think the latter is required, you'll need to disable the audit backend. It's permissions are being reverted because that directory isn't backed by a persistent volume (so it's in the tempfs).

[laurentiuspurba]

Hi @jasonodonnell , thanks for your comment on this. I will try solution on this.

[laurentiuspurba]

Hi @jasonodonnell By disabling vault audit in main vault, after the migration, I was able to spin up a new vault cluster without any issue.