vault snapshot save redirecting to the pod ip

[rshiva777]

Describe the bug
A clear and concise description of what the bug is.

When i try to take snapshot of vault from CLI, it is redirecting to the pod IP and failing

To Reproduce
Steps to reproduce the behavior:

export VAULT_ADDR=http://vault.test.com
vault login  
root@test:~/$ vault status
Key                      Value
---                      -----
Recovery Seal Type       shamir
Initialized              true
Sealed                   false
Total Recovery Shares    5
Threshold                3
Version                  1.10.3
Build Date               n/a
Storage Type             raft
Cluster Name             vault-cluster-45fd4af5
Cluster ID               3d06fd2a-990f-5eda-e468-f502f0a6d674
HA Enabled               true
HA Cluster               https://vault-0.vault-internal:8201
HA Mode                  active
Active Since             2022-12-25T12:00:41.995055563Z
Raft Committed Index     70
Raft Applied Index       70

root@test:~/vaultback$ vault operator raft snapshot save primary.snap
Error taking the snapshot: redirect failed: Get "http://100.96.19.119:8200/v1/sys/storage/raft/snapshot": dial tcp 100.96.19.119:8200: i/o timeout

Note: Snapshot is working within the pod.

Expected behavior
A clear and concise description of what you expected to happen.

Snapshot backup should working from CLI as well outside the vault pod.

Environment:

• Vault Server Version (retrieve with vault status):
  Key Value

---

Recovery Seal Type shamir
Initialized true
Sealed false
Total Recovery Shares 5
Threshold 3
Version 1.10.3
Build Date n/a
Storage Type raft
Cluster Name vault-cluster-45fd4af5
Cluster ID 3d06fd2a-990f-5eda-e468-f502f0a6d674
HA Enabled true
HA Cluster https://vault-0.vault-internal:8201
HA Mode active
Active Since 2022-12-25T12:00:41.995055563Z
Raft Committed Index 70
Raft Applied Index 70

• Vault CLI Version (retrieve with vault version):
  Vault v1.11.0 (ea296cc), built 2022-06-17T15:48:44Z

• Server Operating System/Architecture:
  Ubuntu

[hsimon-hashicorp]

This may be because of your k8s cluster configuration, if your ingress / port forwarder is disallowing the connection. You could try to work around this with something like using kubectl port-forward <name of pod> 8200:8200 and run the snapshot through the port-forward. Please let me know if that helps.

[rshiva777]

@hsimon-hashicorp,

I am able to login to vault outside the pod with ingress fqdn, and do vault operation like create secrets etc.. This issue is faced only during the vault snapshot

[maxb]

I believe this is similar or the same to my issue #15258 - the snapshot operations do not work via request forwarding, in the way almost everything in Vault does.

[bhargav2427]

Hi @rshiva777, Can you provide the service details?

[rshiva777]

@bhargav2427, what detailed do you need ? I am using opensource vault installed on kubernetes environment

[bhargav2427]

@rshiva777 Snapshots can be taken on active pod only. Your service might not routing traffic to only active pod. I faced the same issue. To take snapshots use vault active service only.

[bhargav2427]

Hey @rshiva777, Does using vault-active service solve the issue?

[rshiva777]

Hello @bhargav2427 ,

I am able to take snapshot within vault pods, but unable to take outside the pod