Move raft cluster to another subnet.

[welderpb]

I have a raft cluster(v 1.6.1) in one subnet, and trying move it to another subnet. I'v shutdown all nodes, and copy "data" folder of one instance to another instances in different location.
Trying to start new instance with old data. Vault is started and unsealed, but i can see old "HA Cluster" address in status.
Old ip address of one node: 172.16.16.133
i'v change config to new address for example: 192.168.1.1
node_id also changed, due to it is dynamically configured while bootstrap.

ui = true
disable_mlock = false
api_addr      = "http://192.168.1.1:8100"
cluster_addr  = "http://192.168.1.1:8101"

listener "tcp" {
  address = "0.0.0.0:8100"
  cluster_address = "0.0.0.0:8101"
  tls_disable = true
}

storage "raft" {
  path = "/vault/data"
  node_id = "i-05eacb84e130423ff"
  retry_join {
    auto_join = "provider=aws region=us-east-1 tag_key=vault-servers tag_value=vault-cluster"
    auto_join_scheme = "http"
    auto_join_port = 8100
  }
}

vault status:

Key                      Value
---                      -----
Recovery Seal Type       shamir
Initialized              true
Sealed                   false
Total Recovery Shares    1
Threshold                1
Version                  1.6.1
Storage Type             raft
Cluster Name             vault-cluster-6f79dae1
Cluster ID               658da5e7-0396-710f-49c9-5e888a2e6dac
HA Enabled               true
HA Cluster               https://172.16.16.133:8101
HA Mode                  standby
Active Node Address      http://172.16.16.133:8100
Raft Committed Index     3307
Raft Applied Index       3307

We can see old address for "HA Cluster".

logs:

2021-01-14T18:26:50.613Z [INFO]  core: vault is unsealed
2021-01-14T18:26:50.613Z [INFO]  core: unsealed with stored key
2021-01-14T18:26:50.613Z [INFO]  core: entering standby mode
2021-01-14T18:26:50.613Z [INFO]  storage.raft: entering follower state: follower="Node at 192.168.1.1:8101 [Follower]" leader=
2021-01-14T18:26:59.198Z [WARN]  storage.raft: not part of stable configuration, aborting election
2021-01-14T18:27:55.962Z [ERROR] core: error during forwarded RPC request: error="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 172.16.16.133:8101: i/o timeout""
2021-01-14T18:27:55.962Z [ERROR] core: forward request error: error="error during forwarding RPC request"

So question, how can i force to change "HA Cluster" field? i guess it is stored inside vault db..

[victorgetz]

Hi we had the same issue and the solution is to go with dns.

You need to adjust your old vault setup to use dns before you can migrate.

We use the helm chart and had to adjust it like this:

server:
extraEnvironmentVars:
  VAULT_API_ADDR: "http://$(HOSTNAME).vault-internal:8200"
ha:
  enabled: true
  replicas: 3
  raft:
    setNodeId: true
    enabled: true
    config: |
      ui = true
      disable_mlock = true
      cluster_addr = "https://HOSTNAME.vault-internal:8201"
      service_registration "kubernetes" {}
      listener "tcp" {
        tls_disable = 1
        proxy_protocol_behavior = "use_always"
        address = "[::]:8200"
        cluster_address = "[::]:8201"
      }
      storage "raft" {
        path = "/vault/data"
        retry_join {
          leader_api_addr = "http://vault-0.vault-internal:8200"
        }
        retry_join {
          leader_api_addr = "http://vault-1.vault-internal:8200"
        }
        retry_join {
          leader_api_addr = "http://vault-2.vault-internal:8200"
        }
      }

Setting VAULT_API_ADDR over extraEnvironmentVars is a hack otherwise we would not be able to set VAULT_API_ADDR depending on the hostname.

I know it is not the exact solution for your problems but maybe it helps.

[ncabatoff]

Hi @welderpb,

Sorry for the late response, but I think the way to handle this address change would've been via the manual recovery process.

As this is a question and not a bug report / feature request, I'm going to close this issue. Feel free to reopen or open a new issue if you feel there's more to do here.