multiple proxmox clusters (nodes without cluster)

[schmitch]

Is your feature request related to a problem? Please describe.
currently sometimes it's better to have individual nodes not within a proxmox cluster and instead use them on their own.
however with the current proxmox provider that works, but it's inconvienient. first of all node migration does not work (besides that this is possible, as seen in proxmox data center) and second it's a little bit akward to have tons of provider inside the state

i.e. like:

provider "proxmox" {
  alias    = "nodes"
  endpoint = "https://${var.proxmox_nodes[each.key].hostname}:8006"
  username = var.proxmox_nodes[each.key].username
  password = var.proxmox_nodes[each.key].password
  insecure = true
  ssh {
    agent = true
  }

  for_each = local.proxmox_providers
}

while it works, it will break some functions completly when migrating vms between clusters for example.

Describe the solution you'd like
currently dell redshift has a cool pattern for that by specifcing a map of servers instead: https://search.opentofu.org/provider/dell/redfish

Describe alternatives you've considered

Additional context
currently we host kubernetes on top of proxmox but we do not really need the cluster features of proxmox and we also have a higher latency between some clusters (wan) however we still sometimes need to move vms between nodes.

[bpg]

Hey @schmitch 👋🏼

If I understand you correctly, you'd like to define all connection details per node in a single place, like a map, and then let the resource's node name act as a key into that map to pick up the connection details for a particular node?

Otherwise, you'd need to explicitly specify where to connect for each resource. In that case, the current "awkward" multi-provider configuration is actually a better option, and it can be nicely handled using modules. I.e.:

provider "proxmox" {
  alias     = "dev-cluster"
  endpoint  = var.pve_endpoint
  api_token = var.pve_api_token
  insecure  = true
}


provider "proxmox" {
  alias     = "prod-cluster"
  endpoint  = var.pve_endpoint
  api_token = var.pve_api_token
  insecure  = false
  ssh {
    agent    = true
    username = var.pve_ssh_username

    dynamic "node" {
      for_each = var.pve_cluster2_nodes
      content {
        name    = node.key
        address = node.value
      }
    }
  }
}

module "k8s-prod" {
  providers = {
    proxmox = proxmox.prod-cluster
  }

  source      = "./modules/k8s-infra"
}

module "k8s-dev" {
  providers = {
    proxmox = proxmox.dev-cluster
  }

  source      = "./modules/k8s-infra"
}

Also, please keep in mind that the provider supports lots of other cluster-wide resources that don't have a reference to a node, but still need the connection info.

If all the above makes sense, I think this ticket boils down to the matter of VM migration support, which isn’t entirely clear to me from your description. AFAIK, PVE supports VM migration only within a cluster, as it uses the cluster network for that. Even if you can "hack" it a bit and use a separate network that spans multiple PVE clusters, I'm not sure how you can specify a target node for migration outside of the source cluster 🤔
But regardless, if you can run that migration in the PVE UI, you should be able to do the same with the provider if you set the migration flag.

Am I getting you right?

[schmitch]

AFAIK, PVE supports VM migration only within a cluster, as it uses the cluster network for that.

No migration also works outside of cluster nodes. There is the proxmox datacenter alpha 2 which supports that, there you need to map the storage and the network. But if you do not enter anything it will automatically set something.
It uses ‘qm remote-migrate’ https://pve.proxmox.com/pve-docs/qm.1.html.

But regardless, if you can run that migration in the PVE UI, you should be able to do the same with the provider if you set the migration flag.

No that does not work because of the multiple providers it would never work. (Basically I use modules already) the problem is, is that terraform can’t move resources between different providers (and with the for_each syntax you will have multiple providers)

Well I think it would be good to have both ways of configuring the provider, single cluster or multiple, but if you use multiple cluster you need to provide the cluster name to all resources, basically that is how redfish works. (Node name would be not enough if you have 3clusters each with 3 nodes)
I also tested it and it always created new nodes.

[bpg]

Ah, I see... remote migration is an experimental feature:

https://pve.proxmox.com/pve-docs/api-viewer/#/nodes/{node}/qemu/{vmid}/remote_migrate

It requires API token from the remote cluster, and it seems to be available only for VM / LXC.

[schmitch]

and it seems to be available only for VM / LXC

I've started out with proxmox, does proxmox support more?

It requires API token from the remote cluster

wouldn't it be possible to generate an api token no matter what? or at least only support the option when the api token is used?

as far as I understand? the normal migration method would also have create/destroy in opentofu/terraform?

I would really like this feature but I'm not sure if I would understand the code good enough to make a PR and I'm also not sure if it would even work since the provider limitations. As far as I've understood is that at the moment the migration/clone only works when the provider is the same: https://github.com/bpg/terraform-provider-proxmox/blob/main/proxmoxtf/resource/vm/vm.go#L1842