[3.0.1-rc1] Proxmox 8.1.4 Cloud init not working #935

TheLillo · 2024-02-11T14:19:33Z

When provisioning a virtual machine (VM) using the Telmate Terraform plugin for Proxmox, with a base on a pre-configured cloud-init template, the resulting VM loses all its configuration settings, such as boot disk, RAM, and core allocations.

Environment Details
Proxmox version: 8.1.4
Telmate Terraform plugin version: 3.0.1-rc1
Terraform version: v1.7.1 on linux_amd64
Cloud-init template settings taken from Proxmox Documentation

terraform {
  required_providers {
    proxmox = {
      source = "Telmate/proxmox"
      version = "3.0.1-rc1"
    }
  }
}

provider "proxmox" {
  # Configuration options
   pm_api_url = "https://127.0.0.1:8006/api2/json"
}

resource "proxmox_vm_qemu" "test" {
   name        = "test"
   target_node = "pve"
   clone       = "focal64"
}

The resource example is take from plugin documentation

Another strange behaviour is that as soon as start all works

After some times all configuration disappear

The text was updated successfully, but these errors were encountered:

electropolis · 2024-02-12T12:44:55Z

I have almost the same issue. But in my provisioning process the Cloudinit dissapear Some people say that its working but officially that doesn't work and is mentioned here
#922
I believe here is a PR to fix this : TechDufus/home.io#4

vasekhodina · 2024-02-12T21:16:19Z

@TheLillo Hi, I've been trying to find a workaround for this for some time and I finally found it: #901 (comment)

Hope it helps.

electropolis · 2024-02-12T22:30:59Z

@TheLillo Hi, I've been trying to find a workaround for this for some time and I finally found it: #901 (comment)

Hope it helps.

I doubt that this is not the case. :/

USBAkimbo · 2024-02-13T13:21:22Z

I've had the same problem - here's what I did to fix it

On your Proxmox server, run the following to create your cloud-init template VM
In my case I'm using the latest Ubuntu 22.04 cloud-init image
Note as well that I'm using virt-customize to install the qemu guest agent to the VM as it's required for terraform destroy and to confirm that the agent is installed

wget https://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-amd64.img
apt install guestfs-tools
virt-customize -a jammy-server-cloudimg-amd64.img --install qemu-guest-agent
qm create 9000 --name ci-template --memory 2048 --net0 virtio,bridge=vmbr2 --scsihw virtio-scsi-pci
qm set 9000 --scsi0 local-lvm:0,import-from=/root/jammy-server-cloudimg-amd64.img
qm set 9000 --ide2 local-lvm:cloudinit
qm set 9000 --boot order=scsi0
qm set 9000 --serial0 socket --vga serial0
qm template 9000

The docs for this next part are wrong

cloudinit_cdrom_storage - string - Set the storage location for the cloud-init drive. Required when specifying cicustom.

This isn't true and needs to be defined in your Terraform config even if you're not using cicustom
In my case I just needed to define the variable cloudinit_cdrom_storage as local-lvm

After doing all of this, my Terraform applies now work

AirP0WeR · 2024-02-13T15:21:30Z

Same issue for me. I use local-zfs.

electropolis · 2024-02-13T17:17:37Z

I've had the same problem - here's what I did to fix it

On your Proxmox server, run the following to create your cloud-init template VM

In my case I'm using the latest Ubuntu 22.04 cloud-init image

Note as well that I'm using virt-customize to install the qemu guest agent to the VM as it's required for terraform destroy and to confirm that the agent is installed
wget https://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-amd64.img
apt install guestfs-tools
virt-customize -a jammy-server-cloudimg-amd64.img --install qemu-guest-agent
qm create 9000 --name ci-template --memory 2048 --net0 virtio,bridge=vmbr2 --scsihw virtio-scsi-pci
qm set 9000 --scsi0 local-lvm:0,import-from=/root/jammy-server-cloudimg-amd64.img
qm set 9000 --ide2 local-lvm:cloudinit
qm set 9000 --boot order=scsi0
qm set 9000 --serial0 socket --vga serial0
qm template 9000
The docs for this next part are wrong
cloudinit_cdrom_storage - string - Set the storage location for the cloud-init drive. Required when specifying cicustom.
This isn't true and needs to be defined in your Terraform config even if you're not using cicustom

In my case I just needed to define the variable cloudinit_cdrom_storage as local-lvm

After doing all of this, my Terraform applies now work

Awesome, but you what is the universal solution? You for example are using cloudinit during template creation

qm set 9000 --ide2 local-lvm:cloudinit

That's ok but worked in Proxmox v7. Now In proxmox 8 doesn't. Some say that cloudinit shouldn't be set at all in template. Other say that it should be on ide3. None of that worked. I'm using cicustom and when there is phase when terraform resize the disk to bigger one and sets cloudinit suddenly cloudinit isn't attached at all and on ide2 empty cdrom appears. And that's the problem. Anyone who fix this isn't using cicustom using snippets.

USBAkimbo · 2024-02-13T17:25:16Z

Interesting, I think there's a PR coming to fix this for good, but in the meantime this works for people like me just doing a simple cloud init config Annoying that this doesn't work for people using cicustom though

…

On Tue, 13 Feb 2024, 17:17 Konrad, ***@***.***> wrote: I've had the same problem - here's what I did to fix it - On your Proxmox server, run the following to create your cloud-init template VM - In my case I'm using the latest Ubuntu 22.04 cloud-init image - Note as well that I'm using virt-customize to install the qemu guest agent to the VM as it's required for terraform destroy and to confirm that the agent is installed wget https://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-amd64.img apt install guestfs-tools virt-customize -a jammy-server-cloudimg-amd64.img --install qemu-guest-agent qm create 9000 --name ci-template --memory 2048 --net0 virtio,bridge=vmbr2 --scsihw virtio-scsi-pci qm set 9000 --scsi0 local-lvm:0,import-from=/root/jammy-server-cloudimg-amd64.img qm set 9000 --ide2 local-lvm:cloudinit qm set 9000 --boot order=scsi0 qm set 9000 --serial0 socket --vga serial0 qm template 9000 - The docs for this next part are wrong cloudinit_cdrom_storage - string - Set the storage location for the cloud-init drive. Required when specifying cicustom. - This isn't true and needs to be defined in your Terraform config even if you're not using cicustom - In my case I just needed to define the variable cloudinit_cdrom_storage as local-lvm After doing all of this, my Terraform applies now work Awesome, but you what is the universal solution? You for example are using cloudinit during template creation qm set 9000 --ide2 local-lvm:cloudinit That's ok but worked in Proxmox v7. Now In proxmox 8 doesn't. Some say that cloudinit shouldn't be set at all in template. Other say that it should be on ide3. None of that worked. I'm using cicustom and when there is phase when terraform resize the disk to bigger one and sets cloudinit suddenly cloudinit isn't attached at all and on ide2 empty cdrom appears. And that's the problem. Anyone who fix this isn't using cicustom using snippets. — Reply to this email directly, view it on GitHub <#935 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ARBSAZ6DR2VO3AAN4YRZUA3YTON33AVCNFSM6AAAAABDDSNMVOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNBSGA2DCNZZGQ> . You are receiving this because you commented.Message ID: ***@***.***>

electropolis · 2024-02-13T17:51:35Z

@USBAkimbo you are referring to this #937 ? I hope this will help as it removes that damn ISO which is so useless.

Torque475 · 2024-02-16T20:06:21Z

I have almost the same issue. But in my provisioning process the Cloudinit dissapear Some people say that its working but officially that doesn't work and is mentioned here #922 I believe here is a PR to fix this : TechDufus/home.io#4

As the owner of that PR, my workaround is setting the agent = 0 during VM creation. That allowed my cloud init settings (set in the vm module) to properly apply. After that I change the agent = 1 and move on

And my PR is my personal setup, not for the provider. I'm still a novice that's just using the providers other people are making 😅

MixedOne · 2024-02-18T21:25:11Z

hello, I've tried electropolis solution without any success, it still removes disks during after VM creation, during cloudinit boot (haven't tried other options)
agent toggling doesnt change anything in my case.

update:
I'm on proxmox 8.1.4, so from my pov it's a choice between providers:
2.9.11 - which crashes and prints stack traces or
3.0.1-rc1 - which removes disks and causes loop during VM creation

hestiahacker · 2024-02-19T18:07:56Z

I've posted a minimal example over here that works to deploy a VM with cloud init enabled. I'm testing on Proxmox 7.4-17 and my template VM has cloud init enabled.

I've also submitted a merge request to update the documentation to indicate that cloudinit_cdrom_storage is always required when using cloud init. #939

Hopefully the example either works for others and can help people determine what's different between their terraform and my example, or it doesn't work and we can figure out if it's another issue (e.g. an incompatibility with the Proxmox 8.x release).

None of this addresses the problem of cicustom not working, but we'll get a separate ticket to focus specifically on that problem.

allen353299 · 2024-02-20T12:45:45Z

I've posted a minimal example over here that works to deploy a VM with cloud init enabled. I'm testing on Proxmox 7.4-17 and my template VM has cloud init enabled.

I've also submitted a merge request to update the documentation to indicate that cloudinit_cdrom_storage is always required when using cloud init. #939

Hopefully the example either works for others and can help people determine what's different between their terraform and my example, or it doesn't work and we can figure out if it's another issue (e.g. an incompatibility with the Proxmox 8.x release).

None of this addresses the problem of cicustom not working, but we'll get a separate ticket to focus specifically on that problem.

I also encounter the issue of cicustom not working as expected, but I want to execute scripts post VM deployment.

In my case, if I set cicustom correctly, then cicustom, ciuser, cipassword, etc., will not work either, leading to an inability to log into my VM to check any messages, and the default ubuntu also don't work.

If not use cicustom to execute script, I use Provisioner, but I don't know how that SSH login to the VMs inside PVE using passwords (in connection segment), I only can't SSH login to main console of my node.

Any helpful suggestions would be greatly appreciated.

hestiahacker · 2024-02-20T18:49:17Z

I need to investigate the "cicustom doesn't work" issue because it should work and as best I can tell, it doesn't.

Until that is resolved, I can only offer you a workaround, which is using the pattern that I use when deploying VMs:

Use Terraform and Cloud Init to create the VM and set the IP address, netmask, gateway, DNS server, and SSH keys
Use ansible to SSH in and run any scripts that need run (installing nginx, setting up Nextcloud, etc.)

I chain these two things together with a resource "null_resource" "provisioning" block in my terraform, and then a block in there that looks something like this:

  provisioner "local-exec" {
    command = "cd ../../ansible/test_environment; ansible-playbook --limit ${var.fqdn} ../${var.fqdn}.yml"
  }

I know this is not a solution, only a workaround, but it's the best I have for you at the moment.

electropolis · 2024-02-21T11:23:43Z

I need to investigate the "cicustom doesn't work" issue because it should work and as best I can tell, it doesn't.

Until that is resolved, I can only offer you a workaround, which is using the pattern that I use when deploying VMs:

Use Terraform and Cloud Init to create the VM and set the IP address, netmask, gateway, DNS server, and SSH keys

Use ansible to SSH in and run any scripts that need run (installing nginx, setting up Nextcloud, etc.)

I chain these two things together with a resource "null_resource" "provisioning" block in my terraform, and then a block in there that looks something like this:
  provisioner "local-exec" {
    command = "cd ../../ansible/test_environment; ansible-playbook --limit ${var.fqdn} ../${var.fqdn}.yml"
  }
I know this is not a solution, only a workaround, but it's the best I have for you at the moment.

We can debate what solution is working but that's not the case. It's a workaround that doesn't even applied the current scenario with cicustom . By definition workaround should be something that address current problem and relies with the same methods that doesn't work by providing a capable workaround using the same schema. Setting those minimalistic settings using those ciuser cipassword and so on is not the case here because that limits the functionality of cloudinit. With cicustom that relies on yaml snippets with cloudinit give SO MUCH more. Makes an ultimate images that doesn't need to be set additionally with ansible and all other roles that will prepare the image after post provisioning. Plus using ansible doesn't gives you immutable infrastructure. So basically you have an Image that can be change freely when it comes to cloudinit inside image you cannot. And that's the whole point of it. This is also the schema that included in many cloud providers when setting up flavours. All images in those flavours are pre-defined with some cloud settings and cannot be changed by any customer until he uses his own images but then he will lose much more of the cloud functionality when it comes monitoring that vm for example. This is what NOT having a more extensive cloudinit gives you.

ivan-danilov · 2024-03-05T00:22:32Z

As suggested above, I solved creation issues with cloudinit_cdrom_storage, and it works. But another problem is that an attempt to execute terraform apply again wants to re-create everything again. The thing that "forces replacement" is cicustom that it wants to set no null.
Other things it detects as need changing include ciuser, numa, qemu_os, smbios, network.mtu.

Finally, it wants to drop vga { type = "serial0", memory = 0 } and create another vga { type = "serial0" }.

Not sure if I need to create another issue to report all this, seems to be caused by the same thing.

Tinyblargon · 2024-03-07T20:10:26Z

This should be resolved in the latest build #959 has an example.

seckin-oter · 2024-03-20T15:10:44Z

I've posted a minimal example over here that works to deploy a VM with cloud init enabled. I'm testing on Proxmox 7.4-17 and my template VM has cloud init enabled.
I've also submitted a merge request to update the documentation to indicate that cloudinit_cdrom_storage is always required when using cloud init. #939
Hopefully the example either works for others and can help people determine what's different between their terraform and my example, or it doesn't work and we can figure out if it's another issue (e.g. an incompatibility with the Proxmox 8.x release).
None of this addresses the problem of cicustom not working, but we'll get a separate ticket to focus specifically on that problem.

I also encounter the issue of cicustom not working as expected, but I want to execute scripts post VM deployment.

In my case, if I set cicustom correctly, then cicustom, ciuser, cipassword, etc., will not work either, leading to an inability to log into my VM to check any messages, and the default ubuntu also don't work.

If not use cicustom to execute script, I use Provisioner, but I don't know how that SSH login to the VMs inside PVE using passwords (in connection segment), I only can't SSH login to main console of my node.

Any helpful suggestions would be greatly appreciated.

If you can't ssh into the cloud-init VM and want to ssh into it using user/password ; you can exec into VM using xterm.js on proxmox UI. Head into the directory "/etc/ssh/sshd_config.d" , there might be a .conf file. Change the value to "PasswordAuthentication yes" and then restart sshd service. Try ssh into the VM again using username and password.

leogout · 2024-04-23T21:21:53Z

I've had the same problem - here's what I did to fix it

* On your Proxmox server, run the following to create your cloud-init template VM

* In my case I'm using the latest Ubuntu 22.04 cloud-init image

* Note as well that I'm using virt-customize to install the qemu guest agent to the VM as it's required for `terraform destroy` and to confirm that the agent is installed

wget https://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-amd64.img
apt install guestfs-tools
virt-customize -a jammy-server-cloudimg-amd64.img --install qemu-guest-agent
qm create 9000 --name ci-template --memory 2048 --net0 virtio,bridge=vmbr2 --scsihw virtio-scsi-pci
qm set 9000 --scsi0 local-lvm:0,import-from=/root/jammy-server-cloudimg-amd64.img
qm set 9000 --ide2 local-lvm:cloudinit
qm set 9000 --boot order=scsi0
qm set 9000 --serial0 socket --vga serial0
qm template 9000

* The docs for this next part are wrong

cloudinit_cdrom_storage - string - Set the storage location for the cloud-init drive. Required when specifying cicustom.

* This isn't true and needs to be defined in your Terraform config even if you're not using `cicustom`

* In my case I just needed to define the variable `cloudinit_cdrom_storage ` as `local-lvm`

After doing all of this, my Terraform applies now work

Thank you very much for this, I've been looking everywhere for this answer... Everything clicked in place with this setting. For future reference and to facilitate google searches, here are the keywords I've been typing the past day :

No CloudInit Drive found
CloudInit configuration ignored
Terraform doesn't attributes an IP to my VM

And here is my Terraform config :

terraform {
  required_providers {
    proxmox = {
      source = "telmate/proxmox"
      version = "3.0.1-rc1"
    }
  }
}

provider "proxmox" {
  pm_api_url = "https://192.168.1.20:8006/api2/json"

  pm_api_token_id = "XXX@pve!XXX"
  pm_api_token_secret = "XXXXXX"
  pm_tls_insecure = true
}

resource "proxmox_vm_qemu" "test_server" {
  count = 1 
  name = "test-vm-${count.index + 1}"
  target_node = var.proxmox_host
  clone = var.template_name
  agent = 1
  os_type = "cloud-init"
  cloudinit_cdrom_storage = "local-lvm"
  cores = 2
  sockets = 1
  cpu = "host"
  memory = 1024
  scsihw = "virtio-scsi-pci"
  bootdisk = "scsi0"

  disks {
    scsi {
      scsi0 {
        disk {
          size = 10
          storage = "local-lvm"
        }
      }
    }
  }

  network {
    model = "virtio"
    bridge = "vmbr0"
  }

  ipconfig0 = "ip=192.168.1.9/24,gw=192.168.1.1"

  ssh_user = "root"
  sshkeys = <<EOF
  ${var.ssh_key}
  EOF
}

And for completness' sake, here is how I create my template

wget https://cloud.debian.org/images/cloud/bookworm/latest/debian-12-genericcloud-amd64.qcow2
virt-customize -a debian-12-genericcloud-amd64.qcow2 --install qemu-guest-agent --run-command 'systemctl start qemu-guest-agent.service && systemctl enable qemu-guest-agent.service'
virt-customize -a debian-12-genericcloud-amd64.qcow2 --truncate /etc/machine-id
qm create 8100 --memory 2048 --name debian-cloud --net0 virtio,bridge=vmbr0 --scsihw virtio-scsi-pci
qm importdisk 8100 debian-12-genericcloud-amd64.qcow2 local-lvm
qm set 8100 --scsihw virtio-scsi-pci --scsi0 local-lvm:vm-8100-disk-0   
qm set 8100 --ide2 local-lvm:cloudinit
qm set 8100 --boot order=scsi0
qm set 8100 --serial0 socket --vga serial0
qm set 8100 --agent enabled=1
qm template 8100

elg0ch0 · 2024-05-25T04:31:16Z

Does anyone have been able to deploy a VM successfully using Packer (create template with cloud-init enabled) + v3.0.1-rc[12] + Proxmox v8.1.4?
In case you did it, would you mind to share the config files and the steps? I've been working on this for a few days now but so far no luck... I can boot the VM but the cloud-init is not applied (not even the ip address indicated by

ipconfig0 = "ip=192.168.20.27/23,gw=192.168.20.1"

I haven't tried @leogout's solution yet (I need to create my image with Packer)

xlanor · 2024-05-27T23:07:59Z

To whoever is on the latest terraform provider, cloudinit_cdrom_storage has been deprecated

https://github.com/Telmate/terraform-provider-proxmox/pull/1008/files

use disks/ide/ide3/cloudinit instead

hestiahacker pushed a commit to hestiahacker/terraform-provider-proxmox that referenced this issue Feb 18, 2024

docs: corrected documentation to address issue Telmate#935

d508c21

This was referenced Feb 18, 2024

docs: corrected documentation to address issue #935 #939

Merged

Cloud-init drive delete when cloning vm with cloud-init drive #901

Open

MixedOne mentioned this issue Feb 28, 2024

cloud-init no longer working. #944

Closed

Tinyblargon mentioned this issue Apr 23, 2024

[3.0.1-rc2] Status #993

Closed

Tinyblargon mentioned this issue May 19, 2024

[3.0.1-rc3] Status #1009

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[3.0.1-rc1] Proxmox 8.1.4 Cloud init not working #935

[3.0.1-rc1] Proxmox 8.1.4 Cloud init not working #935

TheLillo commented Feb 11, 2024 •

edited

electropolis commented Feb 12, 2024 •

edited

vasekhodina commented Feb 12, 2024

electropolis commented Feb 12, 2024

USBAkimbo commented Feb 13, 2024 •

edited

AirP0WeR commented Feb 13, 2024

electropolis commented Feb 13, 2024 •

edited

USBAkimbo commented Feb 13, 2024 via email

electropolis commented Feb 13, 2024

Torque475 commented Feb 16, 2024

MixedOne commented Feb 18, 2024 •

edited

hestiahacker commented Feb 19, 2024

allen353299 commented Feb 20, 2024

hestiahacker commented Feb 20, 2024

electropolis commented Feb 21, 2024

ivan-danilov commented Mar 5, 2024

Tinyblargon commented Mar 7, 2024

seckin-oter commented Mar 20, 2024

leogout commented Apr 23, 2024

elg0ch0 commented May 25, 2024

xlanor commented May 27, 2024

[3.0.1-rc1] Proxmox 8.1.4 Cloud init not working #935

[3.0.1-rc1] Proxmox 8.1.4 Cloud init not working #935

Comments

TheLillo commented Feb 11, 2024 • edited

electropolis commented Feb 12, 2024 • edited

vasekhodina commented Feb 12, 2024

electropolis commented Feb 12, 2024

USBAkimbo commented Feb 13, 2024 • edited

AirP0WeR commented Feb 13, 2024

electropolis commented Feb 13, 2024 • edited

USBAkimbo commented Feb 13, 2024 via email

electropolis commented Feb 13, 2024

Torque475 commented Feb 16, 2024

MixedOne commented Feb 18, 2024 • edited

hestiahacker commented Feb 19, 2024

allen353299 commented Feb 20, 2024

hestiahacker commented Feb 20, 2024

electropolis commented Feb 21, 2024

ivan-danilov commented Mar 5, 2024

Tinyblargon commented Mar 7, 2024

seckin-oter commented Mar 20, 2024

leogout commented Apr 23, 2024

elg0ch0 commented May 25, 2024

xlanor commented May 27, 2024

TheLillo commented Feb 11, 2024 •

edited

electropolis commented Feb 12, 2024 •

edited

USBAkimbo commented Feb 13, 2024 •

edited

electropolis commented Feb 13, 2024 •

edited

MixedOne commented Feb 18, 2024 •

edited