CloudInit configuration is not getting applied #586
Labels
Comments
|
PS:I don't really know if this is not implemented, of if its just a issue from my side. |
|
Hi @some-random-username-123! 👋🏼 (nice GH username btw :)
Could you pls. post here the error that that you saw? Also I'd like to clarify whther the original VM that you're cloning has a cloudinit config? |
bpg
added
the
⌛ pending author's response
|
Hi @bpg,
The exact issue is, that when I clone my vm with the TF provider, after the vm starts the cloudinit config (user and hostname, etc) is not applied in the template. But when I clone it manually and press the button, all the cloudinit configuration is applied. If you want to have a look at my code feel free to look at my repo: I hope this gives you some more insight. Regards |
bpg
removed
the
⌛ pending author's response
|
@bpg I took a brief look at the example @some-random-username-123 provided and apparently the packer template is built by specifying bios to ovmf therefore I think this is related to #575 |
|
Hello, I'm also having this problem I found this small thread in the Proxmox forums about the Regeneration process. -> https://forum.proxmox.com/threads/regenerate-cloud-init-image-using-ansible.89964/ It is basically a remove/re-add the cloudinit drive. I hope it could help solve this problem. Thank you |
|
Hey @some-random-username-123 👋🏼 I'm trying to reproduce this issue with a minimal TF configuration, but everything seems to be working for me (tm) Here is my template: resource "proxmox_virtual_environment_vm" "ubuntu_vm" {
name = "586-template"
node_name = var.virtual_environment_node_name
vm_id = 586
machine = "q35"
bios = "ovmf"
agent {
enabled = true
}
efi_disk {
datastore_id = "local-lvm"
file_format = "raw"
type = "4m"
}
disk {
datastore_id = "local-lvm"
file_id = proxmox_virtual_environment_file.ubuntu_cloud_image.id
interface = "virtio0"
iothread = true
discard = "on"
size = 20
}
network_device {
bridge = "vmbr0"
}
operating_system {
type = "l26"
}
started = false
}
resource "proxmox_virtual_environment_file" "ubuntu_cloud_image" {
content_type = "iso"
datastore_id = "local"
node_name = var.virtual_environment_node_name
source_file {
path = "https://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-amd64.img"
}
}
data "local_file" "ssh_public_key" {
filename = "./id_rsa.pub"
}
resource "proxmox_virtual_environment_file" "cloud_config" {
content_type = "snippets"
datastore_id = "local"
node_name = var.virtual_environment_node_name
source_raw {
data = <<EOF
#cloud-config
runcmd:
- apt update
- apt install -y qemu-guest-agent net-tools
- timedatectl set-timezone America/Toronto
- systemctl enable qemu-guest-agent
- systemctl start qemu-guest-agent
- echo "done" > /tmp/vendor-cloud-init-done
EOF
file_name = "cloud-config.yaml"
}
}
resource "proxmox_virtual_environment_vm" "ubuntu_vm_cloned" {
depends_on = [proxmox_virtual_environment_vm.ubuntu_vm]
name = "587-cloned"
node_name = var.virtual_environment_node_name
vm_id = 587
clone {
vm_id = 586
}
initialization {
# interface = "scsi0"
ip_config {
ipv4 {
address = "dhcp"
}
}
user_account {
username = "ubuntu"
keys = [trimspace(data.local_file.ssh_public_key.content)]
}
vendor_data_file_id = proxmox_virtual_environment_file.cloud_config.id
}
started = true
}Both VMs got deployed, the "cloned" one has cloudinit: Without pressing "regenerate" there, I was able to login into the VM using configured public key: So far, no ideas what could be wrong 🤔 |
|
@some-random-username-123 this could be the reason. If you're not overriding |
|
@bpg Thanks for the response. |
|
Hi @webtroter, @tobiasehlert! 👋🏼 I see you upvoted this issue, do you have a similar problem in your deployments? If yes, would you mind sharing your configuration? Thanks! |
bpg
added
the
⌛ pending author's response
|
On my part, after your other reply I seemed to made it work (see my
librenms IAC repo)
But I need to retry the provider to be certain or at least report what I
did wrong.
…On Wed, Oct 11, 2023, 22:28 Pavel Boldyrev ***@***.***> wrote:
Hi @webtroter <https://github.com/webtroter>, @tobiasehlert
<https://github.com/tobiasehlert>! 👋🏼
I see you upvoted this issue, do you have a similar problem in your
deployments? If yes, would you mind sharing your configuration?
Thanks!
—
Reply to this email directly, view it on GitHub
<#586 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AB77R6BIXTOD2RODIAG7G53X65ITZANCNFSM6AAAAAA5DIECAY>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
|
@bpg, well slightly the same. Example terraform file: resource "proxmox_virtual_environment_file" "vm_cloud_init" {
content_type = "snippets"
datastore_id = "local"
node_name = var.proxmox_settings.node
source_raw {
file_name = "cloud-init.user-data.yml"
data = file("cloud-init/user-data.yml")
}
}Example cloud-init/user-data.yml file: #cloud-config
package_upgrade: true
packages:
- qemu-guest-agent
timezone: Europe/Stockholm
users:
- name: myusername
groups: [adm, cdrom, dip, plugdev, lxd, sudo]
lock-passwd: false
sudo: ALL=(ALL) NOPASSWD:ALL
shell: /bin/bash
ssh_authorized_keys:
- ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAINyDI3c8l5Se4w0B2ejL36WJh/PdD5MKKsL9zX0a9sT
power_state:
delay: now
mode: reboot
message: Rebooting after cloud-init completion
condition: trueProxmox GUI: My expectation was to see my user that was provisioned and the ssh-key, but that's not shown. I run version Proxmox version 8.0.3 and Terraform provider bpg/proxmox version 0.33.0. |
|
Also (maybe related to cloud-init issue), the proxmox_virtual_environment_file resource for the cloud-init file shown in comment returns following error to me when applying: |
|
@tobiasehlert could you pls check if |
|
I've spent a whole day yesterday and quite a bit of overtime experimenting with provider and testing cloud-init functionality specifically and here's my observations. PART I I'm not sure if people are aware but things that Proxmox exposes in it's GUI for Cloud-Init is just a small subset of what cloud-config is capable of. These things are what provider can control via "initialization.dns", "initialization.ip_config" and "initialization.user_account" (btw @bpg there seem to be no way for provider to change "Upgrade packages" option as in GUI?): But that is in no way a full extent of what could be done via cloud-config and by using provider's arguments like "user_data_file_id" (or network/vendor/meta ones) we could inject our own custom cloud-init cloud-config. # freshly baked template with an injected cloud-config shows nothing for it
root@prox-srv1:~# qm cloudinit dump 1011 user
#cloud-config
hostname: template-ubuntu-22
manage_etc_hosts: true
fqdn: template-ubuntu-22.04
chpasswd:
expire: False
users:
- default
package_upgrade: true
# i try to inject cloud-config via console tool but it still shows nothing afterwads
root@prox-srv1:~# qm set 1011 --cicustom "user=local:snippets/userdata-proxmox.yml"
update VM 1011: -cicustom user=local:snippets/userdata-proxmox.yml
root@prox-srv1:~# qm cloudinit dump 1011 user
#cloud-config
hostname: template-ubuntu-22
manage_etc_hosts: true
fqdn: template-ubuntu-22.04
chpasswd:
expire: False
users:
- default
package_upgrade: true
# but if i change "User" field of Cloud-Init in Proxmox's GUI it detects changes
root@prox-srv1:~# qm cloudinit dump 1011 user
#cloud-config
hostname: template-ubuntu-22
manage_etc_hosts: true
fqdn: template-ubuntu-22.04
user: WOLOLO
chpasswd:
expire: False
users:
- default
package_upgrade: trueIn the end it's very weird for me how Proxmox handles cloud-init and i'm not sure what exactly it shows via "qm cloudinit dump". Why it's only reflection of GUI's changes i've no clue... PART II ratiborus@HOMEWORLD:~/WORKSPACE/terraform_yandex/proxmox$ cat ../keys/userdata-proxmox.yml
#cloud-config
hostname: px-cloud-config-applied
users:
- name: tf
groups: adm,sudo
shell: /bin/bash
sudo: ['ALL=(ALL) NOPASSWD:ALL']
ssh-authorized-keys:
- ssh-rsa AAAABsomelongstring== tf
packages:
- qemu-guest-agent
package_upgrade: true
runcmd:
- systemctl enable --now qemu-guest-agent
- echo "done" > /tmp/user-cloud-init-done
final_message: |
cloud-init has finished
version: $version
timestamp: $timestamp
datasource: $datasource
uptime: $uptime
Ratiborus was here... And here's 3 different implementations for injecting custom cloud-init cloud-config for userdata, and all of them work:
#==================#
# CLOUD-CONFIG #
#==================#
resource "proxmox_virtual_environment_file" "cloud_config_userdata" {
content_type = "snippets"
datastore_id = "local"
node_name = "prox-srv1"
source_file {
path = "../keys/userdata-proxmox.yml"
}
}
resource "proxmox_virtual_environment_file" "cloud_config_userdata_raw_file" {
content_type = "snippets"
datastore_id = "local"
node_name = "prox-srv1"
source_raw {
data = file("../keys/userdata-proxmox.yml")
file_name = "userdata-proxmox-raw-file.yml"
}
}
resource "proxmox_virtual_environment_file" "cloud_config_userdata_raw" {
content_type = "snippets"
datastore_id = "local"
node_name = "prox-srv1"
source_raw {
data = <<EOF
#cloud-config
hostname: px-cloud-config-applied
packages:
- qemu-guest-agent
users:
- name: tf
groups: sudo
shell: /bin/bash
ssh-authorized-keys:
- ssh-rsa some_long_key
sudo: ALL=(ALL) NOPASSWD:ALL
runcmd:
- systemctl enable --now qemu-guest-agent
- echo "done" > /tmp/user-cloud-init-done
EOF
file_name = "userdata-proxmox-raw.yml"
}
}And now about the issue with the first implementation i mentioned earlier. Here's an example: |
|
And another longread. I'm sorry... @bpg Also, even with "started = false" i'm still getting this "architecture" update for template VM after it was already created (once on the next plan generation): It happens only once after template VM resource creation and also does not happen if i recreate resource (i.e. with "terraform taint"). No difference whether "template = true" or "false". Not sure but maybe all that info should be covered in a dedicated issue? Just feel like i'm a bit off-topic here and deluting main question :) |
|
Thanks for the wealth of information, @ratiborusx. That's a really impressive investigation! I'll need to set aside some time to go over all the details here. I'll get back to you at some point over the weekend :) |
|
Just a little bit of new information that was revealed at today's past-midnight meeting and demo. An example:
I saw those in resource description but thought they just overwrite each other. UPD. @tobiasehlert |
Yes, that fixed my problem with snippet, thanks @bpg! |
|
Hello ! I also have this issue (I didn't wanted to open a duplicate, feel free to ask me to open my own issue). Host :
Template :
Terraform code (redacted) : terraform {
required_providers {
proxmox = {
source = "bpg/proxmox"
version = "0.37.0"
}
}
}
provider "proxmox" {
endpoint = "https://my.awesome.server:8006/"
api_token = "user@pve!token1=my-awesome-token"
insecure = true
tmp_dir = "/var/tmp"
}
resource "proxmox_virtual_environment_vm" "ubuntu_vm" {
name = "terraform-provider-proxmox-ubuntu-vm"
description = "Managed by Terraform"
tags = ["terraform", "ubuntu"]
node_name = "node1"
vm_id = 4321
agent {
# read 'Qemu guest agent' section, change to true only when ready
enabled = true
}
clone {
vm_id = "101"
}
bios = "ovmf"
reboot = true
boot_order = ["ide2","virtio0","net0"]
disk {
datastore_id = "datastore1"
file_format = "raw"
interface = "virtio0"
size = 25
}
disk {
datastore_id = "datastore1"
file_format = "raw"
interface = "virtio1"
size = 4
}
network_device {
bridge = "vmbr0"
vlan_id = "1"
}
operating_system {
type = "l26"
}
scsi_hardware = "virtio-scsi-single"
initialization {
datastore_id = "datastore1"
interface = "ide2"
ip_config {
ipv4 {
address = "1.2.3.4/24"
gateway = "1.2.3.1"
}
}
}
}
The VM is created from template with correct boot order, but after disk resize, the boot order is changed to I tried to remove the Cloud Init Drive to let Terraform create it (Packer create this drive at ide0 at the last step) and set it to ide2, but I have the same issue. Since the Cloud init drive doesn't appear in boot order, the VM doesn't see it, so Cloud Init never configure the IP address. And the Terraform execution hangs up undefinitively (well, I killed it at 2min and at 10min before). When I create a VM with |
|
@Fabiosilvero I don't think Terraform's provider will be able to save whatever you backed in your image about boot order, it will always rewrite it. Even if you don't specify it in the resource i think it'll default to "boot_order = []" which most likely means to boot up from the first drive it detects. Just to be sure i'd recommend you to try to create a template with an official cloud image of Debian 12 and see if the issue is image-related. I used this one and it works without any problems with cloud-config: UPD. |
|
Hello and thanks for your reply ! I can't get the cloud image to boot with OVMF even with the efi_disk fully configured on terraform side (I used the tar.xz image as qcow2 wasn't valid for the provider - maybe setting the type to snipplet would have been sufficient tho). I'll try again later but I don't think the template is the issue as it works perfectly with Telmate provider. I get what you mean by asking to remove the boot_order directive, that was a workaround since with OVMF, it seems that if disks aren't present in boot_order, they aren't properly recognized (see https://forum.proxmox.com/threads/cloud-init-drive-not-auto-regenerating.117318/ and https://forum.proxmox.com/threads/ide-cloud-init-volumes-are-not-usable-when-cold-booting-a-vm.117345/). I tried first without setting it, and the result was the same. My configuration quoted was misleading, that was a fluke but I tried with virtio0 first (and ide2 at the end) and I tried without ide2 : still same thing. I don't think that this provider enforce the order in the Terraform config at the very end of the execution, but instead before any disks resize - and the boot order is altered at that moment again, causing config revert to default (that's just a guess). Maybe it's a "Regenerate Image" trigger issue in this provider : if I stop the VM then click on "Regenerate Image" on the Web GUI, the Cloud Init drive appears in Boot Order and the Cloud Init service kicks in at startup. I'll try to generate a Seabios template and see if the issue persists. EDIT : It works fine with BIOS Template, so I think it's a problem with UEFI/OVMF-based templates :/ |
|
Combination Cloud-init docs do not mention anything about different requirements when booted using UEFI. Added 2 lines to a debian example: and created the VM. Cloud-init correctly applied the configuration. It is likely unsafe to modify the provider to change |
|
Hey again, After a long time of testing around with different configs, I still got no results. But now I tried the Config from @otopetrik (thank you btw) and it works. When i use scsi1 here, all the CI config is getting applied to the VM on first boot. So thanks to all of you that helped. |
|
Well, thank you ! I just changed the interface from ide0/2 to scsi1, and it worked flawlessly in OVMF setting. Maybe a VM with OVMF requires that IDE drives should be present at boot order to be available in VMs at all, and SCSI drives don't need that (I don't see the Cloud Init drive in Boot order). |
|
I've learned some things about Proxmox+cloud-init which may add some value to some areas of this thread.
In general, the clobbering nature of |
|
@windowsrefund
|
bpg
added
✨ enhancement
|
@all-contributors please add @Fabiosilvero and @windowsrefund for testing |
|
I've put up a pull request to add @Fabiosilvero! 🎉 |
|
@all-contributors please add @windowsrefund for testing |
|
I've put up a pull request to add @windowsrefund! 🎉 |
|
Hello guys, Cloud init with IDE2 (default) and EFI is working on my side, you can find the attached apply i've posted in an other issue #734 (comment) Tested with Proxmox Virtual Environment 8.0.9, Terraform v1.6.4 and Ubuntu 22.04.3. I'm creating a template with packer, with some ansible scripts (in the same repo), at the end I reset cloud init instance with Note : I use cloud init file |
This was referenced Nov 28, 2023
|
Hello guys, I don't really understand why the cloud-init configuration files are split in this way? Can't we just put one cloud-init file and then --cicustom in the backend? The way it's handled here seems cumbersome to me |
|
Hi @cursedforever! That's what the Proxmox API provides us:
Well, they missed I'm not a cloud-init expert by any means, so can't comment on "why" it's done this way. Tough I did try passing user config in |
|
Ok thank you very much, I've tried to make a global cloud-init file and pass it to the user_config_file variable and it works. |
|
Using v0.49.0 with Proxmox 8.1.4, I am experiencing the same issue as described in the very first comment. Both Adding |
|
Just for reference, more Proxmox users seem to have related issues with the cloud-init IDE interface, where switching to SCSI fixes the issue: https://forum.proxmox.com/threads/cloud-init-image-only-applies-configuration-on-second-boot.93414/post-452830 Maybe the configuration default should be changed from IDE to SCSI? |
|
I personally never had problems with IDE interface for cloud-init but i must admit i never used UEFI ('bios = ovmf') either because i didn't need PCI devices pass-through so far. lifecycle {
ignore_changes = [
initialization[0].datastore_id,
initialization[0].interface
]
}But i'm absolutely not against someone smart fixing that thing for good. I'm just mentally handicapped in that relation. |
|
Marking this issue as stale due to inactivity in the past 180 days. This helps us focus on the active issues. If this issue is reproducible with the latest version of the provider, please comment. If this issue receives no comments in the next 30 days it will automatically be closed. If this issue was automatically closed and you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thank you! |
|
I'm using 0.64.0 and the issue is still there. Just writing this to prevent it from being closed as stale. Default cloud-image configuration from snippets isn't applied. |
|
Hey @ElForastero 👋🏼 The bottom line is that, in most cases discussed above, the problem wasn't with the provider but with VM configuration, cloud-init settings, the base OS image, etc. I'm keeping this issue open mainly to summarize those aspects in the documentation. I suggest opening a new issue and explaining your use case so we can investigate and offer guidance. Thanks! |
Repository owner
locked as resolved and limited conversation to collaborators
Sep 22, 2024
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Describe the bug
My issue is: When applying a terraform manifest that clones a vm and adds cloud init configuration, the cloudinit image doesnt get regenerated.
I the UI this is done via the Button Regererate image:
There is also a API call that can do this: API Refference
To Reproduce
Steps to reproduce the behavior:
terraform applyMy TF Module:
Screenshots
If applicable, add screenshots to help explain your problem.
Additional context
Add any other context about the problem here.
The text was updated successfully, but these errors were encountered: