New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

failed to fetch config: not a config (empty) when using terraform #2456

Open
jaschaio opened this Issue Jun 12, 2018 · 4 comments

Comments

Projects
None yet
3 participants
@jaschaio

jaschaio commented Jun 12, 2018

I first thought that this might be a bug of terraform like reported in terraform-providers/terraform-provider-digitalocean#100, but after investigating a bit further it seems that the ignition file is actually properly supplied to the Droplet.

This is my ignition file

{
    "ignition": {
        "config": {},
        "security": {
            "tls": {}
        },
        "timeouts": {},
        "version": "2.2.0"
    },
    "networkd": {},
    "passwd": {},
    "storage": {
    "files": [{
            "filesystem": "root",
            "path": "/etc/ssh/sshd_config",
            "contents": {
                "source": "data:,%23%20Use%20most%20defaults%20for%20sshd%20configuration.%0AUsePrivilegeSeparation%20sandbox%0ASubsystem%20sftp%20internal-sftp%0AUseDNS%20no%0A%0APermitRootLogin%20no%0AAllowUsers%20core%0AAuthenticationMethods%20publickey%0A",
                "verification": {}
            },
            "mode":384
        },
        {
            "filesystem": "root",
            "path": "/etc/sysctl.d/vm.conf",
            "contents": {
                "source": "data:,vm.max_map_count%3D262144%0A",
                "verification": {}
            },
            "mode":420
        },
        {
            "filesystem": "root",
            "path": "/etc/coreos/update.conf",
            "contents": {
                "source": "data:,%0AREBOOT_STRATEGY%3D%22etcd-lock%22",
                "verification": {}
            },
            "mode":420
        }]
    },
    "systemd": {
        "units": [{
            "dropins": [{
                "contents": "[Unit]\nRequires=coreos-metadata.service\nAfter=coreos-metadata.service\n\n[Service]\nEnvironmentFile=/run/metadata/coreos\nExecStart=\nExecStart=/usr/lib/coreos/etcd-wrapper $ETCD_OPTS \\\n  --listen-peer-urls=\"http://${COREOS_DIGITALOCEAN_IPV4_PRIVATE_0}:2380\" \\\n  --listen-client-urls=\"http://0.0.0.0:2379,http://0.0.0.0:4001\" \\\n  --initial-advertise-peer-urls=\"http://${COREOS_DIGITALOCEAN_IPV4_PRIVATE_0}:2380\" \\\n  --advertise-client-urls=\"http://${COREOS_DIGITALOCEAN_IPV4_PRIVATE_0}:2379,http://${COREOS_DIGITALOCEAN_IPV4_PRIVATE_0}:4001\" \\\n  --discovery=\"https://discovery.etcd.io/43890d7ee9635609b84a067bd7214623\"",
                "name": "20-clct-etcd-member.conf"
            }],
            "enable":true,
            "name": "etcd-member.service"
        },
        {
            "dropins": [{
                "contents": "[Socket]\nListenStream=\nListenStream=12345\n",
                "name": "10-sshd-port.conf"
            }],
            "name": "sshd.socket"
        },
        {
            "contents": "[Unit]\nDescription=DockerPS\nAfter=docker.service\nRequires=docker.service\n\n[Service]\nType=oneshot\nExecStart=-/usr/bin/docker ps\n\n[Install]\nWantedBy=multi-user.target\n",
            "enabled":true,"name": "dockerps.service"
        }]
    }
}

If I supply it as user_data via the DigitalOcean UI it works.

If I supply it via terraform it doesn't.

This is the output from journalctl --identifier=ignition --all on a manually provisioned Droplet via the Digitalocean UI (supplying user_data in the respective field):

[…]
Jun 12 14:04:04 localhost ignition[405]: GET http://169.254.169.254/metadata/v1/user-data: attempt hashicorp/terraform#6
Jun 12 14:04:04 localhost ignition[405]: GET result: OK
Jun 12 14:04:04 localhost ignition[405]: parsing config: {"ignition":{"config":{},"security":{"tls":{}},"timeouts":{},"version":"2.2.0"},"networkd":{},"passwd":{},"storage":{"files":[{"filesystem":"root",
[…]

This is the output from journalctl --identifier=ignition --all on a Droplet that was provisioned using terraform (with the user_data directive):

Jun 12 13:37:24 localhost ignition[415]: GET http://169.254.169.254/metadata/v1/user-data: attempt hashicorp/terraform#6
Jun 12 13:37:24 localhost ignition[415]: GET result: OK
Jun 12 13:37:24 localhost ignition[415]: parsing config:
Jun 12 13:37:24 localhost ignition[415]: failed to fetch config: not a config (empty)
Jun 12 13:37:24 localhost ignition[415]: not a config (empty): ignoring user-provided config

If I ssh into the droplet and curl http://169.254.169.254/metadata/v1/user-data myself I get back the ignition config posted above.

So in theory this should work, but it doesn't.

@sdemos

This comment has been minimized.

Show comment
Hide comment
@sdemos

sdemos Jun 12, 2018

Member

Hi! Thanks for the report. Unfortunately, I can't seem to reproduce this issue. I used your issue on terraform-provider-digitalocean to try and recreate a terraform config that would have the same issue. This is what I came up with -

# main.tf
provider "digitalocean" {
  token = "${chomp(file("~/.config/digital-ocean/token"))}"
}

resource "digitalocean_droplet" "test" {
  image = "coreos-stable"
  size = "s-1vcpu-2gb"
  count = 1
  name = "sdemos-test"
  region = "sfo2"
  private_networking = true
  resize_disk = false
  user_data = "${file("config/ignition")}"
  ssh_keys = ["46:9e:68:ed:21:b0:1e:76:75:16:6d:e1:3a:a8:49:6a"]
}

I left out the provisioner "file" and provisioner "remote-exec" sections since you mentioned omitting those didn't fix the issue.

# config/ignition
{
  "ignition": {
    "config": {},
    "security": {
      "tls": {}
    },
    "timeouts": {},
    "version": "2.2.0"
  },
  "networkd": {},
  "passwd": {
    "users": [
      {
        "name": "core",
        "sshAuthorizedKeys": [
          "ssh-rsa [omitted] demos@anduin"
        ]
      }
    ]
  },
  "storage": {
    "files": [
      {
        "filesystem": "root",
        "path": "/home/core/test",
        "contents": {
          "source": "data:,Hello%2C%20world!",
          "verification": {}
        },
        "mode": 420
      }
    ]
  },
  "systemd": {}
}

Then I just used terraform apply and it successfully created a droplet with the expected configuration. It shows up as a machine running the current coreos-stable image on DO (1745.6.0). Are you doing something else on top of that, either in your terraform configuration or your invocation, that might cause something to be configured differently?

Member

sdemos commented Jun 12, 2018

Hi! Thanks for the report. Unfortunately, I can't seem to reproduce this issue. I used your issue on terraform-provider-digitalocean to try and recreate a terraform config that would have the same issue. This is what I came up with -

# main.tf
provider "digitalocean" {
  token = "${chomp(file("~/.config/digital-ocean/token"))}"
}

resource "digitalocean_droplet" "test" {
  image = "coreos-stable"
  size = "s-1vcpu-2gb"
  count = 1
  name = "sdemos-test"
  region = "sfo2"
  private_networking = true
  resize_disk = false
  user_data = "${file("config/ignition")}"
  ssh_keys = ["46:9e:68:ed:21:b0:1e:76:75:16:6d:e1:3a:a8:49:6a"]
}

I left out the provisioner "file" and provisioner "remote-exec" sections since you mentioned omitting those didn't fix the issue.

# config/ignition
{
  "ignition": {
    "config": {},
    "security": {
      "tls": {}
    },
    "timeouts": {},
    "version": "2.2.0"
  },
  "networkd": {},
  "passwd": {
    "users": [
      {
        "name": "core",
        "sshAuthorizedKeys": [
          "ssh-rsa [omitted] demos@anduin"
        ]
      }
    ]
  },
  "storage": {
    "files": [
      {
        "filesystem": "root",
        "path": "/home/core/test",
        "contents": {
          "source": "data:,Hello%2C%20world!",
          "verification": {}
        },
        "mode": 420
      }
    ]
  },
  "systemd": {}
}

Then I just used terraform apply and it successfully created a droplet with the expected configuration. It shows up as a machine running the current coreos-stable image on DO (1745.6.0). Are you doing something else on top of that, either in your terraform configuration or your invocation, that might cause something to be configured differently?

@jaschaio

This comment has been minimized.

Show comment
Hide comment
@jaschaio

jaschaio Jun 13, 2018

Thanks for your quick reply.

Your config worked for me as well. So I took another look at my terraform config to see what might be causing the issue and I think its the Following:

I first build an image based on coreos-beta with packer

Instead of than using image = "coreos-beta" in my terraform droplet ressource, I use the image ID I got back from packer.

Is it possible that I can only ever supply user_data and make use of it on the inital provision of a Droplet? So when I create the packer image without user_data (or ignition config) and than try to create a new droplet based on that image, this time with a custom ignition config it ignores it?

What I am basically trying to do is to make the droplet creation faster by doing some configuration in the packer image. I initially had two ignition files, one for packer and one for terraform. The packer ignition file made general changes to the system which would apply to all droplets. The terraform ignition file initiated the etcd cluster. I couldn't add the etcd part to the packer ignition file as the IP of the droplet which is used by packer to build the image shouldn't be added to etcd and the IPs are only known after a droplet has been created through terraform. I think that the second ignition file supplied through terraform was never applied so thats why I moved all over to terraform.

It seems that in the combination of using packer & terraform only one of them can have a ignition file that will be applied upon boot

jaschaio commented Jun 13, 2018

Thanks for your quick reply.

Your config worked for me as well. So I took another look at my terraform config to see what might be causing the issue and I think its the Following:

I first build an image based on coreos-beta with packer

Instead of than using image = "coreos-beta" in my terraform droplet ressource, I use the image ID I got back from packer.

Is it possible that I can only ever supply user_data and make use of it on the inital provision of a Droplet? So when I create the packer image without user_data (or ignition config) and than try to create a new droplet based on that image, this time with a custom ignition config it ignores it?

What I am basically trying to do is to make the droplet creation faster by doing some configuration in the packer image. I initially had two ignition files, one for packer and one for terraform. The packer ignition file made general changes to the system which would apply to all droplets. The terraform ignition file initiated the etcd cluster. I couldn't add the etcd part to the packer ignition file as the IP of the droplet which is used by packer to build the image shouldn't be added to etcd and the IPs are only known after a droplet has been created through terraform. I think that the second ignition file supplied through terraform was never applied so thats why I moved all over to terraform.

It seems that in the combination of using packer & terraform only one of them can have a ignition file that will be applied upon boot

@sdemos

This comment has been minimized.

Show comment
Hide comment
@sdemos

sdemos Jun 13, 2018

Member

I don't have experience with packer, but from my understanding of what you are saying, it sounds like you are provisioning a CoreOS machine with packer, which then takes a snapshot of the resulting machine, and lets you boot from that snapshot?

Ignition only runs once on a particular machine, only on the very first boot. After that, if you make changes to your ignition config and want them applied to your machine, you have to reprovision it from scratch. If packer is provisioning a machine, then terraform is using that already provisioned machine, ignition will never run again, and so no ignition configuration that you provide later will ever be applied.

Most of the time, terraform is aware of this constraint, and if you run terraform apply with a different ignition config, it will reprovision all your machines for you. However, if it thinks it's starting with a fresh image, it has no way to know about that.

Member

sdemos commented Jun 13, 2018

I don't have experience with packer, but from my understanding of what you are saying, it sounds like you are provisioning a CoreOS machine with packer, which then takes a snapshot of the resulting machine, and lets you boot from that snapshot?

Ignition only runs once on a particular machine, only on the very first boot. After that, if you make changes to your ignition config and want them applied to your machine, you have to reprovision it from scratch. If packer is provisioning a machine, then terraform is using that already provisioned machine, ignition will never run again, and so no ignition configuration that you provide later will ever be applied.

Most of the time, terraform is aware of this constraint, and if you run terraform apply with a different ignition config, it will reprovision all your machines for you. However, if it thinks it's starting with a fresh image, it has no way to know about that.

@bendrucker

This comment has been minimized.

Show comment
Hide comment
@bendrucker

bendrucker Jun 22, 2018

Ran into this and came across this issue as well as #2090. It's not perfect but it seems like the following workaround causes Ignition to run as expected the second time:

sudo touch /boot/coreos/first_boot
sudo rm /etc/machine-id

When I encountered this I was building an image (for AWS) with Packer. I wasn't provisioning that image with Ignition but hit this anyway when provisioning via Terraform/user_data.

I haven't had a chance to track down why it's parsed as an empty response resulting in the confusing error.

bendrucker commented Jun 22, 2018

Ran into this and came across this issue as well as #2090. It's not perfect but it seems like the following workaround causes Ignition to run as expected the second time:

sudo touch /boot/coreos/first_boot
sudo rm /etc/machine-id

When I encountered this I was building an image (for AWS) with Packer. I wasn't provisioning that image with Ignition but hit this anyway when provisioning via Terraform/user_data.

I haven't had a chance to track down why it's parsed as an empty response resulting in the confusing error.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment