Terraform crashes (while creating instances) #541
Comments
ghost
added
|
Maybe to add:
What I am unsure is, why this "just" worked (an hour ago), and now I get these errors. |
|
@till Yes, this does seem similar to #506. The similarity between this and #506 is that you two are making some kind of Kubernetes infrastructure? Are you basing your configuration off of some published project or article? From what I can see from the debug logs, you two are using two different cloud providers, so it wouldn't be a case of a single cloud causing this. Can you provide a Terraform confirmation ( Have you been able to reproduce this issue across multiple multiple builds (meaning: has this happened, say, 2 out of 10 times)? |
It's doubtful that any changes since the last release will fix this problem. The better solution would be to apply #539 and see if that helps. If you are familiar with compiling Go binaries, then the following should work: $ go get github.com/terraform-providers/terraform-provider-openstack
$ cd $GOPATH/src/github.com/terraform-providers/terraform-provider-openstack
$ make build
$ cp $GOPATH/bin/terraform-provider-openstack /path/to/where/original/is |
|
@till I think I see the problem. In the debug log you've provided, search for: The later: I think what's happening is that the ports are being created before the subnet has finished creating. This means that the ports will not have a fixed IP at the time of their creation. This is a common situation to be in because a port doesn't have a natural dependency on a subnet, even though 99% of the time, you want a port associated with a subnet. Therefore, you have to make an explicit dependency: resource "openstack_networking_subnet_v2" "subnet_1" {
name = "subnet_1"
network_id = "${openstack_networking_network_v2.network_1.id}"
cidr = "192.168.1.0/24"
ip_version = 4
enable_dhcp = true
no_gateway = true
}
resource "openstack_networking_port_v2" "port_1" {
depends_on = [
"openstack_networking_subnet_v2.subnet_1",
]
name = "port_1"
network_id = "${openstack_networking_network_v2.network_1.id}"
admin_state_up = "true"
}I recommend trying that in your configuration and seeing how things go. If that works, then I have two ideas:
|
|
@jtopjian I had my network stuff in one module, and the instance creation, including ports, in another. I will read up on |
|
Depends on doesn't seem to work across modules. When I try to run |
|
@jtopjian part of this seemed to have worked. I did the following:
However, there is a new issue: When I run I listed the ports with |
|
Part solved: The port mess was from a previous crash. Cleaning up the ports helped, but eventually, I just end up with a crash with: The effects are:
|
(I think that's still what your PR is trying to address, but unsure as of "root" cause.) |
|
Thank you for looking into this.
haha - sounds good ;) Since both reported issues were using a module to build Kubernetes infrastructure, I thought there might have been a published module somewhere.
Correct. This is a popular issue. One workaround is to define an See here for similar issues and workarounds:
Indeed - the same area of code is being hit. I'm quite sure this is still an ordering issue of some sort. I'll see if I can reproduce it locally. |
|
@till well, this is interesting. I reproduced the problem, but the two clouds I tested both returned a proper error: Here's the short config I used: resource "openstack_networking_network_v2" "network_1" {
name = "network_1"
}
resource "openstack_networking_subnet_v2" "subnet_1" {
name = "subnet_1"
network_id = "${openstack_networking_network_v2.network_1.id}"
cidr = "192.168.1.0/24"
ip_version = 4
enable_dhcp = true
}
resource "openstack_networking_port_v2" "port_1" {
/*
depends_on = [
"openstack_networking_subnet_v2.subnet_1",
]
*/
name = "port_1"
network_id = "${openstack_networking_network_v2.network_1.id}"
admin_state_up = "true"
}
resource "openstack_compute_instance_v2" "instance_1" {
name = "instance_1"
network {
port = "${openstack_networking_port_v2.port_1.id}"
}
} I'm curious if you see the same error or if you get the panic/crash. If you don't see the panic, would you mind posting your setup so I can try to reproduce what you're seeing? |
|
@jtopjian yeah, I can definitely share. I am trying to refactor a few things to get working. Btw, I can report that when I keep network setup and instance/port creation separate, it succeeds. As I said above, my work around was to not just return the I then used a But that still doesn't seem to work. I end up with ports without IPs, etc.. When I execute the same code sequentially ( Your PR in #539 fixes it in a way where, some resources fail, but I am not left with a full on crash (which doesn't seem to preserve all the state despite best attempts made). |
OK cool. At least it's stopping the crash. Once I'm able to reproduce this locally, I can look at a better fix. Thank you :) |
|
@jtopjian I will need some additional time to pick the code apart. I am gonna try to do this first week of January. But yeah, it doesn't fix the original issue yet (instance doesn't have IPs) as the dependencies are not resolved, but the crash is gone. |
|
Sounds good :) |
|
Just faced the same issue. Panic OutputSteps to ReproducePlease list the steps required to reproduce the issue, for example:
Important Factoidsresource "openstack_networking_port_v2" "my_app" {
name = "my_app"
network_id = "505e3d28-863d-44be-a4ba-3a2ffb5e9abe"
no_fixed_ip = true
}
resource "openstack_compute_instance_v2" "my_app" {
name = "my_app"
flavor_id = "20"
key_pair = "keypair"
image_id = "67680990-4cd7-4b7f-9ea1-2249e67d0a9c"
network {
port = "${openstack_networking_port_v2.my_app.id}"
}
} |
|
@jtopjian 🎉 Thank you for fixing it. Do you know when a new release is coming? |
|
@till Yes, 30 minutes ago 😉 |
|
Hi @jtopjian, Error: Error applying plan:
1 error(s) occurred:
* module.serena1-control-01.openstack_compute_instance_v2.instance: 1 error(s) occurred:
* openstack_compute_instance_v2.instance: Error creating OpenStack server: Bad request with: [POST https://10.75.11.228:13774/v2.1/os-volumes_boot], error message: {"badRequest": {"message": "Port 2e4a4f1f-1bbe-4bc8-b58c-fa9f8224114d requires a FixedIP in order to be used.", "code": 400}}Terraform Version：
openstack.tf resource "openstack_networking_port_v2" "port_net-02_v4" {
name = "${var.name_prefix}-${var.hostname}"
network_id = "${var.net-02_net_uuid}"
admin_state_up = "true"
no_security_groups = "false"
no_fixed_ip = true
count = "${var.count}"
}
resource "openstack_compute_instance_v2" "instance" {
name = "${var.name_prefix}-${var.hostname}"
key_pair = "${var.keypair_name}"
flavor_id = "${var.flavor_uuid}"
image_id = "${var.image_uuid}"
config_drive = "true"
availability_zone = "${var.zone_of_instance}"
network = {
port = "${openstack_networking_port_v2.port_net-02_v4.id}"
}
} |
|
@SerenaLi279 Unfortunately I'm not sure what the problem might be. This issue was opened because Terraform was actually crashing (panic + stack trace) so the goal was to stop that from happening and have Terraform report a proper error (which is what you're seeing). Per my comment here (https://github.com/terraform-providers/terraform-provider-openstack/issues/541#issuecomment-448451486), I was unable to actually get an instance created without an IP (I got the same error as you). I'm not sure if others are able to because their clouds have a special configuration or if they are using a different set of Terraform resources. |
|
Hi @jtopjian, |
|
@jtopjian, upon review ip less interfaces while supported by neutrons api are not supported by nova in OSP 13
currently no release of openstack nova upstream allows the use of neutron ports with ip_allocation=none.
this will be considerded as a possible Future Feature for OSP 16 based on discussion with upstream.so looks like attaching a port without fixed ip is currently only supported for already running VMs. so no_fixed_ip could works well with openstack_compute_interface_attach_v2 to attach ports after vm has created like: resource "openstack_networking_port_v2" "port_net-02_v4" {
name = "${var.name_prefix}-${var.hostname}"
network_id = "${var.net-02_net_uuid}"
admin_state_up = "true"
security_group_ids = [
]
no_fixed_ip = true
count = "${var.count}"
}
resource "openstack_networking_port_v2" "port_net-03_v4" {
name = "${var.name_prefix}-${var.hostname}"
network_id = "${var.net-02_net_uuid}"
admin_state_up = "true"
security_group_ids = [
]
no_fixed_ip = true
count = "${var.count}"
}
resource "openstack_networking_port_v2" "port_net-01_v4" {
name = "${var.name_prefix}-${var.hostname}"
network_id = "${var.net-01_net_uuid}"
admin_state_up = "true"
security_group_ids = [
]
fixed_ip {
"subnet_id" = "${var.net-01_v4_sub_uuid}"
"ip_address" = "${var.net-01_static_ipv4_address}"
}
allowed_address_pairs {
ip_address = "${var.overlay_network}"
}
count = "${var.count}"
}
resource "openstack_compute_instance_v2" "instance" {
name = "${var.name_prefix}-${var.hostname}"
key_pair = "${var.keypair_name}"
flavor_id = "${var.flavor_uuid}"
image_id = "${var.image_uuid}"
config_drive = "true"
availability_zone = "${var.zone_of_instance}"
network = {
port = "${openstack_networking_port_v2.port_net-01_v4.id}"
}
count = "${var.count}"
user_data = "${var.user_data}"
}
resource "openstack_compute_interface_attach_v2" "test2" {
instance_id = "${openstack_compute_instance_v2.instance.id}"
port_id = "${openstack_networking_port_v2.port_net-02_v4.id}"
}
resource "openstack_compute_interface_attach_v2" "test3" {
depends_on = ["openstack_compute_interface_attach_v2.test2"]
instance_id = "${openstack_compute_instance_v2.instance.id}"
port_id = "${openstack_networking_port_v2.port_net-03_v4.id}"
} |
Terraform Version
Affected Resource(s)
I am using the OpenStack provider only to:
I can say that the first 3 steps always work. Creating the ports also seems to be successful always, but then it crashes while creating the instances.
I am creating a total of 4 instances in my test — 3 using my custom module (which wraps port and instance creating — see below) and one with straight Terraform (a jump host, which is connected to the private network it created and an existing public network).
Terraform Configuration Files
I have a terraform repository with the usual (variables, deploy, ...) and two sub-modules which wrap:
k8s_network)k8s_node)deploy.tfto add a jump host once the rest is doneI can share all that (sans
variables.tf), if required.Panic Output
https://gist.github.com/till/a5e401bf883c91f02e703235ca26b7ee
Expected Behavior
Network gets created, ports get created, instances are attached. Clean exit.
Actual Behavior
Most of the above happens, but Terraform crashes (and is unable to capture state).
In addition this is output:
Steps to Reproduce
source /etc/kolla/admin-openrcterraform applyQuestions I had
I noticed lots of PRs closed since last release, but I am unable to figure out how to run the provider from "master". Any hints appreciated, then I will re-test.
The text was updated successfully, but these errors were encountered: