New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Error creating backend service #4336

Closed
yosssi opened this Issue Dec 16, 2015 · 15 comments

Comments

Projects
None yet
8 participants
@yosssi

yosssi commented Dec 16, 2015

I got the following error when I try to create a backend service:

google_compute_backend_service.my-backend-service: Creating...
  backend.#:                               "" => "1"
  backend.499478644.balancing_mode:        "" => "UTILIZATION"
  backend.499478644.capacity_scaler:       "" => "1"
  backend.499478644.description:           "" => ""
  backend.499478644.group:                 "" => "https://www.googleapis.com/replicapool/v1beta2/projects/myproject-dev/zones/asia-east1-a/instanceGroupManagers/gke-myproject-api-27da66d4-group"
  backend.499478644.max_rate:              "" => ""
  backend.499478644.max_rate_per_instance: "" => ""
  backend.499478644.max_utilization:       "" => "0.8"
  description:                             "" => "xxxx"
  fingerprint:                             "" => "<computed>"
  health_checks.#:                         "" => "1"
  health_checks.2686024285:                "" => "https://www.googleapis.com/compute/v1/projects/myproject-dev/global/httpHealthChecks/myi-http-basic-check"
  name:                                    "" => "my-backend-service"
  port_name:                               "" => "http"
  protocol:                                "" => "HTTP"
  self_link:                               "" => "<computed>"
  timeout_sec:                             "" => "<computed>"

Error applying plan:

1 error(s) occurred:

* google_compute_backend_service.my-backend-service: Error creating backend service: googleapi: Error 400: Invalid value for field 'resource.backends[0]': ''.  Invalid group url, invalid

Terraform does not automatically rollback in the face of errors.
Instead, your Terraform state file has been partially updated with
any resources that successfully completed. Please address the error
above and apply again to incrementally change your infrastructure.

Following is my template:

resource "google_compute_backend_service" "my-backend-service" {
    name = "my-backend-service"
    description = "xxxx"
    port_name = "http"
    protocol = "HTTP"
    backend {
      group = "${element(google_container_cluster.my_cluster.instance_group_urls, 1)}"
    }
    health_checks = ["${google_compute_http_health_check.my-http-basic-check.self_link}"]
}

I don't know why this kind of error occurs and how to fix it.

What is the reason of this error and how can I fix it?

Thanks.

@phinze

This comment has been minimized.

Member

phinze commented Dec 16, 2015

I'm not familiar offhand with this API, but perhaps @lwander can shed some light on this?

@lwander

This comment has been minimized.

Contributor

lwander commented Dec 17, 2015

@yosssi, this looks like a bug with the Google API. The instance_group_url field should be exporting the underlying instance groups' URLs, instead it seems to providing the URLs of the instance group managers, which is why this call is failing. I will file a bug internally to address this.

@yosssi

This comment has been minimized.

yosssi commented Dec 17, 2015

@lwander Thanks for your investigation. I understand. I hope this bug will be fixed.

@lwander lwander added the upstream label Jan 20, 2016

@dcgoss

This comment has been minimized.

Contributor

dcgoss commented Mar 12, 2016

@lwander any updates on this?

@lwander

This comment has been minimized.

Contributor

lwander commented Mar 12, 2016

No, nothing yet. I'm hesitant to fix this on the Terraform end, since it will only break again once the API is fixed.

@dcgoss

This comment has been minimized.

Contributor

dcgoss commented Mar 12, 2016

@lwander Makes sense, thanks for the quick response!
@yosssi The solution/hack below may help you, it worked for me.
For the time being, I just used the instance group API docs to hardcode in the value.
If you inspect the .tfstate file, you will see in the container cluster section attributes that look like this:

"instance_group_urls.#": "1",
"instance_group_urls.0": "https://www.googleapis.com/replicapool/v1beta2/projects/your_project_name-here/zones/your_zone_here/instanceGroupManagers/gke-your_cluster_name_here-hash_key_here-group",

If you take that last part of the url: gke-your_cluster_name_here-hash_key_here-group, you can put it into the correct API url form specified in the instance group API docs:

I had to put this in parentheses so the link wouldn't highlight -
(https://www.) googleapis.com/compute/v1/projects/project/zones/your_zone_here/instanceGroups/gke-your_cluster_name_here-your_hash_key_here-group

Then you can hardcode in the group value:

backend {
        group = "https://www.googleapis.com/compute/v1/projects/exampleproject/zones/us-central1-b/instanceGroups/gke-examplecluster-hdja738s-group"
        balancing_mode = "UTILIZATION"
        max_utilization = 0.8
    }
@mihirat

This comment has been minimized.

mihirat commented Oct 21, 2016

@dcgoss
Although its just a workaround, but I've got a simple solution.
The instance_group_url returns
https://container.googleapis.com/v1/projects/myproject-dev/zones/asia-east1-a/instanceGroupManagers/gke-myproject-blahblah-group
and what we want is
https://container.googleapis.com/v1/projects/myproject-dev/zones/asia-east1-a/instanceGroups/gke-myproject-blahblah-group

Thus, just replacing "instanceGroupManagers" with "instanceGroups", it works.
In this workaround,

    backend {
      group = "${replace(element(google_container_cluster.my_cluster.instance_group_urls, 1), "Manager", "")}"
    }

That's it.

@paddycarver

This comment has been minimized.

Member

paddycarver commented Dec 12, 2016

@lwander would a possible way forward on this issue be to examine the URL we're getting back from the API, attempt to parse it according to the known format (https://container.googleapis.com/v1/projects/myproject-dev/zones/asia-east1-a/instanceGroupManagers/gke-myproject-blahblah-group) and see if it uses instanceGroupManagers? If it does, then we change it to use instanceGroups. If it fails to parse, or doesn't use instanceGroupManagers, we don't do any special processing. That way it should fix things for now, so people don't have to implement workarounds in their config files, and when the issue is fixed properly, the workaround won't break anything. What do you think?

@lwander

This comment has been minimized.

Contributor

lwander commented Dec 12, 2016

Hey @paddyforan, rather than do string substitution on matching URLs it would be neater to lookup the returned instance group manager to read which instance group it's managing, and then supply that. Depending on naming conventions feels brittle.

@paddycarver

This comment has been minimized.

Member

paddycarver commented Jan 3, 2017

Hey @lwander, you're right, that does feel more robust, though it comes at the cost of an extra network request. In this case, I feel like the network request is worthwhile, even though we still need to rely on naming conventions to detect that the API has not been fixed, unless I'm missing something.

I'll try to get a test case for this that fails, then get it fixed. Can't promise an ETA on the PR (I'm still in catch-up mode after the holiday) but hopefully I'll get to this soon. If anyone beats me to the punch, I'm happy to review.

@paddycarver paddycarver self-assigned this Jan 3, 2017

@lwander

This comment has been minimized.

Contributor

lwander commented Jan 4, 2017

Sounds great, thanks @paddyforan

paddycarver added a commit that referenced this issue Mar 7, 2017

provider/google: fix container instance group URLs
Google Container Engine's cluster API returned instance group manager
URLs when it meant to return instance group URLs. See #4336 for details
about the bug.

While this is undeniably an upstream problem, this PR:

* detects the error, meaning it will work as expected when the API is
  fixed.
* corrects the error by requesting the instance group manager, then
  retrieving its instance group URL, and using that instead.
* adds a test that exercises the error and the solution, to ensure it is
  functioning properly.
@paddycarver

This comment has been minimized.

Member

paddycarver commented Mar 7, 2017

I have, at long last, opened a PR with the change. Apologies for the delay. I tagged you for a review, @lwander, as you seemed the most knowledgeable about it, but if you don't have time, there are others that can review. :)

@paddycarver

This comment has been minimized.

Member

paddycarver commented Mar 13, 2017

PR just landed in master. This should be fixed in our next release. Thanks for the patience, and for reporting!

@dinvlad

This comment has been minimized.

dinvlad commented Jul 9, 2018

@paddycarver have you by any chance asked Google folks to solve this issue upstream? would be curious to join in on the conversation

@paddycarver

This comment has been minimized.

Member

paddycarver commented Jul 11, 2018

I, unfortunately, don't quite recall--it's not showing up in my https://issuetracker.google.com history, which means if I did, I did so through an informal back-channel and don't have a link, anyways. :( If you're interested in it, I'd definitely suggest opening an issue on https://issuetracker.google.com!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment