Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add importer for helm release resource #150

Closed
wants to merge 8 commits into from

Conversation

Lemmons
Copy link

@Lemmons Lemmons commented Nov 8, 2018

This pr adds the ability to import helm release resources.

It takes the release name and repository as the import id and currently is using --- as the delineator between the two. I chose --- simply because it provides good separation between the two pieces of the import id and is not very likely to be a part of either. It is, admittedly, quite naive to assume nobody is going to use --- in their release or repository names, so if anyone has a better idea there it would be appreciated. Another solution would be to use a single character, such as - and then escape any -s in either part of the id, but this would mean adding a lot more logic to the import id parsing, and would make it much more confusing to import release with a - in them.

It then gets the values out of the release and stores them as sets in state. This means that if you define your release in terraform using values, or a combination of values and sets, your first plan will show an in-place modification of the release, though likely it will result in no changes in helm. I went back and forth on which one to use, set or values, but in the end, either will lead to the same issue, because, as far as I can tell, Helm stores everything as a combined list of values, which means there is no way to distinguish between the two once a release is out there.

I'm expecting a bit of feedback, but at least wanted to get this out there so we can start a discussion on the best way to implement this.

@ghost ghost added the size/XL label Nov 8, 2018
@Lemmons
Copy link
Author

Lemmons commented Nov 8, 2018

Related to #113

@meyskens
Copy link
Contributor

Didn’t find the time yet to review the code. On the use of --- could I suggest using the .? The dot is illegal in kubernetes resource names thus also for releases. It also feels more natural in terraform since it uses the dot when calling sub-resources in the cli.

@Lemmons
Copy link
Author

Lemmons commented Nov 12, 2018

Oh good idea. A repository can have .s in its name, as a local path is a valid repository, but that is fine as long as the release can't, as the first . would be the delineator. That being said, it will look a little weird in some cases. For example my-resource.../../../my-charts/stable, but I think that isn't too bad. I'll make this change when I have a chance.

@meyskens
Copy link
Contributor

How about doing repo.release? Then use the last dot?

@Lemmons
Copy link
Author

Lemmons commented Nov 14, 2018

I updated the pr to have the suggested changes: using . as the delimiter and swapping the order. So now my example import looks like ../../../my-charts/stable.my-resource

@legal90
Copy link
Contributor

legal90 commented Nov 20, 2018

I verified that PR and the importer works fine. Thank you @Lemmons !
The only caveat I found is that repository and chart properties are parsed from the import command as is, so for example if we use this command:

terraform import helm_release.my_mariadb stable.mariadb

then the resource should be defined in the terraform code like this:

# Works OK
resource "helm_release" "my_mariadb" {
  name       = "my_mariadb"
  chart      = "mariadb"
  repository = "stable"
  namespace  = "test_namespace"
  # <...>
}

But it will not work properly if you put repository name as part of the chart name (as recommended on the README.md), e.q.:

# Doesn't work OK
resource "helm_release" "my_mariadb" {
  name      = "my_mariadb"
  chart     = "stable/mariadb"
  namespace = "test_namespace"
  # <...>
}

In the second case after the import, terraform plan will show changes of both repository and chart fields, forcing the resource recreation.
Though, I'm not sure is it an issue of the Importer, or that needs to be just better explained in the entire resource documentation?

@legal90
Copy link
Contributor

legal90 commented Nov 26, 2018

⚠️ Unfortunately, the plugin crashes when you try to import the release with values having an integer there. Example:

  1. Deploy a test release with an integer value:
printf "revisionHistoryLimit: 10\n" > input_values.yaml
# terraform.tf

resource "helm_repository" "stable" {
  name = "stable"
  url  = "https://kubernetes-charts.storage.googleapis.com"
}

resource "helm_release" "test-nginx-ingress" {
  namespace = "test"
  name      = "test-nginx-ingress"

  repository = "stable"
  chart      = "nginx-ingress"
  version    = "0.28.1"

  values = ["${file("${path.module}/input_values.yaml")}"]
}
terraform apply
  1. It is installed successfully. Let's remove this resource from the state and try to import it:
terraform state rm helm_release.test-nginx-ingress
terraform import helm_release.test-nginx-ingress stable.test-nginx-ingress

Result

helm_release.test-nginx-ingress: Importing from ID "volvocars.test-nginx-ingress"...

Error: helm_release.test-nginx-ingress (import id: volvocars.test-nginx-ingress): import helm_release.test-nginx-ingress (id: volvocars.test-nginx-ingress): unexpected EOF


panic: Unknown: %!!(MISSING)s(float64=10)
2018-11-26T17:21:36.950+0100 [DEBUG] plugin.terraform-provider-helm:
2018-11-26T17:21:36.950+0100 [DEBUG] plugin.terraform-provider-helm: goroutine 28 [running]:
2018-11-26T17:21:36.950+0100 [DEBUG] plugin.terraform-provider-helm: github.com/terraform-providers/terraform-provider-helm/vendor/github.com/hashicorp/terraform/flatmap.flatten(0xc000b6c300, 0xc00090e280, 0x14, 0x23020c0, 0xc000041358, 0x8e)
2018-11-26T17:21:36.950+0100 [DEBUG] plugin.terraform-provider-helm: 	/Users/legal/go/src/github.com/terraform-providers/terraform-provider-helm/vendor/github.com/hashicorp/terraform/flatmap/flatten.go:46 +0x48d
2018-11-26T17:21:36.951+0100 [DEBUG] plugin.terraform-provider-helm: github.com/terraform-providers/terraform-provider-helm/vendor/github.com/hashicorp/terraform/flatmap.Flatten(0xc000877dd0, 0xa8f)
2018-11-26T17:21:36.951+0100 [DEBUG] plugin.terraform-provider-helm: 	/Users/legal/go/src/github.com/terraform-providers/terraform-provider-helm/vendor/github.com/hashicorp/terraform/flatmap/flatten.go:19 +0xb4
2018-11-26T17:21:36.951+0100 [DEBUG] plugin.terraform-provider-helm: github.com/terraform-providers/terraform-provider-helm/helm.resourceReleaseImportState(0xc0003601c0, 0x254a160, 0xc0001842d0, 0xc, 0xc00082d818, 0x1, 0x0, 0xc00097a630)
2018-11-26T17:21:36.951+0100 [DEBUG] plugin.terraform-provider-helm: 	/Users/legal/go/src/github.com/terraform-providers/terraform-provider-helm/helm/resource_release.go:470 +0x470
2018-11-26T17:21:36.951+0100 [DEBUG] plugin.terraform-provider-helm: github.com/terraform-providers/terraform-provider-helm/vendor/github.com/hashicorp/terraform/helper/schema.(*Provider).ImportState(0xc00036cc40, 0xc0001840a0, 0xc000900240, 0x1c, 0x28, 0x28, 0xc00097a630, 0x0, 0xc0000b0400)
2018-11-26T17:21:36.951+0100 [DEBUG] plugin.terraform-provider-helm: 	/Users/legal/go/src/github.com/terraform-providers/terraform-provider-helm/vendor/github.com/hashicorp/terraform/helper/schema/provider.go:365 +0x2c4
2018-11-26T17:21:36.951+0100 [DEBUG] plugin.terraform-provider-helm: github.com/terraform-providers/terraform-provider-helm/vendor/github.com/hashicorp/terraform/plugin.(*ResourceProviderServer).ImportState(0xc000927560, 0xc000926040, 0xc000926200, 0x0, 0x0)
2018-11-26T17:21:36.951+0100 [DEBUG] plugin.terraform-provider-helm: 	/Users/legal/go/src/github.com/terraform-providers/terraform-provider-helm/vendor/github.com/hashicorp/terraform/plugin/resource_provider.go:560 +0x57
2018-11-26T17:21:36.951+0100 [DEBUG] plugin.terraform-provider-helm: reflect.Value.call(0xc0001737a0, 0xc00016f5b8, 0x13, 0x26208d4, 0x4, 0xc000081f18, 0x3, 0x3, 0xc00058b7c0, 0xc000000300, ...)
2018-11-26T17:21:36.951+0100 [DEBUG] plugin.terraform-provider-helm: 	/usr/local/Cellar/go/1.11.1/libexec/src/reflect/value.go:447 +0x449
2018-11-26T17:21:36.951+0100 [DEBUG] plugin.terraform-provider-helm: reflect.Value.Call(0xc0001737a0, 0xc00016f5b8, 0x13, 0xc000162f18, 0x3, 0x3, 0x3, 0x0, 0xc000172120)
2018-11-26T17:21:36.951+0100 [DEBUG] plugin.terraform-provider-helm: 	/usr/local/Cellar/go/1.11.1/libexec/src/reflect/value.go:308 +0xa4
2018-11-26T17:21:36.951+0100 [DEBUG] plugin.terraform-provider-helm: net/rpc.(*service).call(0xc00058bd80, 0xc000586690, 0xc000969478, 0xc0009694a0, 0xc00087b300, 0xc000927b80, 0x22c1860, 0xc000926040, 0x16, 0x22c18a0, ...)
2018-11-26T17:21:36.951+0100 [DEBUG] plugin.terraform-provider-helm: 	/usr/local/Cellar/go/1.11.1/libexec/src/net/rpc/server.go:384 +0x14e
2018-11-26T17:21:36.951+0100 [DEBUG] plugin.terraform-provider-helm: created by net/rpc.(*Server).ServeCodec
2018-11-26T17:21:36.951+0100 [DEBUG] plugin.terraform-provider-helm: 	/usr/local/Cellar/go/1.11.1/libexec/src/net/rpc/server.go:481 +0x47e
2018/11/26 17:21:36 [ERROR] root: eval: *terraform.EvalImportState, err: import helm_release.test-nginx-ingress (id: volvocars.test-nginx-ingress): unexpected EOF
2018/11/26 17:21:36 [ERROR] root: eval: *terraform.EvalSequence, err: import helm_release.test-nginx-ingress (id: volvocars.test-nginx-ingress): unexpected EOF
2018/11/26 17:21:36 [TRACE] [walkImport] Exiting eval tree: helm_release.test-nginx-ingress (import id: volvocars.test-nginx-ingress)
2018/11/26 17:21:36 [TRACE] dag/walk: upstream errored, not walking "provider.helm (close)"
2018/11/26 17:21:36 [DEBUG] plugin: waiting for all plugin processes to complete...
2018-11-26T17:21:36.954+0100 [DEBUG] plugin: plugin process exited: path=/Users/legal/.terraform.d/plugins/terraform-provider-helm
2018-11-26T17:21:36.954+0100 [WARN ] plugin: error closing client during Kill: err="connection is shut down"



!!!!!!!!!!!!!!!!!!!!!!!!!!! TERRAFORM CRASH !!!!!!!!!!!!!!!!!!!!!!!!!!!!

Terraform crashed! This is always indicative of a bug within Terraform.
A crash log has been placed at "crash.log" relative to your current
working directory. It would be immensely helpful if you could please
report the crash with Terraform[1] so that we can fix this.

When reporting bugs, please include your terraform version. That
information is available on the first line of crash.log. You can also
get it by running 'terraform --version' on the command line.

[1]: https://github.com/hashicorp/terraform/issues

!!!!!!!!!!!!!!!!!!!!!!!!!!! TERRAFORM CRASH !!!!!!!!!!!!!!!!!!!!!!!!!!!!

Reason

It seems that yaml.Unmarshal returns transforms YAML to the object containing float64, which is not supported by flatmap.Flatten:
https://github.com/hashicorp/terraform/blob/3b49028b77b083f355166f1315f1e021d06d7f78/flatmap/flatten.go#L46

@Lemmons
Copy link
Author

Lemmons commented Nov 27, 2018

Dang. Good catch.

I'll see if I can carve out some time to fix this in the next week or so.

@Lemmons
Copy link
Author

Lemmons commented Nov 27, 2018

Also, regarding your previous comment, @legal90, when I'm working on the fix for the type issue, I'll see if I can find a way to support including the repo in the chart name. I started off doing it that way, and decided against it at some point, but I don't clearly remember why now. I think there might be a way to support it, but it also might require a fairly significant refactor of other logic in the provider not directly related to the import. If that is the case, I will hold off on those changes and do them as a subsequent pr to keep the complexity of this pr as low as possible.

@legal90
Copy link
Contributor

legal90 commented Nov 27, 2018

@Lemmons Thank you! Yeah, the first thing is just a limitation and could be solved separately (I guess so).

But I also found that the importer doesn't respect existing values and set_string attributes if those are set in the configuration. It just dumps all values to set key-value pairs, which causes the undesired diff in terraform plan right after the successful import.

I thought about this today and came up with a PoC solution (which actually also fixes the crash mentioned above). It's in the commit of the fork of your branch:
legal90@982f221 (UPDATED)

It adds a new attribute overrides which is a computed value get merging values, set and set_string attributes via getValues() helper. So that, the resource state becomes absolutely agnostic to the way how these values are defined.
I'm still not sure that it is a right way to do that. Please let me know what you think about it.

Maybe we can reuse metadata.values for that purpose, instead of creating a new "fake" attribute? Suggestions are welcome!

The new "overrides" attribute saved the compiled version of all "values" + "set" + "set_string"
values merged. It is assumed that "overrides" will be the only attribute showing the diff
of these values. Diffs of "values", "set" and "set_string" attributes are suppressed.
Update the release only if any of values, chart version or reuse_values are changed,
OR any of recreate_pods or force_update are set to true
@legal90
Copy link
Contributor

legal90 commented Dec 12, 2018

Pinging @Lemmons @meyskens
Is anyone interested in changes I offered above? As well as others I did in my branch Lemmons/terraform-provider-helm@add-importer...legal90:add-importer ?

I believe that will help a lot with an existing workflow of this provider and will improve the user experience in terms of values definition. Let me know if I should send it as an additional pull-request. Cheers! ;)

Add overrides to support better diff logic
@ghost ghost added the documentation label Dec 17, 2018
@Lemmons
Copy link
Author

Lemmons commented Dec 17, 2018

@legal90 Thanks for the work and the ping. I definitely think this is the correct way to go. I've merged your changes into this pr.

@Lemmons
Copy link
Author

Lemmons commented Dec 19, 2018

@meyskens any thoughts about the path to get this merged? I will deal with the merge conflicts in the next few days, but beyond that, what else should we do before we can move forward on this?

@legal90
Copy link
Contributor

legal90 commented Jan 11, 2019

Since #153 just got merged, we might need to wrap the newly introduced set_sensitive attribute to the overrides, like we did that with set, set_string and values here.

Since that part (with overrides and stuff) in not directly related to the importer feature, I can create a separate PR for that.

@legal90
Copy link
Contributor

legal90 commented Jan 16, 2019

I moved the implementation of "overrides" attribute to the separated PR (with some changes): #184

Hopefully, it will make both PRs easier to review & merge.

@embik
Copy link

embik commented Apr 5, 2019

Looks like this PR is stale now. Is there any plan to move forward here?

@selslack
Copy link

Is there any update on this PR?

@caarlos0
Copy link

can I help with this in any way?

@Lemmons
Copy link
Author

Lemmons commented Sep 14, 2019

Unfortunately I haven't had time to pick this back up. Closing this in favor of #337

@Lemmons Lemmons closed this Sep 14, 2019
@ghost ghost locked and limited conversation to collaborators Apr 27, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

6 participants