Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature Request: Custom Resource #215

Closed
timthesinner opened this issue Nov 14, 2018 · 88 comments
Closed

Feature Request: Custom Resource #215

timthesinner opened this issue Nov 14, 2018 · 88 comments

Comments

@timthesinner
Copy link

@timthesinner timthesinner commented Nov 14, 2018

Enabling the K8S provider to apply and manage the lifecycle for custom resources has a number of advantages:

  • Allow terraform to bring an entire K8S cluster under management, without waiting for formal resource models.
  • Allow terraform to manage resources for custom APIs.

Terraform Configuration Files

Inline Configuration

resource "kubernetes_custom" "cluster-issuer" {
  apiVersion = "certmanager.k8s.io/v1alpha1"
  kind       = "ClusterIssuer"

  metadata {
    name = "lets-encrypt-prod-issuer"
  }

  set {
    name = "spec"

    value = {
      acme = {
        email  = "admin@example.com"
        server = "https://acme-v02.api.letsencrypt.org/directory"

        privateKeySecretRef {
          name = "lets-encrypt"
        }

        http01 {}
      }
    }
  }
}

Inline configuration

locals {
  wildcard-cert = <<WILDCARD_CERT
apiVersion: certmanager.k8s.io/v1alpha1
kind: Certificate
spec:
 secretName: ingress-wildcard-tls
 issuerRef:
   name: '${kubernetes_custom.cluster-issuer.metadata.name}'
   kind: ClusterIssuer
 dnsNames:
   - '*.example.com'
 acme:
   config:
     - http01:
       domains:
         - '*.example.com'
WILDCARD_CERT
}

resource "kubernetes_custom" "wildcard-ingress-certificate" {
  metadata {
    name      = "ingress-tls-wildcard"
    namespace = "kube-system"
  }

  values = "${local.wildcard-cert}"
}

File based configuration

resource "kubernetes_custom" "default-ingress-certificate" {
  metadata {
    name      = "default-ingress-tls-wildcard"
    namespace = "kube-system"
  }

  values = [
    "${file("default-certificate.yaml")}",
  ]
}

Expected Behavior

Define and manage the lifecycle of any K8S resources using the kubernetes provider.

Actual Behavior

This resource type is not supported

References

Open feature requests for formal models

@alexsomesan
Copy link
Contributor

@alexsomesan alexsomesan commented Nov 14, 2018

Thanks for documenting this request.

We do want to support custom resources (and generic resources in general). We've been discussing various ways to approach this for a while. The main challenge with the current state of things in Terraform is achieving a diff-ing behaviour that is consistent with current Terraform resource diffs. The plan is to wait for Terraform 0.12 to land first and try to make use of some of its upcoming enhancements in implementing such a resource.

@timthesinner
Copy link
Author

@timthesinner timthesinner commented Nov 14, 2018

Sounds great, thanks for the quick response.

@bryanlarsen
Copy link

@bryanlarsen bryanlarsen commented Dec 14, 2018

The most obvious workaround is https://github.com/ericchiang/terraform-provider-k8s

But what I'm doing is creating a simple helm chart with my custom resources and using the helm provider, providing a local path as the chart name. That makes sense for us because we're using the helm provider to install a bunch of other stuff too.

@nfisher
Copy link

@nfisher nfisher commented Jan 16, 2019

Hi @alexsomesan given HCL is JSON compatible what are the big obstacles you see to adopting it prior to 0.12 landing? Or is it more a matter of prioritisation and bandwidth?

@pdecat
Copy link
Contributor

@pdecat pdecat commented Jan 16, 2019

This was also discussed in #195

@bcornils bcornils added this to the Research milestone Feb 7, 2019
timthesinner added a commit to timthesinner/terraform-provider-kubernetes that referenced this issue Feb 18, 2019
Notes:
 - Created an ExtendedClientset allowing generic access to K8S REST client
 - Refactored all usage of the provider to the K8S interfaces
 - Added a dependency on "github.com/ghodss/yaml"
 - Handle create, update, delete for custom resources
 - Custom resource update state delta between K8S, TF, and Desired is handled through transient fields
@benishak
Copy link

@benishak benishak commented Mar 15, 2019

we at mobfox a year ago wrote a provider that allows the user to create Custom Resources and even despites the needs to create Custom Provider.

Now we are open sourcing it so Check it out here:
https://github.com/mobfox/terraform-provider-multiverse

you can even use AWS Lambda or execute a function locally in any language you like to manage your resources, it also keep state of your resource, so you can delete, read, update them too. It creates a resource, so it is not like External Data and behaves exactly like Custom Resources in AWS CloudFormation

@awfm9
Copy link

@awfm9 awfm9 commented May 26, 2019

This is going to be needed to use the upcoming v2 release of traefik as ingress, as they are using custom resources to improve configuration: https://docs.traefik.io/v2.0/providers/kubernetes-crd/

@aelbarkani
Copy link

@aelbarkani aelbarkani commented May 27, 2019

@alexsomesan now that 0.12 has landed, are there any plans to implement this feature ? it seems CRDs is the way to go according to many Kubernetes experts, and Redhat is investing a lot on it through Openshift operators. so it seems to me that it will become very soon a must have feature for Terraform I think

@kalebwalton
Copy link

@kalebwalton kalebwalton commented May 31, 2019

Agree. Operators for 'day 2 operations' are becoming a big thing in IBM / RedHat. We're using the 'helm_resource' right now, but soon we're going to want to use an Operator CRD to manage our applications. It'll be a shame if we have to patch it into our orchestration outside of Terraform.

@davisford
Copy link

@davisford davisford commented Jun 2, 2019

I wish the official Terraform k8s provider supported things like CRD, Jobs, and other smaller tasks that currently are unsupported, but my guess is it won't come any time real soon. I've had some luck using the nice community provider that deals with raw yaml.

The original author indicated he does not have time to push it forward any farther, so I forked it and updated it for Terraform 0.12. It is working very nicely for us. We use the Couch infrastructure which does use an operator that pushes its own CRD. So far so good. The current state of the provider needs some work with respect to diffs and patches, but it does what we need it to do.

Fork is here that updates for Go modules and TF@0.12

@gabzim
Copy link

@gabzim gabzim commented Jun 15, 2019

Is there an estimate ETA for this feature? As more deployments require CRDs the number of things we can deploy to k8 with terraform grows smaller and smaller (I'm looking at you cert-manager)

@alexsomesan
Copy link
Contributor

@alexsomesan alexsomesan commented Jun 15, 2019

@curtbushko
Copy link

@curtbushko curtbushko commented Jul 5, 2019

Adding to use cases for a simple CRD option. Basically, we only care if it just passes everything onto the kubernetes API and saves the state.

We are using the istio ingressgateway and to do so you need the VirtualService kind.

I would really like to use the kubernetes provider for setting up our services instead of helm.

@arnonki
Copy link

@arnonki arnonki commented Aug 4, 2019

This is a must feature it prevents us from using terraform in production

@benishak
Copy link

@benishak benishak commented Aug 4, 2019

@pescetto
Copy link

@pescetto pescetto commented Aug 14, 2019

@alexsomesan I know it's only been a couple months since your last update, but any chance this can get added this year? I feel like CDR's are the biggest missing feature from the kubernetes terraform provider. Thanks again and I'm really looking forward to CDR's in the provider!

@zdevi
Copy link

@zdevi zdevi commented Sep 26, 2019

Hi, any news on this ? Did they abandon implementing CRD's ??? This is a vital feature for me, CRD's are used to configure backends on GCP... Please let us know what is happening, and if there is an ETA, i see it has been more than a year people are asking for that feature.

@alexsomesan
Copy link
Contributor

@alexsomesan alexsomesan commented Sep 26, 2019

We have not abandoned CRD support.
However, it's a radical change that requires intervention into multiple layers of the provider SDKs (and overriding some of them). It going to be released as a separate experimental provider in Alpha state when we have it working enough for testing. We hope to be able to demo it after the end of Oct.

@zdevi
Copy link

@zdevi zdevi commented Sep 26, 2019

So we are talking about Years just for this feature ? I can see it has already been discussed in early 2018. It is becoming hard to continue using kubernetes provider in production as it is lacking a lot, no custom configuration possible without CRD's.... I am gona be forced to find another solution, else than the official provider, that's sad.

@tombh
Copy link

@tombh tombh commented Sep 26, 2019

I hope this isn't off topic, but what do people consider as alternative options? I'm thinking more of the declarative approaches like Cloudformation and maybe Pulumi?

I haven't really researched much recently but it still seems to me that Terraform is the best we have, even without CRD support. Is there something inherent about Terraform's design, or declarative Infrastructure As Code frameworks in general that makes it inextricably difficult to maintain feature parity with Kubernetes?

I believe it when I hear that implementing CRD support is a significant task. But I'm happy to wait and very grateful for the efforts being made because Terraform still seems like the best approach out there. Sure I have a few YAML files here and there that I need to manually manage but it's tiny compared to the swathes of HCL. It boggles my mind every time I see Kubernetes tutorials referencing YAML snippets, how can people be productive like that!?

@benabineri
Copy link

@benabineri benabineri commented Sep 26, 2019

I believe (although I may be wrong) that Pulumi uses the Terraform providers underneath so you'd probably have the same issue there.

@stuart-c
Copy link

@stuart-c stuart-c commented Sep 26, 2019

We use Helm charts via the Helm Terraform provider, with small pieces handled directly through this provider where it doesn't make sense for a Helm chart.

@BarnumD
Copy link

@BarnumD BarnumD commented May 29, 2020

@drebes That looks great. We're using terraform cloud for our deploys. Is there a way to pull in a provider like that in the cloud that's not from the public repository?

@Sytten
Copy link

@Sytten Sytten commented May 29, 2020

@BarnumD You have to commit it in the repo in .terraform.d/plugins. See https://www.terraform.io/docs/plugins/basics.html.

@BarnumD
Copy link

@BarnumD BarnumD commented May 29, 2020

@Sytten Thanks. I had to put it in terraform.d/plugins (without the dot). I also had to run git update-index --chmod=+x .\terraform.d\plugins\linux_amd64\terraform-provider-kubernetes-alpha otherwise terraform cloud would complain about permissions. Didn't see that in the link you provided.

@alexsomesan
Copy link
Contributor

@alexsomesan alexsomesan commented Jun 3, 2020

Support for CRD is being rolled out in the new alpha provider. You can try out the alpha provider here: https://github.com/hashicorp/terraform-provider-kubernetes-alpha

Once it's been graduated out of alpha state, it's codebase will be merged into this provider and the features will become available here as additional resources.

@Sytten
Copy link

@Sytten Sytten commented Jun 3, 2020

@alexsomesan Is there a timeline/roadmap for that to happen?

@nadworny
Copy link

@nadworny nadworny commented Aug 20, 2020

Do you happen to know more about the roadmap?

@0dragosh
Copy link

@0dragosh 0dragosh commented Aug 20, 2020

Also interested in a roadmap so I know what provider to choose. HashiCorp is really lagging behind on this.

@mcfedr
Copy link

@mcfedr mcfedr commented Aug 21, 2020

I've found this provider to do a really great job with any kubernetes resources this one cannot handle, https://registry.terraform.io/providers/banzaicloud/k8s/latest/docs/resources/manifest - and now its really easy to use with the registry.

@nadworny
Copy link

@nadworny nadworny commented Aug 21, 2020

thanks @mcfedr , I'll check it out

@jurgenweber
Copy link

@jurgenweber jurgenweber commented Aug 21, 2020

if you weren't aware, I think they are deprecating this provider in favour of: https://github.com/hashicorp/terraform-provider-kubernetes-alpha

@aareet
Copy link
Member

@aareet aareet commented Aug 21, 2020

Hi folks! We're currently evaluating the feasibility of merging the manifest resource into our official provider. We are also working on some SDK improvements that are required to support the alpha provider. We'll keep our roadmap up to date with timelines and changes as soon we have them.

@pawelprazak
Copy link

@pawelprazak pawelprazak commented Oct 2, 2020

Any news on the roadmap?

CRDs are GA since Kubernetes 1.16 and almost everyone needs them now that every cloud provider and project use some kind of operator.

Btw. the plugin is available in the registry, I was able to install it on tf 0.13

terraform {
  required_providers {
    kubernetes-alpha = {
      source = "hashicorp/kubernetes-alpha"
      version = "0.2.1"
    }
  }
}

provider "kubernetes-alpha" {
  # Configuration options
}

https://registry.terraform.io/providers/hashicorp/kubernetes-alpha/latest/docs/resources/kubernetes_manifest

@viceice
Copy link

@viceice viceice commented Dec 8, 2020

Also a workaround for deploying simple yaml manifests: https://registry.terraform.io/providers/gavinbunney/kubectl/latest

@vdahmane
Copy link

@vdahmane vdahmane commented Jan 13, 2021

Hello, do you have ETA for this useful feature?

@jurgenweber
Copy link

@jurgenweber jurgenweber commented Jan 13, 2021

yeah, in the end we abstracted away from TF deploying directly....

We have TF deploying ArgoCD apps via helm charts....

ArgoCD does the rest.

@yashbhutwala
Copy link

@yashbhutwala yashbhutwala commented Jan 19, 2021

An update on the plans on the roadmap for this feature would help assess if/whether we should use kubernetes-alpha provider or not.

Info and links from @aareet's comment from last year has not been kept up to date, i.e.: https://github.com/hashicorp/terraform-provider-kubernetes/blob/master/_about/ROADMAP.md

Hi folks! We're currently evaluating the feasibility of merging the manifest resource into our official provider. We are also working on some SDK improvements that are required to support the alpha provider. We'll keep our roadmap up to date with timelines and changes as soon we have them.

@aareet aareet added the hold label Jan 25, 2021
@aareet
Copy link
Member

@aareet aareet commented Jan 25, 2021

Hi, the SDK improvements have now shipped and we have been working on integrating them into the alpha provider. As before, we continue to work to find a way to bring the alpha provider to a production-capable state.

@alexsomesan
Copy link
Contributor

@alexsomesan alexsomesan commented Mar 17, 2021

Quick update here: the recently released v0.3.1 of the Kubernetes Alpha provider has quite usable support for CRDs. I would encourage everyone to give that a try a provide us with feedback. That would help us to graduate that provider to GA as soon as possible.

@nomius
Copy link

@nomius nomius commented May 13, 2021

Hi folks, any updates on moving this functionality to this (stable channel) provider?

Thanks!

@nikolay
Copy link

@nikolay nikolay commented May 14, 2021

@alexsomesan "Usable" and "alpha" can't be used in one sentence!

@asaphe
Copy link

@asaphe asaphe commented May 30, 2021

Hi! I love the Kubernetes Alpha provider and I'm using it here https://github.com/asaphe/terraform-keda to deploy a KEDA crd.

Would love to be able to use dynamic blocks for manifest.

When will the Kubernetes Alpha provider be merged with the Kubernetes provider? I would love to already start using it and plan ahead.

@jrhouston
Copy link
Contributor

@jrhouston jrhouston commented Aug 3, 2021

The kubernetes_manifest resource from the kubernetes-alpha project has now been merged into this provider as a beta feature under an experiment flag. Here is an example of how to use it:

provider "kubernetes" {
  experiments {
    manifest_resource = true
  }
  config_path = "~/.kube/config"
}

resource "kubernetes_manifest" "example" {
  manifest = {
    apiVersion = "v1"
    kind = "ConfigMap"
    metadata = {
      name = "example"
    }
    data = {
      provider = "Terraform"
    }
  }
}

I'm going to close this issue now, as we would like to address any issues with the resource separately. Please open a new issue on this repo if you encounter any problems.

@jrhouston jrhouston closed this Aug 3, 2021
@github-actions
Copy link

@github-actions github-actions bot commented Sep 3, 2021

I'm going to lock this issue because it has been closed for 30 days . This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Sep 3, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet