Support Kubernetes Provider #29

osdemah · 2019-05-06T02:14:46Z

Implmentation of #3

Note: I add kubernetes provider. resource types are dynamically being generated by matching list of available resources in terraform provider and existing resource types in kubernetes api server.

TODO: Add support of CRDS.

ISSUE:

Some certain kubernetes resources are being failed to applied by terraform, e.g. A configmap label is changed to eks "amazonaws" {"com/component" = "coredns"} from eks.amazonaws.com/component" = "coredns". (terraform k8s provider is converting "." to some wiered map!).

I didn't check to see if those issues are fixed in the newer versions or not, but we can get benefits of addressed issues by making terraformer compatible to newer versions of terraform and updating the k8s provider to v1.6.2.

sergeylanzman

Good Job!!!
2 comments:

Can you separate code to 2 commits(your code and vendor code)
Add to Readme about kubernetes support

cmd/kubernetes.go

sergeylanzman · 2019-05-06T08:40:59Z

kubernetes_terraforming/kind.go

+	for i := 0; i < items.Len(); i++ {
+		item := items.Index(i)
+		// Filter to resources that aren't owned by any other resource
+		if item.FieldByName("OwnerReferences").Len() > 0 {


sergeylanzman · 2019-05-06T08:58:34Z

go.mod

+	k8s.io/apimachinery v0.0.0-20190116203031-d49e237a2683
+	k8s.io/client-go v7.0.0+incompatible
+	k8s.io/kubectl v0.0.0-20190502165022-ce8d9f55c93c
+	labix.org/v2/mgo v0.0.0-20140701140051-000000000287 // indirect


Why we need mongo driver?

Apperntally both launchpad.net/gocheck and labix.org/v2/mgo are added by mistake. I removed them.

sergeylanzman · 2019-05-06T10:27:48Z

Testing:

Error on go mod vendor. Why we need mgo driver?

terraformer git:(support_kubernetes_provider) GO111MODULE=on go mod vendor
go: finding github.com/json-iterator/go v1.1.6
go: finding github.com/iancoleman/strcase v0.0.0-20190422225806-e506e3ef7365
go: finding github.com/terraform-providers/terraform-provider-google v1.20.0
go: finding github.com/terraform-providers/terraform-provider-kubernetes v1.4.0
go: finding github.com/imdario/mergo v0.3.7
go: finding github.com/dustinkirkland/golang-petname v0.0.0-20170921220637-d3c2ba80e75e
go: finding github.com/peterbourgon/diskv v2.0.1+incompatible
go: finding k8s.io/client-go v7.0.0+incompatible
go: finding gopkg.in/inf.v0 v0.9.1
go: labix.org/v2/mgo@v0.0.0-20140701140051-000000000287: bzr branch --use-existing-dir https://launchpad.net/mgo/v2 . in /Users/sergeylanz/go/pkg/mod/cache/vcs/ca61c737a32b1e09a0919e15375f9c2b6aa09860cc097f1333b3c3d29e040ea8: exec: "bzr": executable file not found in $PATH
go: finding github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd
go: finding github.com/stoewer/go-strcase v1.0.2
go: finding github.com/modern-go/reflect2 v1.0.1
go: finding github.com/terraform-providers/terraform-provider-random v2.0.0+incompatible
go: finding github.com/Azure/go-autorest v9.9.0+incompatible
go: finding k8s.io/apimachinery v0.0.0-20190116203031-d49e237a2683
go: finding github.com/google/gofuzz v1.0.0
go: finding k8s.io/kubectl v0.0.0-20190502165022-ce8d9f55c93c
go: finding github.com/googleapis/gnostic v0.2.0
go: finding k8s.io/api v0.0.0-20190116205037-c89978d5f86d
go: finding github.com/howeyc/gopass v0.0.0-20170109162249-bf9dde6d0d2c
go: finding github.com/gregjones/httpcache v0.0.0-20190212212710-3befbb6ad0cc
go: launchpad.net/gocheck@v0.0.0-20140225173054-000000000087: bzr branch --use-existing-dir https://launchpad.net/~niemeyer/gocheck/trunk . in /Users/sergeylanz/go/pkg/mod/cache/vcs/f46ce2ae80d31f9b0a29099baa203e3b6d269dace4e5357a2cf74bd109e13339: exec: "bzr": executable file not found in $PATH
go: error loading module requirements

know issue

metadata {
    annotations {}
    labels      {}
    name        = "system:kube-scheduler"
  }
Error: kubernetes_cluster_role_binding.system--kube-scheduler: metadata.0.name a DNS-1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. 'example.com', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*')

know issue

spec {
        active_deadline_seconds = "0"


Error: kubernetes_deployment.kube-system--l7-default-backend: spec.0.template.0.spec.0.active_deadline_seconds must be greater than 0 //pod, deployment, replicationcontrollers,

Error: kubernetes_deployment.kube-system--l7-default-backend: spec.0.template.0.metadata: attribute supports 1 item maximum, config has 2 declared

know issue secret,services and many,

Error: kubernetes_secret.kube-system--ttl-controller-token-f557l: data (ca): '' expected type 'string', got unconvertible type '[]map[string]interface {}'

All checking with 1.2.0 and lastest provider version + GKE

osdemah · 2019-05-06T14:04:08Z

The testing issues:

version controller of those removed libraries are "bazaar", by eliminating them this issues is fixed.
it's reported here: Name validation regex is invalid hashicorp/terraform-provider-kubernetes#214. (Should I ignore the resources with ":" in them or wait until being fixed by terraform?)
I belive it'll fix by considering optianal items of each resource, going to fix it very soon!
Apears to be same as the issue which I'm experiencing for many resources. .e.g. While the actual lable is eks.amazonaws.com/component: coredns it's genrating this labels block:

    labels {
      eks "amazonaws" {
        "com/component" = "coredns"
      }
    }

sergeylanzman · 2019-05-06T14:27:50Z

Issues:

Now I get changes files after go mod vendor

go: extracting github.com/keybase/go-crypto v0.0.0-20181031135447-f919bfda4fc1
go: extracting github.com/pierrec/lz4 v2.0.5+incompatible
root@b1f42b7dba1d:/go/src/github.com/GoogleCloudPlatform/terraformer# git statys
git: 'statys' is not a git command. See 'git --help'.

Did you mean this?
	status
root@b1f42b7dba1d:/go/src/github.com/GoogleCloudPlatform/terraformer# git status
On branch support_kubernetes_provider
Your branch is up-to-date with 'origin/support_kubernetes_provider'.
Changes not staged for commit:
  (use "git add/rm <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

	deleted:    vendor/contrib.go.opencensus.io/exporter/ocagent/LICENSE
	deleted:    vendor/contrib.go.opencensus.io/exporter/ocagent/version.go
	deleted:    vendor/github.com/Azure/go-autorest/autorest/adal/version.go
	deleted:    vendor/github.com/census-instrumentation/opencensus-proto/LICENSE
	deleted:    vendor/golang.org/x/sync/AUTHORS
	deleted:    vendor/golang.org/x/sync/CONTRIBUTORS
	deleted:    vendor/golang.org/x/sync/LICENSE
	deleted:    vendor/golang.org/x/sync/PATENTS

Untracked files:
  (use "git add <file>..." to include in what will be committed)

	vendor/github.com/Azure/go-autorest/autorest/adal/msi.go
	vendor/github.com/google/gofuzz/LICENSE
	vendor/github.com/googleapis/gnostic/compiler/main.go
	vendor/github.com/modern-go/concurrent/LICENSE
	vendor/github.com/modern-go/reflect2/reflect2_amd64.s
	vendor/github.com/modern-go/reflect2/relfect2_386.s
	vendor/github.com/modern-go/reflect2/relfect2_amd64p32.s
	vendor/github.com/modern-go/reflect2/relfect2_arm.s
	vendor/github.com/modern-go/reflect2/relfect2_arm64.s
	vendor/github.com/modern-go/reflect2/relfect2_mips64x.s
	vendor/github.com/modern-go/reflect2/relfect2_mipsx.s
	vendor/github.com/modern-go/reflect2/relfect2_ppc64x.s
	vendor/github.com/modern-go/reflect2/relfect2_s390x.s
	vendor/golang.org/x/time/AUTHORS
	vendor/golang.org/x/time/CONTRIBUTORS
	vendor/golang.org/x/time/LICENSE
	vendor/golang.org/x/time/PATENTS

no changes added to commit (use "git add" and/or "git commit -a")
root@b1f42b7dba1d:/go/src/github.com/GoogleCloudPlatform/terraformer#

I checking in clean env(Docker container)
2.

terraformer git:(support_kubernetes_provider) ./terraformer import kubernetes --resources=clusterrolebindings,configmaps,deployments,horizontalpodautoscalers,limitranges,namespaces,persistentvolumeclaims,persistentvolumes,pods,replicationcontrollers,resourcequotas,secrets,serviceaccounts,services,statefulsets,storageclasses
2019/05/06 17:25:08 kubernetes importing... clusterrolebindings
2019/05/06 17:25:21 unknown resource type: kubernetes_cluster_role_binding
2019/05/06 17:25:21 unknown resource type: kubernetes_cluster_role_binding
2019/05/06 17:25:21 unknown resource type: kubernetes_cluster_role_binding
2019/05/06 17:25:21 unknown resource type: kubernetes_cluster_role_binding
2019/05/06 17:25:21 unknown resource type: kubernetes_cluster_role_binding
2019/05/06 17:25:21 unknown resource type: kubernetes_cluster_role_binding
2019/05/06 17:25:21 unknown resource type: kubernetes_cluster_role_binding
2019/05/06 17:25:21 unknown resource type: kubernetes_cluster_role_binding
2019/05/06 17:25:21 unknown resource type: kubernetes_cluster_role_binding
2019/05/06 17:25:21 unknown resource type: kubernetes_cluster_role_binding
2019/05/06 17:25:21 unknown resource type: kubernetes_cluster_role_binding
2019/05/06 17:25:21 unknown resource type: kubernetes_cluster_role_binding
2019/05/06 17:25:21 unknown resource type: kubernetes_cluster_role_binding
2019/05/06 17:25:21 unknown resource type: kubernetes_cluster_role_binding
2019/05/06 17:25:21 unknown resource type: kubernetes_cluster_role_binding
2019/05/06 17:25:21 unknown resource type: kubernetes_cluster_role_binding
2019/05/06 17:25:21 unknown resource type: kubernetes_cluster_role_binding
2019/05/06 17:25:21 unknown resource type: kubernetes_cluster_role_binding
2019/05/06 17:25:21 unknown resource type: kubernetes_cluster_role_binding
2019/05/06 17:25:21 unknown resource type: kubernetes_cluster_role_binding
2019/05/06 17:25:21 unknown resource type: kubernetes_cluster_role_binding
2019/05/06 17:25:21 unknown resource type: kubernetes_cluster_role_binding
2019/05/06 17:25:21 unknown resource type: kubernetes_cluster_role_binding
2019/05/06 17:25:21 unknown resource type: kubernetes_cluster_role_binding
2019/05/06 17:25:21 unknown resource type: kubernetes_cluster_role_binding
2019/05/06 17:25:21 unknown resource type: kubernetes_cluster_role_binding
2019/05/06 17:25:21 unknown resource type: kubernetes_cluster_role_binding
2019/05/06 17:25:21 unknown resource type: kubernetes_cluster_role_binding
2019/05/06 17:25:21 unknown resource type: kubernetes_cluster_role_binding
2019/05/06 17:25:21 unknown resource type: kubernetes_cluster_role_binding
2019/05/06 17:25:21 unknown resource type: kubernetes_cluster_role_binding
2019/05/06 17:25:21 unknown resource type: kubernetes_cluster_role_binding
2019/05/06 17:25:21 unknown resource type: kubernetes_cluster_role_binding
2019/05/06 17:25:21 unknown resource type: kubernetes_cluster_role_binding
2019/05/06 17:25:21 unknown resource type: kubernetes_cluster_role_binding
2019/05/06 17:25:21 unknown resource type: kubernetes_cluster_role_binding
2019/05/06 17:25:21 unknown resource type: kubernetes_cluster_role_binding
2019/05/06 17:25:21 unknown resource type: kubernetes_cluster_role_binding
2019/05/06 17:25:21 unknown resource type: kubernetes_cluster_role_binding
2019/05/06 17:25:21 unknown resource type: kubernetes_cluster_role_binding
2019/05/06 17:25:22 unknown resource type: kubernetes_cluster_role_binding
2019/05/06 17:25:22 unknown resource type: kubernetes_cluster_role_binding
2019/05/06 17:25:22 unknown resource type: kubernetes_cluster_role_binding
2019/05/06 17:25:22 unknown resource type: kubernetes_cluster_role_binding
2019/05/06 17:25:22 unknown resource type: kubernetes_cluster_role_binding
2019/05/06 17:25:22 unknown resource type: kubernetes_cluster_role_binding
2019/05/06 17:25:22 unknown resource type: kubernetes_cluster_role_binding
2019/05/06 17:25:22 unknown resource type: kubernetes_cluster_role_binding
2019/05/06 17:25:22 unknown resource type: kubernetes_cluster_role_binding
2019/05/06 17:25:22 unknown resource type: kubernetes_cluster_role_binding
2019/05/06 17:25:22 unknown resource type: kubernetes_cluster_role_binding
2019/05/06 17:25:22 unknown resource type: kubernetes_cluster_role_binding
2019/05/06 17:25:22 kubernetes importing... configmaps
2019/05/06 17:25:58 kubernetes importing... deployments
2019/05/06 17:26:03 unknown resource type: kubernetes_deployment
2019/05/06 17:26:03 unknown resource type: kubernetes_deployment
2019/05/06 17:26:03 unknown resource type: kubernetes_deployment
2019/05/06 17:26:03 unknown resource type: kubernetes_deployment
2019/05/06 17:26:03 unknown resource type: kubernetes_deployment
2019/05/06 17:26:03 unknown resource type: kubernetes_deployment
2019/05/06 17:26:03 unknown resource type: kubernetes_deployment
2019/05/06 17:26:03 unknown resource type: kubernetes_deployment
2019/05/06 17:26:03 unknown resource type: kubernetes_deployment
2019/05/06 17:26:03 unknown resource type: kubernetes_deployment
2019/05/06 17:26:03 unknown resource type: kubernetes_deployment
2019/05/06 17:26:03 unknown resource type: kubernetes_deployment

kubernetes_deployment => kubernetes_deployments??

sergeylanzman · 2019-05-06T14:30:39Z

Can you add to Readme support kubernetes Here:

osdemah · 2019-05-06T16:39:45Z

Thanks for the review. I updated the readme files.

For the second issue, the resource types are correct but it seems that you're using a kubernetes provider which are not supporting all of the specefied resources. However it'll filter out the resource that are not supported by terraform in if _, ok := tfk8s.Provider().(*tfschema.Provider).ResourcesMap[extractTfResourceName(resource.Kind)]; !ok { line.

I checked both xyz and clusterrolebindings resources in the 1.1.0 and the erros were same:

./terraformer import kubernetes --resources=clusterrolebindings
2019/05/06 12:07:41 kubernetes importing... clusterrolebindings
2019/05/06 12:07:44 kubernetes: clusterrolebindings not supported resource

./terraformer import kubernetes --resources=xyz
2019/05/06 12:08:21 kubernetes importing... xyz
2019/05/06 12:08:24 kubernetes: xyz not supported resource

It seems taht as the problem is because of using some provider which is oldet than v.1.4.0. The module will filter out the modules which are not supported by the v1.4.0, but when genrating the tf files the compiled provider will be failed to do it. So I updated minimum version to v.1.4.0 to be same as the go.mod.

osdemah · 2019-05-06T16:45:13Z

For ignoring the optional fields: Each k8s module has tons of optional fields, I was thinking to add a boolean flag to Resource structure for ignoring every empty values in a resource. Something like IgnoreEveryEmptyValues. Do you think if it's good idea or I need to list the optinal fields for each resource?

sergeylanzman · 2019-05-06T18:35:58Z

Providers version now it's >1.4 and no =>1.4, maybe we start from latest version in go.mod?

By default shared infra in terraformer delete empty value, but you can define white list

terraformer/terraform_utils/resource.go

Lines 98 to 113 in 7f0e17b

    
           // delete empty keys with empty value, but not from AllowEmptyValue list 
        
           for keyAttribute, value := range r.InstanceState.Attributes { 
        
           	if value != "" { 
        
           		continue 
        
           	} 
        
           	allowEmptyValue := false 
        
           	for _, patter := range r.AllowEmptyValues { 
        
           		match, err := regexp.MatchString(patter, keyAttribute) 
        
           		if match && err == nil && patter != "" { 
        
           			allowEmptyValue = true 
        
           		} 
        
           	} 
        
           	if !allowEmptyValue { 
        
           		delete(attributes, keyAttribute) 
        
           	} 
        
           }

you get write example tf with empty values?

Maybe need add multi cluster options, like region in other providers?
3.1 Add option flag --clusters=
3.2 Write files in generated/kubernetes/NAME_OF_CLUSTER/RESOURCE, like regions in gcp/aws

osdemah · 2019-05-06T23:30:46Z

Yeah, It's being failed to delete some optional values! e.g. spec.0.template.0.spec.0.active_deadline_seconds for deployments.

I'm trying to debug it, but seems that it's only assuming "" as empty value, while it varies across different types. for instance it's "0" for active_deadline_seconds!

osdemah · 2019-05-07T02:16:12Z

Does cluster make sense here? I was thinking about considering kubeconfig context for separate paths but I don't think if it make sense here, because it's more likely to be sth like aws profile rahter than zone.

If we need to just use it for deviding paths, I'd say the flag should be --cluster, rather than --clusters

osdemah · 2019-05-07T02:53:26Z

Also I have to say the problem of converting "." to maps is related to calling Expand(m map[string]string, key string) for generating terraform files. This method wis unflattening the flattened resouce based on the "." in key name.

sergeylanzman · 2019-05-07T18:55:16Z

spec.0.template.0.spec.0.active_deadline_seconds - maybe we need write code in postConvertHook for this issues(int optional with default 0) and open separate issue with zero value. Add in code TODO and link to issue.
You can get name of cluster from API? if not, ok without cluster name
converting "." to maps - yes terraform standard tfstate file, it's keys with "." for to separate keys.
After work with many resources in GCP and AWS providers, I think it's provider issue.

I think let's finish 1 and 2 and to merge this, write about know issue in readme + open issue for this.
WDYT?

osdemah · 2019-05-08T00:44:38Z

Yeah, you right. Let's fix them in other issues.

cluster name: While there is a cluster name specified for k8s in the major cloud providers (as the resource name), the k8s itself doesn't have anything like cluster name.

sergeylanzman · 2019-05-08T08:46:20Z

kubernetes_terraforming/kind.go

+			map[string]string{},
+		))
+
+		k.Resources[len(k.Resources)-1].IgnoreKeys = []string{


Why you need this?
Terraformer have method for get ignoreKeys from provider automatically
https://github.com/GoogleCloudPlatform/terraformer/blob/master/terraform_utils/service.go#L69
You only need add

g.PopulateIgnoreKeys()

I didn't see that method :)) fixed!

sergeylanzman · 2019-05-08T08:48:02Z

Ok, only 1 comment about IgnoreKeys.
Can you squash all commits to 2 commits(vendor and your code), if it's hard can be 1 commit?

* doc: fix typo in documentation * doc: add template for PRs * fix: fix PR template * fead: add support for Certificate Manager * feat: implement Private CrossConnect * fix: use proper depth for API calls * feat: implement S3 keys * fix: rename a variable and fix a typo in a warning message (#12) * feat: add support for Container Registry (#11) * feat: add support for Data Platform (#13) * feat: add support for IONOSCLOUD shares (#14) * feat: add support for IONOSCLOUD users (#15) * feat: add support for CUBE and VCPU servers (#17) * feat: add support for Container Registry Tokens (#18) * feat: add support for ALB forwarding rules (#19) * feat: Add support for DBaaS Mongo Clusters (#22) * feat: add support for DNS Zones (#20) * feat: add support for DNS Zones * fix: Modify user agent and update documentation with info about mongo clusters * feat: add support for DNS Records (#23) * feat: add support for DBaaS Mongo Users (#24) * feat: add support for DBaaS PgSQL users (#25) * feat: add support for DBaaS PgSQL databases (#26) * feat: add support for Logging Pipelines (#28) * feat: add support for IP failover groups (#27) * refactor: fix linter errors (#29)

sergeylanzman requested changes May 6, 2019

View reviewed changes

sergeylanzman reviewed May 6, 2019

View reviewed changes

osdemah force-pushed the support_kubernetes_provider branch from e3539fc to 10ad60e Compare May 6, 2019 12:55

sergeylanzman reviewed May 8, 2019

View reviewed changes

osdemah added 2 commits May 8, 2019 05:28

Add kubernetes provider

4c033f3

Add vendor

0e715b1

osdemah force-pushed the support_kubernetes_provider branch from caa7777 to 0e715b1 Compare May 8, 2019 09:29

sergeylanzman approved these changes May 8, 2019

View reviewed changes

sergeylanzman merged commit f8ecc9a into GoogleCloudPlatform:master May 8, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support Kubernetes Provider #29

Support Kubernetes Provider #29

osdemah commented May 6, 2019

sergeylanzman left a comment

sergeylanzman May 6, 2019

sergeylanzman May 6, 2019

osdemah May 6, 2019

sergeylanzman commented May 6, 2019

osdemah commented May 6, 2019

sergeylanzman commented May 6, 2019

sergeylanzman commented May 6, 2019

osdemah commented May 6, 2019

osdemah commented May 6, 2019

sergeylanzman commented May 6, 2019

osdemah commented May 6, 2019 •

edited

Loading

osdemah commented May 7, 2019

osdemah commented May 7, 2019

sergeylanzman commented May 7, 2019

osdemah commented May 8, 2019

sergeylanzman May 8, 2019

osdemah May 8, 2019

sergeylanzman commented May 8, 2019

Support Kubernetes Provider #29

Support Kubernetes Provider #29

Conversation

osdemah commented May 6, 2019

sergeylanzman left a comment

Choose a reason for hiding this comment

sergeylanzman May 6, 2019

Choose a reason for hiding this comment

sergeylanzman May 6, 2019

Choose a reason for hiding this comment

osdemah May 6, 2019

Choose a reason for hiding this comment

sergeylanzman commented May 6, 2019

osdemah commented May 6, 2019

sergeylanzman commented May 6, 2019

sergeylanzman commented May 6, 2019

osdemah commented May 6, 2019

osdemah commented May 6, 2019

sergeylanzman commented May 6, 2019

osdemah commented May 6, 2019 • edited Loading

osdemah commented May 7, 2019

osdemah commented May 7, 2019

sergeylanzman commented May 7, 2019

osdemah commented May 8, 2019

sergeylanzman May 8, 2019

Choose a reason for hiding this comment

osdemah May 8, 2019

Choose a reason for hiding this comment

sergeylanzman commented May 8, 2019

osdemah commented May 6, 2019 •

edited

Loading