Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Generate NodeJS SDK from schema #1151

Merged
merged 44 commits into from
Jun 9, 2020
Merged

Generate NodeJS SDK from schema #1151

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
44 commits
Select commit Hold shift + click to select a range
bd5612a
WIP
lblackstone Mar 30, 2020
7f384d4
WIP - schema-based codegen
lblackstone Mar 30, 2020
bc0a0d3
Add pulumiComment to resource descriptions
lblackstone Mar 31, 2020
6555eca
Fix module names
lblackstone Mar 31, 2020
5b25b9a
More module name fixes
lblackstone Apr 1, 2020
ba4c492
Fix getters
lblackstone Apr 1, 2020
b39256e
Fix constructors
lblackstone Apr 1, 2020
6f22d57
Add const values
lblackstone Apr 3, 2020
8f4b574
Fix package.json and update to k8s 1.18
lblackstone Apr 3, 2020
f29a318
SDK compiles
lblackstone Apr 6, 2020
be2ee14
Generate yaml.ts
lblackstone Apr 7, 2020
4d91cec
Simplify yaml.ts codegen
lblackstone Apr 10, 2020
68949e7
WIP
lblackstone Apr 14, 2020
b8cc3c0
YAML working
lblackstone May 12, 2020
ee49549
Merge branch 'master' into lblackstone/ts-schema-gen
lblackstone May 21, 2020
ca7cc2c
Merge branch 'master' into lblackstone/ts-schema-gen
lblackstone May 21, 2020
796c43e
More cleanup
lblackstone May 21, 2020
de52a1a
WIP - backward compat
lblackstone May 21, 2020
a7ff111
Merge branch 'master' into lblackstone/ts-schema-gen
lblackstone May 26, 2020
c579335
Fix YAML
lblackstone May 26, 2020
123198d
Add CustomResource overlay
lblackstone May 28, 2020
2fe081c
Pass overlays into codegen
lblackstone May 28, 2020
e2dcf2c
Set compat mode
lblackstone May 28, 2020
63c123f
Cleanup
lblackstone May 28, 2020
d74fb0e
Pull in codegen changes
lblackstone May 28, 2020
31e0efd
Update readme generation
lblackstone May 28, 2020
c013cda
Merge branch 'master' into lblackstone/ts-schema-gen
lblackstone May 29, 2020
0638355
make ensure
lblackstone May 29, 2020
6ec87d6
readme
lblackstone May 29, 2020
b636d6a
Refactoring and docs
lblackstone May 29, 2020
a200713
Doc warning
lblackstone May 29, 2020
310b2f8
More docs
lblackstone May 29, 2020
20918d2
Pull in latest p/p changes
lblackstone Jun 3, 2020
80b7ade
More fixes
lblackstone Jun 5, 2020
5d55eac
Revert makefile changes
lblackstone Jun 5, 2020
812b8e6
changelog
lblackstone Jun 5, 2020
a3e0012
Use common getVersion
lblackstone Jun 5, 2020
3ae203f
Fix go.mod
lblackstone Jun 5, 2020
ed37d8d
Fix package exports
lblackstone Jun 8, 2020
38c3550
Fix output types
lblackstone Jun 8, 2020
fe6f47e
YAML cleanup
lblackstone Jun 8, 2020
f68e5e6
Merge branch 'master' into lblackstone/ts-schema-gen
lblackstone Jun 8, 2020
2e2d197
Regenerate SDK with schema fix
lblackstone Jun 8, 2020
6c6ab4f
Fix YAML
lblackstone Jun 9, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
1 change: 1 addition & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@

### Improvements

- NodeJS SDK updated to align with other Pulumi NodeJS SDKs. (https://github.com/pulumi/pulumi-kubernetes/pull/1151)
- .NET SDK updated to align with other Pulumi .NET SDKs. (https://github.com/pulumi/pulumi-kubernetes/pull/1132)
- Deprecated resources are now marked as `Obsolete`.
- Many classes are moved to new locations on disk while preserving the public namespaces and API.
Expand Down
171 changes: 69 additions & 102 deletions provider/cmd/pulumi-gen-kubernetes/main.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,7 @@ import (
"github.com/pulumi/pulumi/pkg/v2/codegen"
dotnetgen "github.com/pulumi/pulumi/pkg/v2/codegen/dotnet"
gogen "github.com/pulumi/pulumi/pkg/v2/codegen/go"
nodejsgen "github.com/pulumi/pulumi/pkg/v2/codegen/nodejs"
"github.com/pulumi/pulumi/pkg/v2/codegen/schema"
"github.com/pulumi/pulumi/sdk/v2/go/common/util/contract"
)
Expand All @@ -53,12 +54,23 @@ var TemplateDir string
// BaseDir is the path to the base pulumi-kubernetes directory.
var BaseDir string

// Language is the SDK language.
type Language string

const (
DotNet Language = "dotnet"
Go Language = "go"
NodeJS Language = "nodejs"
Python Language = "python"
Schema Language = "schema"
)

func main() {
if len(os.Args) < 4 {
log.Fatal("Usage: gen <language> <swagger-file> <root-pulumi-kubernetes-dir>")
}

language := os.Args[1]
language := Language(os.Args[1])

swagger, err := ioutil.ReadFile(os.Args[2])
if err != nil {
Expand All @@ -81,7 +93,7 @@ func main() {

BaseDir = os.Args[3]
TemplateDir = path.Join(BaseDir, "provider", "pkg", "gen")
outdir := path.Join(BaseDir, "sdk", language)
outdir := path.Join(BaseDir, "sdk", string(language))

// Generate schema
pkgSpec := gen.PulumiSchema(data)
Expand All @@ -93,118 +105,71 @@ func main() {
genK8sResourceTypes(pkg)

switch language {
case "nodejs":
case NodeJS:
templateDir := path.Join(TemplateDir, "nodejs-templates")
writeNodeJSClient(data, outdir, templateDir)
case "python":
writeNodeJSClient(pkg, outdir, templateDir)
case Python:
templateDir := path.Join(TemplateDir, "python-templates")
writePythonClient(data, outdir, templateDir)
case "dotnet":
case DotNet:
templateDir := path.Join(TemplateDir, "dotnet-templates")
writeDotnetClient(pkg, data, outdir, templateDir)
case "go":
case Go:
templateDir := path.Join(TemplateDir, "go-templates")
writeGoClient(pkg, outdir, templateDir)
case "schema":
case Schema:
mustWritePulumiSchema(pkgSpec, outdir)
default:
panic(fmt.Sprintf("Unrecognized language '%s'", language))
}
}

func writeNodeJSClient(data map[string]interface{}, outdir, templateDir string) {
inputAPIts, ouputAPIts, indexts, yamlts, packagejson, groupsts, err := gen.NodeJSClient(
data, templateDir)
func writeNodeJSClient(pkg *schema.Package, outdir, templateDir string) {
resources, err := nodejsgen.LanguageResources(pkg)
if err != nil {
panic(err)
}

err = os.MkdirAll(outdir, 0700)
if err != nil {
panic(err)
}

typesDir := fmt.Sprintf("%s/types", outdir)
err = os.MkdirAll(typesDir, 0700)
if err != nil {
panic(err)
}

err = ioutil.WriteFile(fmt.Sprintf("%s/input.ts", typesDir), []byte(inputAPIts), 0777)
if err != nil {
panic(err)
}

err = ioutil.WriteFile(fmt.Sprintf("%s/output.ts", typesDir), []byte(ouputAPIts), 0777)
if err != nil {
panic(err)
}

err = ioutil.WriteFile(fmt.Sprintf("%s/yaml/yaml.ts", outdir), []byte(yamlts), 0777)
if err != nil {
panic(err)
}

for groupName, group := range groupsts {
groupDir := fmt.Sprintf("%s/%s", outdir, groupName)
err = os.MkdirAll(groupDir, 0700)
if err != nil {
panic(err)
templateResources := gen.TemplateResources{}
for _, resource := range resources {
if resource.Package == "" {
continue
}

for versionName, version := range group.Versions {
versionDir := fmt.Sprintf("%s/%s", groupDir, versionName)
err = os.MkdirAll(versionDir, 0700)
if err != nil {
panic(err)
}

for kindName, kind := range version.Kinds {
err = ioutil.WriteFile(fmt.Sprintf("%s/%s.ts", versionDir, kindName), []byte(kind), 0777)
if err != nil {
panic(err)
}
}

err = ioutil.WriteFile(fmt.Sprintf("%s/%s.ts", versionDir, "index"), []byte(version.Index), 0777)
if err != nil {
panic(err)
}
tr := gen.TemplateResource{
Name: resource.Name,
Package: resource.Package,
Token: resource.Token,
}

err = ioutil.WriteFile(fmt.Sprintf("%s/%s.ts", groupDir, "index"), []byte(group.Index), 0777)
if err != nil {
panic(err)
for _, property := range resource.Properties {
tp := gen.TemplateProperty{
ConstValue: property.ConstValue,
Name: property.Name,
Package: property.Package,
}
tr.Properties = append(tr.Properties, tp)
}
templateResources.Resources = append(templateResources.Resources, tr)
}
sort.Slice(templateResources.Resources, func(i, j int) bool {
return templateResources.Resources[i].Token < templateResources.Resources[j].Token
})

err = ioutil.WriteFile(fmt.Sprintf("%s/index.ts", outdir), []byte(indexts), 0777)
if err != nil {
panic(err)
overlays := map[string][]byte{
"apiextensions/customResource.ts": mustLoadFile(filepath.Join(templateDir, "apiextensions", "customResource.ts")),
"helm/v2/helm.ts": mustLoadFile(filepath.Join(templateDir, "helm", "v2", "helm.ts")),
"helm/v3/helm.ts": mustLoadFile(filepath.Join(templateDir, "helm", "v2", "helm.ts")), // v3 support is currently identical to v2
"yaml/yaml.ts": mustRenderTemplate(filepath.Join(templateDir, "yaml", "yaml.tmpl"), templateResources),
}

err = ioutil.WriteFile(fmt.Sprintf("%s/package.json", outdir), []byte(packagejson), 0777)
files, err := nodejsgen.GeneratePackage("pulumigen", pkg, overlays)
if err != nil {
panic(err)
}

err = CopyFile(
filepath.Join(templateDir, "CustomResource.ts"), filepath.Join(outdir, "apiextensions", "CustomResource.ts"))
if err != nil {
panic(err)
}

err = CopyFile(filepath.Join(templateDir, "README.md"), filepath.Join(outdir, "README.md"))
if err != nil {
panic(err)
}
// Internal files that don't need to be exported
files["path.ts"] = mustLoadFile(filepath.Join(templateDir, "path.ts"))
files["tests/path.ts"] = mustLoadFile(filepath.Join(templateDir, "tests", "path.ts"))
Comment on lines +169 to +170
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@pgavlin These two files are not part of the user-facing SDK (i.e. I don't want them in the top level index.ts. This has the side-effect of not including them in the package.json list of files. Is this ok, or do you have another suggestion?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cc @lukehoban

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that's okay. I think our JS doc generator relies upon public surface area being in package.json, but sounds like this is not public surface area. I don't know of anything else that relies on the files list in package.json


err = CopyDir(filepath.Join(templateDir, "helm"), filepath.Join(outdir, "helm"))
if err != nil {
panic(err)
}
fmt.Printf("%s/package.json\n", outdir)
fmt.Println(err)
mustWriteFiles(outdir, files)
}

func writePythonClient(data map[string]interface{}, outdir, templateDir string) {
Expand Down Expand Up @@ -312,8 +277,7 @@ func writeGoClient(pkg *schema.Package, outdir string, templateDir string) {
panic(err)
}

templateResources := gen.TemplateResources{}
imports := codegen.StringSet{}
templateResources := gen.GoTemplateResources{}
for _, resource := range resources {
r := gen.TemplateResource{
Alias: resource.Alias,
Expand All @@ -322,22 +286,19 @@ func writeGoClient(pkg *schema.Package, outdir string, templateDir string) {
Token: resource.Token,
}
templateResources.Resources = append(templateResources.Resources, r)
importPath := fmt.Sprintf(`%s "%s"`, resource.Alias, resource.Package)
imports.Add(importPath)
}
templateResources.Imports = imports.SortedValues()
sort.Slice(templateResources.Resources, func(i, j int) bool {
return templateResources.Resources[i].Token < templateResources.Resources[j].Token
})

files["kubernetes/types.go"] = mustRenderTemplate(filepath.Join(templateDir, "types.tmpl"), templateResources)
files["kubernetes/apiextensions/customResource.go"] = mustRenderTemplate(filepath.Join(templateDir, "apiextensions", "customResource.tmpl"), templateResources)
files["kubernetes/helm/v2/chart.go"] = mustRenderTemplate(filepath.Join(templateDir, "helm", "v2", "chart.tmpl"), templateResources)
files["kubernetes/helm/v2/types.go"] = mustRenderTemplate(filepath.Join(templateDir, "helm", "v2", "types.tmpl"), templateResources)
files["kubernetes/yaml/configFile.go"] = mustRenderTemplate(filepath.Join(templateDir, "yaml", "configFile.tmpl"), templateResources)
files["kubernetes/yaml/configGroup.go"] = mustRenderTemplate(filepath.Join(templateDir, "yaml", "configGroup.tmpl"), templateResources)
files["kubernetes/yaml/transformation.go"] = mustRenderTemplate(filepath.Join(templateDir, "yaml", "transformation.tmpl"), templateResources)
files["kubernetes/yaml/yaml.go"] = mustRenderTemplate(filepath.Join(templateDir, "yaml", "yaml.tmpl"), templateResources)
files["kubernetes/types.go"] = mustRenderGoTemplate(filepath.Join(templateDir, "types.tmpl"), templateResources)
files["kubernetes/apiextensions/customResource.go"] = mustRenderGoTemplate(filepath.Join(templateDir, "apiextensions", "customResource.tmpl"), templateResources)
files["kubernetes/helm/v2/chart.go"] = mustRenderGoTemplate(filepath.Join(templateDir, "helm", "v2", "chart.tmpl"), templateResources)
files["kubernetes/helm/v2/types.go"] = mustRenderGoTemplate(filepath.Join(templateDir, "helm", "v2", "types.tmpl"), templateResources)
files["kubernetes/yaml/configFile.go"] = mustRenderGoTemplate(filepath.Join(templateDir, "yaml", "configFile.tmpl"), templateResources)
files["kubernetes/yaml/configGroup.go"] = mustRenderGoTemplate(filepath.Join(templateDir, "yaml", "configGroup.tmpl"), templateResources)
files["kubernetes/yaml/transformation.go"] = mustRenderGoTemplate(filepath.Join(templateDir, "yaml", "transformation.tmpl"), templateResources)
files["kubernetes/yaml/yaml.go"] = mustRenderGoTemplate(filepath.Join(templateDir, "yaml", "yaml.tmpl"), templateResources)

mustWriteFiles(outdir, files)
}
Expand All @@ -359,15 +320,21 @@ func mustRenderTemplate(path string, resources interface{}) []byte {
if err != nil {
panic(err)
}
formattedSource, err := format.Source(buf.Bytes())
return buf.Bytes()
}

func mustRenderGoTemplate(path string, resources interface{}) []byte {
bytes := mustRenderTemplate(path, resources)

formattedSource, err := format.Source(bytes)
if err != nil {
panic(err)
}

return formattedSource
}

func genPulumiSchemaPackage(pkgSpec schema.PackageSpec) *schema.Package {

pkg, err := schema.ImportSpec(pkgSpec, nil)
if err != nil {
panic(err)
Expand Down Expand Up @@ -396,7 +363,7 @@ func genK8sResourceTypes(pkg *schema.Package) {
}

files := map[string][]byte{}
files["provider/pkg/kinds/kinds.go"] = mustRenderTemplate(path.Join(TemplateDir, "kinds", "kinds.tmpl"), gvk)
files["provider/pkg/kinds/kinds.go"] = mustRenderGoTemplate(path.Join(TemplateDir, "kinds", "kinds.tmpl"), gvk)
mustWriteFiles(BaseDir, files)
}

Expand Down
2 changes: 1 addition & 1 deletion provider/go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ require (
github.com/imdario/mergo v0.3.8
github.com/mitchellh/go-wordwrap v1.0.0
github.com/pkg/errors v0.9.1
github.com/pulumi/pulumi/pkg/v2 v2.3.0
github.com/pulumi/pulumi/pkg/v2 v2.3.1-0.20200607162109-9754465b04db
github.com/pulumi/pulumi/sdk/v2 v2.3.0
github.com/stretchr/testify v1.5.1
google.golang.org/grpc v1.28.0
Expand Down
6 changes: 2 additions & 4 deletions provider/go.sum
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -448,9 +448,8 @@ github.com/prometheus/procfs v0.0.0-20181005140218-185b4288413d/go.mod h1:c3At6R
github.com/prometheus/procfs v0.0.0-20190507164030-5867b95ac084/go.mod h1:TjEm7ze935MbeOT/UhFTIMYKhuLP4wbCsTZCD3I8kEA=
github.com/prometheus/procfs v0.0.2/go.mod h1:TjEm7ze935MbeOT/UhFTIMYKhuLP4wbCsTZCD3I8kEA=
github.com/prometheus/tsdb v0.7.1/go.mod h1:qhTCs0VvXwvX/y3TZrWD7rabWM+ijKTux40TwIPHuXU=
github.com/pulumi/pulumi/pkg/v2 v2.3.0 h1:ujk+Cv9h+PdC/w15xPN0hCbYkhUaanSMPoGvmviGFsM=
github.com/pulumi/pulumi/pkg/v2 v2.3.0/go.mod h1:O/MWGU3wbUGXPzZzWBiPQUjD8iE2AJnrjq6fsRKSB+s=
github.com/pulumi/pulumi/sdk/v2 v2.0.0 h1:3VMXbEo3bqeaU+YDt8ufVBLD0WhLYE3tG3t/nIZ3Iac=
github.com/pulumi/pulumi/pkg/v2 v2.3.1-0.20200607162109-9754465b04db h1:5HrO1GNA53QEp+yN1UCX7BUsJD2I5/k1u2KLO4hOxOA=
github.com/pulumi/pulumi/pkg/v2 v2.3.1-0.20200607162109-9754465b04db/go.mod h1:aqo0wBboEEdy+T5EYKUB6YnzbL80VPjgJPwjttuHqT0=
github.com/pulumi/pulumi/sdk/v2 v2.0.0/go.mod h1:W7k1UDYerc5o97mHnlHHp5iQZKEby+oQrQefWt+2RF4=
github.com/pulumi/pulumi/sdk/v2 v2.3.0 h1:uvRYCmoHILKlyyIbXa5CcLSKKt9n2s8j+GKTffpXQf4=
github.com/pulumi/pulumi/sdk/v2 v2.3.0/go.mod h1:cvivzHVRA5Xu3NSE/obmHzO3L693IJSd5QccQuBOMUE=
Expand Down Expand Up @@ -483,7 +482,6 @@ github.com/spf13/afero v1.2.2/go.mod h1:9ZxEEn6pIJ8Rxe320qSDBk6AsU0r9pR7Q4OcevTd
github.com/spf13/cast v1.3.0/go.mod h1:Qx5cxh0v+4UWYiBimWS+eyWzqEqokIECu5etghLkUJE=
github.com/spf13/cast v1.3.1/go.mod h1:Qx5cxh0v+4UWYiBimWS+eyWzqEqokIECu5etghLkUJE=
github.com/spf13/cobra v0.0.5/go.mod h1:3K3wKZymM7VvHMDS9+Akkh4K60UwM26emMESw8tLCHU=
github.com/spf13/cobra v0.0.6 h1:breEStsVwemnKh2/s6gMvSdMEkwW0sK8vGStnlVBMCs=
github.com/spf13/cobra v0.0.6/go.mod h1:/6GTrnGXV9HjY+aR4k0oJ5tcvakLuG6EuKReYlHNrgE=
github.com/spf13/cobra v1.0.0 h1:6m/oheQuQ13N9ks4hubMG6BnvwOeaJrqSPLahSnczz8=
github.com/spf13/cobra v1.0.0/go.mod h1:/6GTrnGXV9HjY+aR4k0oJ5tcvakLuG6EuKReYlHNrgE=
Expand Down
Loading