Skip to content

Commit

Permalink
Split Go generation into gogen package. (#695)
Browse files Browse the repository at this point in the history
* Make `EnumeratedYANGType.IdentityBaseName` exported.

I think it makes sense for this to be exported. Not sure why I wanted it
to be extracted from the plugin originally.

* Add `LangMapperExt` and `Flags` fields to each level of the `IR`.

The idea is to to call a different method from the interface for each level of hierarchy of the IR to populate the `Flag` fields for that level of the IR. So within `getOrderedDirDetails` for each `ParsedDirectory` and `NodeDetails`, and within the loop of genEnums (in `GenerateIR`) for each `EnumeratedYANGType`. This way the caller of `GenerateIR` is able to produce extra information for any level of the `IR`.

Tests added in genir_test.go

* Generate `YANGType` using `LangMapperExt` mechanism.

* Remove dependency of enumgen on MappedType

* Move enumset and schematree into `LangMapperBase`.

Currently, the individual LangMapper implementation instances (GoLangMapper and ProtoLangMapper) are required to hold these types, which can be confusing to the user since these are not fundamentally part of the LangMapper API -- rather it's their methods (i.e. enum name look-up and leafref resolution) that are part of the LangMapper API interface.

Embedding a special interface (`LangMapperBase`) not only decouples the implementation from the interface, but also provides a clear forward compatibility contract of methods (incl. enum name look-up and leafref resolution) available to implementors should more functionality become available in the future.

The proposed vision is:
* `LangMapper`-defined methods need to be implemented by LangMapper implementors.
* `LangMapperBaseSetup`-defined methods are set-up methods that are called either within `GenerateIR`, or by the unit tests of LangMapper methods.
* `LangMapperBase`-defined methods NOT part of the overall `LangMapper` interface are helper methods available to `LangMapper` implementors in their method implementations.

I will modify the design doc once all is agreed-upon and merged.

* Add `SetupEnumSet` and `SetupSchemaTree` to `LangMapperBase`.

This allows unit tests to be created by custom LangMappers on their methods (e.g. `FieldName`, `KeyLeafType`).

* Delete `YANGCodeGenerator.GetDirectoriesAndLeafTypes`

* Split Go generation into `gogen` package.

The change includes the following notable changes:
* gogen/codegen.go: Instead of using `ygen.NewYANGCodeGenerator` as the struct for all
  code generation from the single `ygen` package, `gogen.NewGoCodeGenerator` is created for Go
  generation. `GoOpts` is moved in `gogen`.
* gogen/genir_test: Since IR generation requires a suitable `LangMapper` implementation,
  the tests using `GoLangMapper` are moved to here.
* gogen/gogen.go, goelements.go: These are moved to this package.
* gogen/helpers.go: Go generation-specific helpers are moved here.
* internal/igenutil/genutil.go: Non-Go-specific generation helpers are
  moved here.
* ygen/genstate_test.go: A fake LangMapper implementation is created for
  unit testing.

* Fix Go workflow

* remove unused code

* Fix workflow

* Use bash instead of overalls

* Overwrite report for gogen

* Improve comments

* Change enumeratedTypedefTypeName to return a bool for whether the input type is actually a typedef

* Rename Setup to Inject and fix-up comments

* Don't install overalls

* Remove Go from some some public types/functions in gogen

* Split Proto generation into `protogen` package. (#696)

* Split Proto generation into `protogen` package.

The change includes the following notable changes:
* protogen/codegen.go: Instead of using `ygen.NewYANGCodeGenerator` as the struct for all
  code generation from the single `ygen` package, `protogen.NewProtoCodeGenerator` is created for Proto
  generation. `ProtoOpts` is moved to `protogen`.
* protogen/genir_test: Since IR generation requires a suitable `LangMapper` implementation,
  the tests using `ProtoLangMapper` are moved to here.
* protogen/protogen.go, protoelements.go: These are moved to this package.
* integration_tests/generate_errs_test.go: Moved `TestGenerateErrs` out of ygen.

* remove unused code

* remove unused variable

* add coverpkg

* Try again with coverpkg

* Try again with coverpkg

* Fix syntax error

* coverpkg=all

* coverpkg=./...

* coverpkg filter out irrelevant packages

* try to fix coverpkg

* try to fix coverpkg

* use bash instead of coveralls

* Remove 'proto' from various exported names

* GenerateProto3 -> Generate

* Delete unused variables and rephrase package comments for ygen. (#697)

* Delete unused variables and rephrase package comments for ygen.

* Fix `IdentityType` classification bug. (#699)

* Fix `IdentityType` classification bug.

Right now all typedefs are automatically categorized as `DerivedEnumerationType`; this is wrong because identity types could also be in a typedef.

Fixed this bug and added tests.

* Remove debugging statement

* Add coverage for proto generation of typedef identityref

* Run go generate ./... for CI check (#703)

* Run go generate ./... for CI check

* Regenerate exampleoc but don't depend on it for tests

* Delete `StoreRawSchema` from `GeneratorConfig` since it's not being used (#705)

* Delete `StoreRawSchema` from `GeneratorConfig` since it's not being used

I don't see the value of why we need this flag: is it to save memory?
For `exampleoc` this variable took 27MB, a negligible amount.

* Reorganize Code Generation flags and Delete `ygen.GeneratorConfig` (#706)

* ygen now only has `IROptions`.
* `CodeGenerator` in gogen/protogen now carry IROptions and their language-specific post-IR generation options.
* Code generation specific flags that are not used during IR generation are moved to `gogen` and `protogen`, duplicating if used by both (e.g. CallerName).
* `SkipEnumDeduplication` is moved from ParseOptions to TransformationOptions.
  • Loading branch information
wenovus committed Jun 9, 2022
1 parent 9a1316c commit 0793c34
Show file tree
Hide file tree
Showing 172 changed files with 5,664 additions and 5,247 deletions.
17 changes: 13 additions & 4 deletions .github/workflows/go.yml
Expand Up @@ -87,7 +87,7 @@ jobs:
# fake ygot and ytype paths specified in generation options.
"openconfig-options-explicit.formatted-txt.go"
)
for f in ygen/testdata/schema/*.formatted-txt; do
for f in gogen/testdata/schema/*.formatted-txt; do
if [[ ${skipped[@]} =~ $(basename $f) ]]; then
continue
fi
Expand All @@ -96,7 +96,7 @@ jobs:
skipped=(
)
for f in ygen/testdata/structs/*.formatted-txt; do
for f in gogen/testdata/structs/*.formatted-txt; do
if [[ ${skipped[@]} =~ $(basename $f) ]]; then
continue
fi
Expand All @@ -114,10 +114,14 @@ jobs:
fi
filename=$(basename $f)
f_prefix="${filename%%.*}"
ygen_file="ygen/testdata/structs/${f_prefix}.formatted-txt"
test-go-build $f $ygen_file
go_file="gogen/testdata/structs/${f_prefix}.formatted-txt"
test-go-build $f $go_file
done
- name: Make sure exampleoc doesn't error or panic
run: go generate ./...
working-directory: go/src/github.com/openconfig/ygot

static_analysis:
name: Static Analysis
runs-on: ubuntu-latest
Expand Down Expand Up @@ -149,6 +153,11 @@ jobs:
go test -covermode count -coverprofile profile.coverprofile -outputdir $dir $p
done
# Overwrite results for gogen/protogen since they also cover ygen's
# code.
go test -covermode count -coverprofile profile.coverprofile -outputdir gogen -coverpkg github.com/openconfig/ygot/gogen,github.com/openconfig/ygot/ygen github.com/openconfig/ygot/gogen
go test -covermode count -coverprofile profile.coverprofile -outputdir protogen -coverpkg github.com/openconfig/ygot/protogen,github.com/openconfig/ygot/ygen github.com/openconfig/ygot/protogen
echo 'mode: count' > concatenated.coverprofile
for p in $pkgs; do
dir=$(echo $p | sed -e 's/^github\.com\/openconfig\/ygot\///')
Expand Down
32 changes: 11 additions & 21 deletions Makefile
@@ -1,30 +1,20 @@
# ygot Makefile
#
# This makefile is used by Travis CI to run tests against the ygot library.
# This makefile is used by GitHub Actions CI to run tests against the ygot library.
#
ROOT_DIR:=$(shell dirname $(realpath $(lastword $(MAKEFILE_LIST))))

test:
go test ./...
generate:
cd ${ROOT_DIR}/demo/getting_started && SRCDIR=${ROOT_DIR} go generate
cd ${ROOT_DIR}/proto/ywrapper && SRCDIR=${ROOT_DIR} go generate
cd $(ROOT_DIR)/proto/yext && SRCDIR=${ROOT_DIR} go generate
cd $(ROOT_DIR)/demo/uncompressed && SRCDIR=${ROOT_DIR} go generate
cd $(ROOT_DIR)/demo/protobuf_getting_started && SRCDIR=${ROOT_DIR} ./update.sh
cd $(ROOT_DIR)/integration_tests/uncompressed && SRCDIR=${ROOT_DIR} go generate
cd $(ROOT_DIR)/integration_tests/annotations/apb && SRCDIR=${ROOT_DIR} go generate
cd $(ROOT_DIR)/integration_tests/annotations/proto2apb && SRCDIR=${ROOT_DIR} go generate
go generate ./demo/getting_started
go generate ./proto/ywrapper
go generate ./proto/yext
go generate ./demo/uncompressed
go generate ./demo/protobuf_getting_started
go generate ./integration_tests/uncompressed
go generate ./integration_tests/annotations/apb
go generate ./integration_tests/annotations/proto2apb
clean:
rm -f ${ROOT_DIR}/demo/getting_started/pkg/ocdemo/oc.go
rm -f ${ROOT_DIR}/demo/uncompressed/pkg/demo/uncompressed.go
deps:
go get -t -d ./ygot
go get -t -d ./ygen
go get -t -d ./generator
go get -t -d ./proto_generator
go get -t -d ./exampleoc
go get -t -d ./ytypes
go get -t -d ./demo/gnmi_telemetry
rm -f demo/getting_started/pkg/ocdemo/oc.go
rm -f demo/uncompressed/pkg/demo/uncompressed.go
install: deps generate
all: clean deps generate test
30 changes: 16 additions & 14 deletions demo/getting_started/interfaces_test.go
Expand Up @@ -9,6 +9,7 @@ import (
"testing"

"github.com/openconfig/ygot/genutil"
"github.com/openconfig/ygot/gogen"
"github.com/openconfig/ygot/ygen"
)

Expand All @@ -21,23 +22,24 @@ var TestRoot string
func TestGenerateCode(t *testing.T) {
tests := []struct {
name string
inConfig *ygen.GeneratorConfig
inIROpts ygen.IROptions
inGoOpts gogen.GoOpts
inFiles []string
inPaths []string
}{{
name: "openconfig interfaces",
inConfig: &ygen.GeneratorConfig{
GoOptions: ygen.GoOpts{
GenerateSimpleUnions: true,
},
inIROpts: ygen.IROptions{
ParseOptions: ygen.ParseOpts{
ExcludeModules: []string{"ietf-interfaces"},
},
TransformationOptions: ygen.TransformationOpts{
CompressBehaviour: genutil.PreferIntendedConfig,
GenerateFakeRoot: true,
},
GenerateJSONSchema: true,
},
inGoOpts: gogen.GoOpts{
GenerateJSONSchema: true,
GenerateSimpleUnions: true,
},
inFiles: []string{
filepath.Join(TestRoot, "yang", "openconfig-interfaces.yang"),
Expand All @@ -46,17 +48,17 @@ func TestGenerateCode(t *testing.T) {
inPaths: []string{filepath.Join(TestRoot, "yang")},
}, {
name: "openconfig interfaces with no compression",
inConfig: &ygen.GeneratorConfig{
GoOptions: ygen.GoOpts{
GenerateSimpleUnions: true,
},
inIROpts: ygen.IROptions{
ParseOptions: ygen.ParseOpts{
ExcludeModules: []string{"ietf-interfaces"},
},
TransformationOptions: ygen.TransformationOpts{
GenerateFakeRoot: true,
},
GenerateJSONSchema: true,
},
inGoOpts: gogen.GoOpts{
GenerateJSONSchema: true,
GenerateSimpleUnions: true,
},
inFiles: []string{
filepath.Join(TestRoot, "yang", "openconfig-interfaces.yang"),
Expand All @@ -66,10 +68,10 @@ func TestGenerateCode(t *testing.T) {
}}

for _, tt := range tests {
cg := ygen.NewYANGCodeGenerator(tt.inConfig)
got, err := cg.GenerateGoCode(tt.inFiles, tt.inPaths)
cg := gogen.New("", tt.inIROpts, tt.inGoOpts)
got, err := cg.Generate(tt.inFiles, tt.inPaths)
if err != nil {
t.Errorf("%s: GenerateGoCode(%v, %v): Config: %v, got unexpected error: %v", tt.name, tt.inFiles, tt.inPaths, tt.inConfig, err)
t.Errorf("%s: Generate(%v, %v): Config: %v, got unexpected error: %v", tt.name, tt.inFiles, tt.inPaths, tt.inIROpts, err)
continue
}

Expand Down
2 changes: 2 additions & 0 deletions demo/protobuf_getting_started/demo.go
Expand Up @@ -17,6 +17,8 @@
// to generate a Protobuf form of the OpenConfig RIB model.
package main

//go:generate ./update.sh

import (
"fmt"

Expand Down
2 changes: 0 additions & 2 deletions demo/protobuf_getting_started/update.sh
Expand Up @@ -17,8 +17,6 @@ go run ../../proto_generator/protogenerator.go \
-base_import_path="github.com/openconfig/ygot/demo/protobuf_getting_started/ribproto" \
-go_package_base="github.com/openconfig/ygot/demo/protobuf_getting_started/ribproto" \
-path=yang -output_dir=ribproto \
-typedef_enum_with_defmod \
-consistent_union_enum_names \
-enum_package_name=enums -package_name=openconfig \
-exclude_modules=ietf-interfaces \
yang/rib/openconfig-rib-bgp.yang
Expand Down
58 changes: 31 additions & 27 deletions generator/generator.go
Expand Up @@ -29,6 +29,7 @@ import (
log "github.com/golang/glog"
"github.com/openconfig/goyang/pkg/yang"
"github.com/openconfig/ygot/genutil"
"github.com/openconfig/ygot/gogen"
"github.com/openconfig/ygot/ygen"
"github.com/openconfig/ygot/ypathgen"
)
Expand Down Expand Up @@ -85,7 +86,7 @@ var (
goyangImportPath = flag.String("goyang_path", genutil.GoDefaultGoyangImportPath, "The import path to use for goyang's yang package.")
generateRename = flag.Bool("generate_rename", false, "If set to true, rename methods are generated for lists within the Go code.")
addAnnotations = flag.Bool("annotations", false, "If set to true, metadata annotations are added within the generated structs.")
annotationPrefix = flag.String("annotation_prefix", ygen.DefaultAnnotationPrefix, "String to be appended to each metadata field within the generated structs if annoations is set to true.")
annotationPrefix = flag.String("annotation_prefix", gogen.DefaultAnnotationPrefix, "String to be appended to each metadata field within the generated structs if annoations is set to true.")
addYangPresence = flag.Bool("yangpresence", false, "If set to true, a tag will be added to the field of a generated Go struct to indicate when a YANG presence container is being used.")
generateAppend = flag.Bool("generate_append", false, "If set to true, append methods are generated for YANG lists (Go maps) within the Go code.")
generateGetters = flag.Bool("generate_getters", false, "If set to true, getter methdos that retrieve or create an element are generated for YANG container (Go struct pointer) or list (Go map) fields within the generated code.")
Expand All @@ -108,10 +109,10 @@ var (
packageSuffix = flag.String("path_struct_package_suffix", "path", "Suffix to append to generated Go package names, when split_pathstructs_by_module=true.")
)

// writeGoCodeSingleFile takes a ygen.GeneratedGoCode struct and writes the Go code
// writeGoCodeSingleFile takes a gogen.GeneratedCode struct and writes the Go code
// snippets contained within it to the io.Writer, w, provided as an argument.
// The output includes a package header which is generated.
func writeGoCodeSingleFile(w io.Writer, goCode *ygen.GeneratedGoCode) error {
func writeGoCodeSingleFile(w io.Writer, goCode *gogen.GeneratedCode) error {
// Write the package header to the supplier writer.
fmt.Fprint(w, goCode.CommonHeader)
fmt.Fprint(w, goCode.OneOffHeader)
Expand Down Expand Up @@ -151,11 +152,11 @@ func writeGoPathCodeSingleFile(w io.Writer, pathCode *ypathgen.GeneratedPathCode

// splitCodeByFileN generates a map, keyed by filename, to a string containing
// the code to be output to that filename. It allows division of a
// ygen.GeneratedGoCode struct into a set of source files. It divides the
// gogen.GeneratedCode struct into a set of source files. It divides the
// methods, interfaces, and enumeration code snippets into their own files.
// Structs are output into files by splitting them evenly among the input split
// number.
func splitCodeByFileN(goCode *ygen.GeneratedGoCode, fileN int) (map[string]string, error) {
func splitCodeByFileN(goCode *gogen.GeneratedCode, fileN int) (map[string]string, error) {
structN := len(goCode.Structs)
if fileN < 1 || fileN > structN {
return nil, fmt.Errorf("requested %d files, but must be between 1 and %d (number of schema structs)", fileN, structN)
Expand Down Expand Up @@ -324,28 +325,30 @@ func main() {
}

// Perform the code generation.
cg := ygen.NewYANGCodeGenerator(&ygen.GeneratorConfig{
ParseOptions: ygen.ParseOpts{
ExcludeModules: modsExcluded,
SkipEnumDeduplication: *skipEnumDedup,
YANGParseOptions: yang.Options{
IgnoreSubmoduleCircularDependencies: *ignoreCircDeps,
cg := gogen.New(
"",
ygen.IROptions{
ParseOptions: ygen.ParseOpts{
ExcludeModules: modsExcluded,
YANGParseOptions: yang.Options{
IgnoreSubmoduleCircularDependencies: *ignoreCircDeps,
},
},
TransformationOptions: ygen.TransformationOpts{
CompressBehaviour: compressBehaviour,
GenerateFakeRoot: *generateFakeRoot,
FakeRootName: *fakeRootName,
SkipEnumDeduplication: *skipEnumDedup,
ShortenEnumLeafNames: *shortenEnumLeafNames,
EnumOrgPrefixesToTrim: enumOrgPrefixesToTrim,
UseDefiningModuleForTypedefEnumNames: *useDefiningModuleForTypedefEnumNames,
EnumerationsUseUnderscores: true,
},
},
TransformationOptions: ygen.TransformationOpts{
CompressBehaviour: compressBehaviour,
IgnoreShadowSchemaPaths: *ignoreShadowSchemaPaths,
GenerateFakeRoot: *generateFakeRoot,
FakeRootName: *fakeRootName,
ShortenEnumLeafNames: *shortenEnumLeafNames,
EnumOrgPrefixesToTrim: enumOrgPrefixesToTrim,
UseDefiningModuleForTypedefEnumNames: *useDefiningModuleForTypedefEnumNames,
EnumerationsUseUnderscores: true,
},
PackageName: *packageName,
GenerateJSONSchema: *generateSchema,
IncludeDescriptions: *includeDescriptions,
GoOptions: ygen.GoOpts{
gogen.GoOpts{
PackageName: *packageName,
GenerateJSONSchema: *generateSchema,
IncludeDescriptions: *includeDescriptions,
YgotImportPath: *ygotImportPath,
YtypesImportPath: *ytypesImportPath,
GoyangImportPath: *goyangImportPath,
Expand All @@ -362,10 +365,11 @@ func main() {
GenerateSimpleUnions: *generateSimpleUnions,
IncludeModelData: *includeModelData,
AppendEnumSuffixForSimpleUnionEnums: *appendEnumSuffixForSimpleUnionEnums,
IgnoreShadowSchemaPaths: *ignoreShadowSchemaPaths,
},
})
)

generatedGoCode, errs := cg.GenerateGoCode(generateModules, includePaths)
generatedGoCode, errs := cg.Generate(generateModules, includePaths)
if errs != nil {
log.Exitf("ERROR Generating GoStruct Code: %v\n", errs)
}
Expand Down

0 comments on commit 0793c34

Please sign in to comment.