slides.html

<!DOCTYPE html>
<html>
<head>
    <title>Exploring Code Generation in Go</title>
    <link href="favicon.png" rel="icon" type="image/png"/>
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/>
    <style type="text/css">
        @import url(https://fonts.googleapis.com/css?family=Yanone+Kaffeesatz);
        @import url(https://fonts.googleapis.com/css?family=Droid+Serif:400,700,400italic);
        @import url(https://fonts.googleapis.com/css?family=Ubuntu+Mono:400,700,400italic);

        body { font-family: 'Droid Serif'; }
        h1, h2, h3 {
            font-family: 'Yanone Kaffeesatz';
            font-weight: normal;
        }
        .remark-code, .remark-inline-code { font-family: 'Ubuntu Mono'; }
    </style>
</head>
<body>
<textarea id="source">

class: center, middle

# Exploring Code Generation with Templating in Go

👨‍💻 __Christian Bargmann (Github: cbrgm)__

🐦 @chrisbargmann

📧 chris@cbrgm.net

---

## What are we going to cover?

* Understand what code generation is
* Examine the basic parts of a code generator
* Look at some (un)popular code generators
* See how we can add code generation to our dev workflow
* Write a simple code generator

...and hopefully we can do all this in 20 min! :D 

---

## So what's code generation?

> `Code generation` is the process of generating code


> `Code generators` are entities that generate code

---

## What is a code generator?

> `Code generation` is the process of generating code


> `Code generators` are entities that generate code

### You are a code generator!

* You are very creative
* You are a smart problem solver
* No seriously, you really rock producing code!
* but you are slow... and bad at repetitive tasks!

---

## What is a code generator?

> `Code generation` is the process of generating code


> `Code generators` are entities that generate code

### Programs are code generators!

* Computers are able to write computer programs
* (2022) not yet as creative as humans, suffer problem solving abilities
* fast! Good at doing repetitive tasks!

---

## Cool, but how does code generation look like?

```text
┌─────┐    ┌─────────┐    ┌──────┐
│INPUT├───►│GENERATOR├───►│OUTPUT│
└─────┘    └─────────┘    └──────┘
```

### Input

* Code, Metadata, Environment Variables, Configuration Files, None...

### GENERATOR

* ??? ¯\_(ツ)_/¯

### OUTPUT

* Go, JSON, XML, YAML, Python, Clojure, Kotlin, Ruby, Java, PNG, GIF, MP4, ...

---

## But today we will focus on go-based code generation

```text
┌─────┐    ┌─────────┐    ┌──────┐
│GO   ├───►│GENERATOR├───►│GO    │
└─────┘    └─────────┘    └──────┘
```

### Input

* Go

### Generator

* An executable written in Go

### Output

* Go

---

## Examples of code generators

* go test

```text
- In: `*_test.go` Go source files
- Out: `package`main` test program source file
```

* [openapi-generator](https://github.com/OpenAPITools/openapi-generator)

```text
- In: OpenAPI Spec (v2, v3)
- Out: API client libraries (SDK generation), server stubs, documentation and configuration
```

* [Protobuf compiler](https://developers.google.com/protocol-buffers/)

```text
- In: *.proto definition files
- Out: API client libraries (SDK generation), server stubs to be used with gRPC (https://grpc.io/)
```

---

## Examples of code generators

* [hugo](https://gohugo.io/)

```text
- In: Go Template Files
- Out: Static website files (html,css,js, ...)
```

* [helm](https://helm.sh/)

```text
- In: Template Files aka "Helm Charts"
- Out: Kubernetes Manifests / YAML
```

* [stringer](https://golang.org/x/tools/cmd/stringer)

```text
- In: type declarations in your Go code
- Out: A given `String()` method for those type declarations

More details in a second! :D 
```

---

## Let's do a stringer deep-dive:

* [stringer](https://golang.org/x/tools/cmd/stringer)

> Stringer is a tool to automate the creation of methods that satisfy the fmt.Stringer interface. Given the name of a (signed or unsigned) integer type T that has constants defined, stringer will create a new self-contained Go source file implementing:

```go
func (t T) String() string
```

TLDR:

* Input: `stringer` takes the name of an integer (`int`) type `T` that has constants defined as __input__
* Output: A Go source file implementing the function signature above

---

## stringer: input

```go
package main

import (
	"fmt"
)

type Genre int

const (
	horror Genre = iota
	fantasy
	crime
	drama
	comedy
)

func main() {
	fmt.Printf("Let's watch a %v movie today!", horror)
}
```

Output:

```bash
go run genre.go
Let's watch a 0 movie today!
```

---

## stringer: input

How can be satisfy the `fmt.Stringer` interface?

```go
func (g Genre) String() string {
    // todo(cbrgm): implement me :(
}
```

```go
// String returns the iota's string representation
func (g Genre) String() string {
	switch g {
	case horror:
		return "horror"
	case comedy:
		return "comedy"
    // pretty repetitive huh...?
	default:
		return ""
	}
}
```

Output:

```bash
go run genre.go
Let's watch a horror movie today!
```

---

## stringer: code generator

Let's generate it!

```
go install  golang.org/x/tools/cmd/stringer
stringer -type=Genre

genre_string.go // I'm generated! :D 
genre.go
```

---

## stringer: output

```go
// Code generated by "stringer -type=Genre"; DO NOT EDIT.

package main

import "strconv"

func _() {
	// An "invalid array index" compiler error signifies that the constant values have changed.
	// Re-run the stringer command to generate them again.
	var x [1]struct{}
	_ = x[horror-0]
	_ = x[fantasy-1]
	_ = x[crime-2]
	_ = x[drama-3]
	_ = x[comedy-4]
}

const _Genre_name = "horrorfantasycrimedramacomedy"

var _Genre_index = [...]uint8{0, 6, 13, 18, 23, 29}

func (i Genre) String() string {
	if i < 0 || i >= Genre(len(_Genre_index)-1) {
		return "Genre(" + strconv.FormatInt(int64(i), 10) + ")"
	}
	return _Genre_name[_Genre_index[i]:_Genre_index[i+1]]
}
```

---

## stringer: output

```go
package main

import (
	"fmt"
)

type Genre int

const (
	horror Genre = iota
	fantasy
	crime
	drama
	comedy
)


func main() {
	fmt.Printf("Let's watch a %v movie today!", horror)
}
```

Output:

```bash
go run .
Let's watch a horror movie today!
```

---

## Go generate directive

> Cool, but do I always have to run stringer by hand?

No! `go generate` helps us to automate the process of running code generators by scanning comments in your Go source code and find commands to execute

```go
//go:generate stringer -type=Genre
```
We can run `go generate` with a list of packages like this:

```bash
go generate ./...
```

Check out `go generate -h`

```bash
usage: go generate [-run regexp] [-n] [-v] [-x] [build flags] [file.go... | packages]
Run 'go help generate' for details.
```

Gotchas:
* `go generate` is not part of `go build`!

---

## stringer: output

```go
package main

import (
	"fmt"
)

type Genre int

const (
	horror Genre = iota
	fantasy
	crime
	drama
	comedy
)

//go:generate stringer -type=Genre
func main() {
	fmt.Printf("Let's watch a %v movie today!", horror)
}
```

Output:

```bash
go run .
Let's watch a horror movie today!
```

---

## Let's build a simple code generator

Recap:
* We know what code generation is
* We know about some (un)popular code generators
* We learned about the `go:generate` directive and how we can add it to our dev workflow
* We just saw a simple code generator (`stringer`) in action

... now it's time to build our own code generator!

---


## Let's build a simple code generator

Let's stick with the `stringer` example for now.

```go
// String returns the iota's string representation
func (g Genre) String() string {
	switch g {
	case horror:
		return "horror"
	case comedy:
		return "comedy"
    // pretty repetitive huh...?
	default:
		return ""
	}
}
```

---

## Let's build a simple code generator

Our code generator will:
* take the name `Type` of an integer type `T` as a CLI argument
* Gather all constant declarations of type `T` in the source package
* Create a new file `gen_<typename>_string.go` which defines the `String()` func

---

## Build a code generator: implementation

> take the name `Type` of an integer type `T` as a CLI argument

```go
// Our Codegenerator is a Go program, so let's start with a main func
func main() {
	// the name of the type we're looking for
	typeName := os.Args[1]
	// constants definitions
	constantNames := []string{}
	// ...
```

---

## Build a code generator: implementation

> Gather all constant declarations of type `T` in the source package

Go has powerful packages to analyze Tokens, (Abstract) Syntax Tree (AST), Objects and much more

* [go/types](https://pkg.go.dev/go/types)
* [go/ast](https://pkg.go.dev/go/ast)
* [go/token](https://pkg.go.dev/go/token)

as well as wrappers around them:

see [go/packages](https://pkg.go.dev/golang.org/x/tools@v0.1.12/go/packages/gopackages)

---

## Build a code generator: implementation

> Gather all constant declarations of type `T` in the source package

see [packages](https://pkg.go.dev/golang.org/x/tools@v0.1.12/go/packages/gopackages)

```go
    //  load types, syntax information, ... of the current package
    cfg := &packages.Config{
		Mode: packages.NeedTypes | packages.NeedTypesInfo |
			packages.NeedSyntax | packages.NeedName,
	}
	pkgs, err := packages.Load(cfg)
	if err != nil {
		panic(err)
	}
	if len(pkgs) != 1 {
		panic(fmt.Errorf("got unexpected number of packages %v", len(pkgs)))
	}
	pkg := pkgs[0]
```

---

## Build a code generator: implementation

```go
    // check whether the target Type (for example "Genre") is present
    targetType := pkg.Types.Scope().Lookup(typeName)
	if targetType == nil {
		panic(fmt.Errorf("failed to find type declaration for %v", typeName))
	}
    
	// lookup all constants of the target type (e.g. "horror", "drama", ...)
	for _, file := range pkg.Syntax {
		for _, decl := range file.Decls {
			gd, ok := decl.(*ast.GenDecl) // keywords type, const or var
			if !ok {
				continue
			}
			if gd.Tok != token.CONST {
				continue
			}
			for _, spec := range gd.Specs {
				spec := spec.(*ast.ValueSpec)
				for _, name := range spec.Names {
					if pkg.TypesInfo.Defs[name].Type() == targetType.Type() {
						constantNames = append(constantNames, name.Name) // FOUND! :D 
					}
				}
			}
		}
```

---

## Build a code generator: implementation

```go
// define our output template
const outputTmpl = `package {{.PackageName}}
import "fmt"

// THIS FILE IS GENERATED, DO NOT EDIT MANUALLY

func (v {{.Type}}) String() string {
	switch v {
		{{range .DeclaredConstants -}}
	case {{.}}:
		return "{{.}}"
	{{end -}}
	default:
		panic(fmt.Errorf("unknown {{.Type}} value %d", v))
	}
}`
```

* [text/template](https://pkg.go.dev/text/template)
* Package template implements data-driven templates for generating textual output.
* Provides Loops, Conditions, Parameters, ...

---

## Build a code generator: implementation

```go
// create the output file
outputFileName := fmt.Sprintf("gen_%v_string.go", strings.ToLower(typeName))
outputFile, err := os.Create(outputFileName)
if err != nil {
	panic(err)
}

// prepare the values passed to the template, execute the template
tmpl := template.Must(template.New("out").Parse(outputTmpl))
if err := tmpl.Execute(outputFile, struct {
	PackageName       string
	Type              string
	DeclaredConstants []string
}{
	PackageName:       pkg.Name,
	Type:              typeName,
	DeclaredConstants: constantNames, // set the constants slice :D
	}); err != nil {
	panic(err)
}

if err := outputFile.Close(); err != nil {
	panic(err)
}
```

---

## Build a code generator: compile!

```go
go build -o generator genres-example-1/gen
```

add `go:generate` directive to `genre.go`

```go
//go:generate ./generator Genre
func main() {
	fmt.Printf("Let's watch a %v movie today!", horror)
}
```

and run `go generate .`

---

## Build a code generator: Execute!

Content of `gen_genre_string.go`:

```go
package main

// THIS FILE IS GENERATED, DO NOT EDIT MANUALLY

import "fmt"

func (v Genre) String() string {
	switch v {
		case comedy:
		return "comedy"
	case crime:
		return "crime"
	case drama:
		return "drama"
	case fantasy:
		return "fantasy"
	case horror:
		return "horror"
	default:
		panic(fmt.Errorf("unknown Genre value %d", v))
	}
}
```

Output:

```bash
go run .
Let's watch a horror movie today!
```

---

## Build a code generator: Improvements!

Format the generated Go code with `gofmt`

```go
// format formats a template using gofmt.
func format(in io.Reader) (io.Reader, error) {
	var out bytes.Buffer

	gofmt := exec.Command("gofmt", "-s")
	gofmt.Stdin = in
	gofmt.Stdout = &out
	gofmt.Stderr = os.Stderr
	err := gofmt.Run()
	return &out, err
}
```

---

## Build a code generator: Improvements!

```go
// ExecuteTemplate renders the named template and writes to io.Writer wr.
func ExecuteTemplate(file string, tmpl string, data interface{}) error {
	wr := os.Stdout
	if output := file; output != "" {
		wri, err := os.Create(output)
		if err != nil {
			return err
		}
		wr = wri
		defer wr.Close()
	}

	buf := new(bytes.Buffer)

	t, err := template.New("").Parse(tmpl)
	if err != nil {
		return err
	}

	err = t.ExecuteTemplate(buf, "", data)
	if err != nil {
		return err
	}

	src, err := format(buf) // format the buffered template
	if err != nil {
		return err
	}
	_, err = io.Copy(wr, src)
	return err
}
```

---

# Let's wrap it up!

---

## Let's wrap it up!

We...
* Understood what code generation is
* Examined the basic parts of a code generator
* Looked at some (un)popular code generators e.g. `stringer`
* Seen how we can add code generation to our dev workflow `go generate`
* Wrote a simple code generator
  * Not well-tested, lacks proper error handling (missing types, ...)
  * not performant
  * not the prettiest code

---

## Where to go from here? 

* All code is available [here](https://github.com/cbrgm/codegen-example)
	* https://github.com/cbrgm/codegen-example
* There's a great Article about `stringer` by Rob Pike [here](https://go.dev/blog/generate)
* Check out Paul Jolly's Talk at GopherCon UK 2019 [here](https://www.youtube.com/watch?v=xcpboZZy-64)
* Tokenizing, ASTs and more theory: Crafting Interpreters by Nystrom, Robert, ISBN: 9780990582939
* Check out [cbrgm/githubevents](https://github.com/cbrgm/githubevents)
  * Generated Go Library for Github Webhook Events (Feel free to contribute :D!!)

...and also, my team @[MOIA](https://www.moia.io/de-DE) is hiring :) Approach me if you're interested!

---

# Thank you!

👨‍💻 __Christian Bargmann (cbrgm)__

🐦 @chrisbargmann

📧 chris@cbrgm.net


    </textarea>
<script src="https://remarkjs.com/downloads/remark-latest.min.js" type="text/javascript">
</script>
<script type="text/javascript">
    var slideshow = remark.create({ ratio: "16:9" });

</script>
</body>
</html>