Suggestions for structuring/organisation in larger projects

[tanordheim]

Hi!

I'm trying to use oapi-codegen to generate server stubs for our API here but I'm struggling to find a way to structure and organise my generated code in a way that feels somewhat idiomatic-ish Go. My usecase is that the application I'm working on is structured into packages by feature (call it what you want; features, bounded contexts, modules - it's a bit besides the point). My HTTP handlers and models are defined in those packages, and I have a global main-entrypoint that wires up all the handlers across the system and spins up a HTTP server with those endpoints routed up.

Now, I'm trying to introduce oapi-codegen to this while keeping that structure intact. I've noticed that it doesn't support external $ref (#42) but that --import-mapping is added (#204) to allow having the models generated in a different package than the handlers to partially work around this issue. My desired structure is something like this:

cmd/
  api/
    main.go
    openapi-server.gen.go
feat1/
  openapi.yaml
  openapi-types.gen.go
feat2/
  openapi.yaml
  openapi-types.gen.go

Ideally I'd want the API endpoints exposed by each feature to also be defined in openapi.yaml within the package, and have all of those merged together when generating the server stubs in cmd/api/. I currently have a proof of concept working of this using a mix of yq merge and swagger-cli bundle to preprocess the files before running oapi-codegen but the whole thing is feeling very hacky and brittle. Has anyone else attempted something similar and found a solution to this that might work? I'd be happy to contribute an example project to the repository if I could find a solution I'd be happy with, any input would be appreciated.

[deepmap-marcinr]

I've done something similar internally where we use oapi-codegen at DeepMap, where I merge individual swagger specifications into one larger one.

Some ideas might be to do partial generation based on Tags or provide some kind of filter argument for what paths to generate, I'm all ears to suggesions.

[tanordheim]

Tags would be good except you cannot put tags on paths; I'd ideally want paths related to feat1 and feat2 to be defined inside those specs. The way I solve this now is:

cmd/api/openapi.yaml

My main spec define all paths for the API as a whole - regardless of which context owns the implementation. Response models used by the paths are defined here and references the external spec from other contexts - like this (somewhat shortened):

paths:
  /foo:
    get:
      operationId: getFoo
      responses:
        "200":
          content:
            application/json:
              schema:
                $ref: "#/components/schemas/Feat1Response"
  /bar:
    get:
      operationId: getBar
      responses:
        "200":
          content:
            application/json:
              schema:
                $ref: "#/components/schemas/Feat2Response"

components:
  schemas:
    Feat1Response:
      $ref: "../../feat1/openapi.yaml#/components/schemas/Feat1Response"
    Feat2Response:
      $ref: "../../feat2/openapi.yaml#/components/schemas/Feat2Response"

feat[1-2]/openapi.yaml

My feature specific specs only contain models right now, like this:

components:
  schemas:
    Feat1Response:
      type: object
      properties:
        nested:
          $ref: "#/components/schemas/Feat1ResponseNested"
    Feat1ResponseNested:
      type: object
      properties:
        foo:
          type: string

Note that the cmd/api/openapi.yaml-spec only reference the one model from feat1/openapi.yaml it needs to reference the response model itself; none of the other models nested underneath it.

generating

I run generation in multiple steps; first I generate types for the features like this:

> oapi-codegen -generate types,skip-prune -package feat1 -o feat1/openapi-types.gen.go feat1/openapi.yaml
> oapi-codegen -generate types,skip-prune -package feat2 -o feat2/openapi-types.gen.go feat2/openapi.yaml

Then I run server generation for the main bundle like this:

> cat oapi-codegen.conf
output: cmd/api/openapi-server.gen.go
package: main
generate:
    - server
import-mapping:
    ../../feat1/openapi.yaml: github.com/myorg/myapp/feat1
    ../../feat2/openapi.yaml: github.com/myorg/myapp/feat1
> oapi-codegen -config=oapi-codegen.conf cmd/api/openapi.yaml

This generates the server interface and using import-mapping it will correctly reference the generated models in the feat1 and feat2 packages wherever models are referenced. The examples above will have no references in the generated server code here, but if you had things like parameter types and such it would correctly reference them.

Finally, I generate the spec so that I can use the validator middleware and such as oapi-codegen doesn't support $ref directly - I build a merged spec using swagger-cli and then generate spec based on that:

> swagger-cli bundle --dereference --type yaml --outfile cmd/api/openapi.merged.yaml cmd/api/openapi.yaml
> oapi-codegen -generate spec -package main -o cmd/api/openapi-spec.gen.go cmd/api/openapi.merged.yaml

composing server

I spin up the server like this (pseudo-code-ish):

package main

type feat1Server = feat1.Server
type feat2Server = feat2.Server

type server struct {
  *feat1Server
  *feat2Server
}

func main() {
  srv := &server{
    feat1Server: feat1.NewServer(), // feat1.Server defines GetFoo()
    feat2Server: feat2.NewServer(), // feat2.Server defines GetBar()
  }
  router := echo.New()
  RegisterHandlers(router, srv)
  router.Start(":31337")
}

All of this gets me "more or less" there; I'm not able to declare the paths for each feature inside the feature specific spec, but at least I can keep my models defined there. It still feels a bit hacky though, so I'm all ears for any input on how to improve this.

[tanordheim]

Some issue with this setup that I just encountered, is if a path has query string parameters. This causes the ServerInterface in cmd/api/ to reference a *Params-type for the handler which is not generated anywhere.

I can have it generated by adding generate: ["server", "types"] to oapi-codegen.conf - this causes the params type to correctly be created. It does however require that the feat1-package depends on the main-package in cmd/api in order to access that type which won't fly.

[oliverbenns]

Was there ever a solution to this? For a large API it's not really scalable to have all the handlers within a single package.

One way around this is to use type embedding on the server interface implementation type.

[nielskrijger]

@oliverbenns

I never found a proper solution. What I ended up doing was just to merge everything into one single schema right before anything else happens.

That way you always have one YAML schema containing everything, and you can structure the files in whichever way you want. I'm quite pleased with that approach and I'm no longer looking for something better/more official.

This is the code I used, it's been a while so I can't remember if I wrote it or just copied something from the web (knowing me, probably a combination of the two...):

package main

import (
	"io/fs"
	"log"
	"os"
	"path/filepath"
	"strings"

	"gopkg.in/yaml.v2"
)

func main() {
	fileContents := make([]string, 0)

	if err := filepath.WalkDir("./api", walk(&fileContents)); err != nil {
		log.Fatal(err)
	}

	result, err := mergeYamlValues(fileContents)
	if err != nil {
		log.Fatal(err)
	}

	if err = os.WriteFile("./api/api.gen.yaml", []byte(result), 0o644); err != nil {
		log.Fatal(err)
	}
}

func walk(contents *[]string) func(path string, d fs.DirEntry, err error) error {
	return func(path string, entry fs.DirEntry, err error) error {
		if err != nil {
			return err
		}

		// Exclude oapi codegen config file
		if entry.Name() == "config.yaml" {
			return nil
		}

		fileName := strings.ToLower(path)

		if strings.HasSuffix(fileName, "yaml") || strings.HasSuffix(fileName, "yml") {
			b, err := os.ReadFile(path)
			if err != nil {
				return err
			}

			*contents = append(*contents, string(b))
		}

		return nil
	}
}

func mergeYamlValues(values []string) (string, error) {
	var result map[any]any

	var bs []byte

	for _, value := range values {
		var override map[any]any

		bs = []byte(value)

		if err := yaml.Unmarshal(bs, &override); err != nil {
			return "", err
		}

		// check if is nil. This will only happen for the first value
		if result == nil {
			result = override
		} else {
			result = mergeMaps(result, override)
		}
	}

	bs, err := yaml.Marshal(result)
	if err != nil {
		return "", err
	}

	return string(bs), nil
}

func mergeMaps(a, b map[any]any) map[any]any {
	out := make(map[any]any, len(a))
	for k, v := range a {
		out[k] = v
	}

	for k, v := range b {
		if v, ok := v.(map[any]any); ok {
			if bv, ok := out[k]; ok {
				if bv, ok := bv.(map[any]any); ok {
					out[k] = mergeMaps(bv, v)
					continue
				}
			}
		}

		out[k] = v
	}

	return out
}

[eliofery]

You can solve it like this:

Folder structure:
Here each api is one big project.

api
├── marketing
│       ├── marketing.gen.yaml
│       └── marketing.openapi.yaml
└── subscription
        ├── subscription.gen.yaml
        └── subscription.openapi.yaml

File marketing.gen.yaml contents:

package: marketing

generate:
  std-http-server: true
  strict-server: true
  embedded-spec: true
  models: true
  client: true

output: ./pkg/api/marketing.gen.go
output-options:
  skip-prune: true
  nullable-type: true

File subscription.gen.yaml contents:

package: subscription

generate:
  std-http-server: true
  strict-server: true
  embedded-spec: true
  models: true
  client: true

output: ./pkg/api/subscription.gen.go
output-options:
  skip-prune: true
  nullable-type: true

Example create servers:

// SubscriptionServer
var _ desc.ServerInterface = (*SubscriptionServer)(nil)

type SubscriptionServer struct {
	desc.ServerInterfaceWrapper
}

func NewSubscriptionServer() *SubscriptionServer {
	return &SubscriptionServer{}
}

// MarketingStrictServer
var _ desc.StrictServerInterface = (*MarketingStrictServer)(nil)

type MarketingStrictServer struct {}

func NewMarketingStrictServer() *MarketingStrictServer {
	return &MarketingStrictServer{}
}

File main.go contents:

func main() {
    mux := http.NewServeMux()
    
    // Example standard mode
    subServer := NewSubscriptionServer()
    subscription.HandlerFromMux(subServer, mux)

    // Example strict mode
    markServer := NewMarketingStrictServer()
    m := marketing.NewStrictHandler(markServer, []marketing.StrictMiddlewareFunc{LoggingMiddleware})
    marketing.HandlerFromMux(m, mux)

    server := http.Server{
        Addr:              ":8080",
        Handler:           mux,
        ReadTimeout:       10 * time.Second,
        ReadHeaderTimeout: 10 * time.Second,
        WriteTimeout:      60 * time.Second,
        IdleTimeout:       60 * time.Second,
    }

    log.Println("Server is running on port 8080...")
    if err := server.ListenAndServe(); err != nil {
        log.Fatalf("failed to serve: %v", err)
    }
}

Makefile contents:

LOCAL_BIN=$(CURDIR)/bin

download-bin:
    GOBIN=$(LOCAL_BIN) go install github.com/deepmap/oapi-codegen/v2/cmd/oapi-codegen@v2.2.0

generate-api:
    # Marketing API
    mkdir -p ./pkg/api/marketing
    $(LOCAL_BIN)/oapi-codegen \
        -config ./api/marketing/marketing.gen.yaml \
        ./api/marketing/marketing.openapi.yaml
    
    # Subscription API
    mkdir -p ./pkg/api/subscription
    $(LOCAL_BIN)/oapi-codegen \
        -config ./api/subscription/subscription.gen.yaml \
        ./api/subscription/subscription.openapi.yaml