Repository navigation
Replies: 6 comments
Revision 1:
|
| Before the reset | With this proposal |
|---|---|
rest.Run(ctx, rest.Title(…), rest.Version(…), rest.Handle(route)) |
httpserver.Main(Init), with Init calling openapi.NewAPI(s, openapi.Info{…}, openapi.CatchAll(…), ops) and httpserver.NewService |
bedrockrest.Build(opts...), a bedrock.Builder[http.Handler] |
openapi.NewAPI, an *openapi.API and an error |
Title, Version, APIDescription; defaults "API" and "0.0.0" |
openapi.Info; Title and Version required |
SpecPath, default /openapi.json |
openapi.Document(path), an operation; no document unless the operations include one |
rest.GET("/carts/{cartId}", h) … DELETE, a path template |
openapi.Get(cart, h, parts...) … Delete, with cart := openapi.Root().Lit("carts").Param(cartID) |
cartID.Read(ep) for a path parameter |
nothing: the path declares it |
p.Read(ep) for a query or header parameter |
openapi.Query(p) or openapi.Header(p), a part |
Summary(s, ep), EndpointDescription, Tags, OperationID, MarkDeprecated |
openapi.Summary(s), Description, Tags, OperationID, Deprecated(), as parts |
PathParam[T any](name, opts...), an unsupported T failing every request with 400 |
openapi.PathParam[T ParamValue](name), a compile error, with constraints as methods |
Pattern(string), a bad expression failing every request |
.Pattern(*regexp.Regexp) |
ParamFrom returns the zero value for a parameter the endpoint didn't read |
it panics, naming the parameter |
chi, chi.URLParam |
chi inside the API, read with openapi.ParamFrom; spans named after the route |
ReadJSON[T] |
the handler's body type: JSON unless it is EmptyBody or a Stream; plus 415, 413 and trailing data |
ReadBinary(ct), WriteBinary(status, ct) |
openapi.Stream[M], with M naming the media type |
ReadFormFile, ReadFormField |
dropped |
WriteJSON[Resp](status), written with the encoder's error ignored |
the handler's response type, and openapi.Status; encoded before the status is written |
a 204 with WriteBinary(204, "", …) and an empty reader |
an EmptyBody response, 204 by default |
ErrorJSON[E](status, ep) on every route |
openapi.Error[E](status), given to an operation or to NewAPI |
CatchAll[E](status, wrap, ep) on every route |
openapi.CatchAll(status, wrap), given to NewAPI, and to an operation that overrides it |
a bad parameter or body: {"param", "error"} and a 400 outside the document |
an openapi.Problem and a documented 400, 413 or 415 |
| chi's plain-text 404 and 405 | openapi.Problems, with Allow on the 405 |
| an error the catch-all answered: lost | an ERROR record and a span event |
a panic: bedrock's RecoverPanics around the runtime |
the catch-all, recorded |
otelhttp.NewHandler(h, "rest") on the globals, spans named rest |
httpserver's instrumentation |
example/rest/shoppingcart in the root module; os.Getenv("DATABASE_URL") and pgxpool.New in main |
the example module; Init sources the URL, the pool dials through the Host, and OnStop closes it |
the example's specs/ (a hand-written api.yaml, sequence diagrams) |
dropped; the served document is the golden file |
The changes outside openapi are this proposal's stories, as in the post. The docs page is called "Serving an API with OpenAPI", and the "Serving HTTP" page's Migration from rest.Run section points at it.
Open questions
Questions 1–3 are as in the post. Question 4 is new:
- Can the end user replace the 404 and 405 answers? As proposed, the package answers both with a
Problem, and nothing replaces them. AnAPIOptionfor each, such asopenapi.NotFound(h http.Handler), would let an API answer in another format, or serve a page. The cost is an API whose 404 isn't a problem detail while its other errors are. Recommendation: no option. Every answer the API gives is then aProblem. An end user who wants a different answer for paths outside the API mounts the API on a mux of their own, at the paths it serves.
Out of scope
"Problem detail bodies for ServeMux's 404 and 405" is removed: they are in scope now. These are added:
- chi's middleware, sub-routers and regular-expression parameters. chi is how the API routes, and isn't part of its API.
- Path parameters that match more than one segment, such as chi's
*, and paths with a trailing slash.
References
These are added:
- chi v5.5.0:
Mux.NotFound,Mux.MethodNotAllowedandMux.Find - The Go spec's method declarations, for the type parameter of
Path.Param, a generic method
The ServeMux patterns link is removed.
Revision 2: open question 2 answered:
|
| Before the reset | With this proposal |
|---|---|
| an OpenAPI 3.0.3 document | an OpenAPI 3.2.0 document, from github.com/z5labs/humus/openapi/v3; following the latest 3.x |
Open questions
Question 2 is answered above. Questions 1, 3 and 4 stand.
Out of scope
These are added:
- Serving a 3.0 or 3.1 document, or choosing which minor version the document is.
- A package for a later major version of OpenAPI, which would be
openapi/v4.
References
The OpenAPI 3.0.3 and 3.1.0 links are replaced:
- OpenAPI 3.2.0, and its published schema-base
- Go modules: major version suffixes
Revision 3: every body names its codec, and humus assumes noneRevision 1 fixed a body's encoding by the handler's types:
This revision replaces it. Three designs were prototyped, and each compiles and serves the shoppingcart:
Each one served the same API in JSON, in XML from a codec in a package of its own, and in streamed CSV. The typed codecs are chosen. They are the best from a types perspective. And because every operation names the content types it reads and writes, the API's code documents itself. Design decisions says why the other two weren't chosen. What changes:
Questions 1, 3 and 4 are still open. Each section below replaces the part of the same name, as revisions 1 and 2 left it. The parts of a section that aren't mentioned stand. SummaryThe bullet "An operation is one call" becomes:
Proposed APIThese replace revision 1's operations, // Decoder reads request bodies of type T in one media type.
type Decoder[T any] interface {
// MediaType is the body's media type, such as application/xml. A request
// whose Content-Type names another media type, parameters aside, is
// answered with a 415.
MediaType() string
// Schema describes T's bodies in the document. NewAPI calls it once,
// and returns its error: a codec that can't handle T says so here.
Schema() (Schema, error)
// Decode reads r's body. An error that wraps *http.MaxBytesError is
// answered with a 413, and any other with a 400 whose detail is the
// error's text. A decoder that streams returns a T that reads the body
// as the handler reads it.
Decode(r *http.Request) (T, error)
}
// Encoder writes bodies of type T in one media type.
type Encoder[T any] interface {
// MediaType is the body's media type. "" is no body: the document lists
// no content, and nothing is written.
MediaType() string
// Schema is as Decoder's.
Schema() (Schema, error)
// Encode prepares v to be written, before the status is. Its error is
// answered by the catch-all. The API then writes the status and calls
// the WriterTo. A failure there is the client's: the response is cut
// short. An encoder that buffers returns its encoded bytes, such as a
// *bytes.Reader. An encoder that streams returns a WriterTo that writes
// as it goes.
Encode(v T) (io.WriterTo, error)
}
// Schema is a body's JSON Schema, as JSON, as the document lists it under
// the body's media type. A nil Schema lists the media type alone, as for
// raw bytes.
type Schema []byte
// SchemaOf returns T's JSON Schema, reflected from its fields, with each
// property named after the struct tag tag, such as "json" or "xml". A codec
// whose bodies are T's fields calls it from its Schema method.
func SchemaOf[T any](tag string) (Schema, error)
// EmptyBody is the body of a request that has none, and the response of a
// handler that writes none.
type EmptyBody struct{}
// EmptyDecoder and EmptyEncoder are EmptyBody's codecs: no body is read or
// written. A response with no body is a 204, unless Status says otherwise.
func EmptyDecoder() Decoder[EmptyBody]
func EmptyEncoder() Encoder[EmptyBody]
// The operations. Each takes a decoder for its request body (Post, Put and
// Patch) and an encoder for its response, as functions, so a generic codec
// function is passed uninstantiated and its type argument inferred from the
// handler. Get and Delete read no body.
func Get[Resp any](p Path, h func(context.Context, Request[EmptyBody]) (Resp, error), resp func() Encoder[Resp], opts ...Option) Operation
func Delete[Resp any](p Path, h func(context.Context, Request[EmptyBody]) (Resp, error), resp func() Encoder[Resp], opts ...Option) Operation
func Post[B, Resp any](p Path, h func(context.Context, Request[B]) (Resp, error), body func() Decoder[B], resp func() Encoder[Resp], opts ...Option) Operation
func Put[B, Resp any](p Path, h func(context.Context, Request[B]) (Resp, error), body func() Decoder[B], resp func() Encoder[Resp], opts ...Option) Operation
func Patch[B, Resp any](p Path, h func(context.Context, Request[B]) (Resp, error), body func() Decoder[B], resp func() Encoder[Resp], opts ...Option) Operation
// Error answers an error that matches E, with errors.As, with status and E,
// written by enc. Given to NewAPI, it answers on every operation, and every
// operation documents it. An operation's own Error for E overrides it.
func Error[E error](status int, enc func() Encoder[E]) APIOption
// CatchAll answers an error that no Error matched, and a panic of the
// handler, with status and E, written by enc: the error itself when it
// matches E, and wrap(err) otherwise. NewAPI requires one. An operation's
// own CatchAll overrides it.
func CatchAll[E error](status int, wrap func(error) E, enc func() Encoder[E]) Fallback
// Problems gives the encoder of the package's own problem responses:
// - the 400 for a parameter or body that doesn't decode;
// - the 413 and the 415;
// - the 404 and 405 of a path or method the API doesn't serve.
//
// NewAPI requires one. Given to an operation, it replaces the API's for that
// operation's 400, 413 and 415.
func Problems(enc func() Encoder[Problem]) ProblemEncoding
// ProblemEncoding is the encoder Problems gives. It is an Option, as a
// Fallback is.
type ProblemEncoding interface{ Option /* and unexported */ }
// NewAPI returns an API that serves ops. problems encodes the package's own
// problem responses. catchAll is the catch-all of every operation that
// doesn't give its own. opts are given to every operation. The rest is
// revision 1's.
func NewAPI(s *humus.Setup, info Info, problems ProblemEncoding, catchAll Fallback, ops []Operation, opts ...APIOption) (*API, error)
// Problem is a problem detail, RFC 9457, in the encoding of the encoder
// Problems gives. Its tags name its members as RFC 9457 does, in both of the
// serializations the RFC defines, JSON and XML. A codec of another encoding
// names them itself.
type Problem struct {
XMLName xml.Name `json:"-" xml:"urn:ietf:rfc:7807 problem"`
Type string `json:"type,omitempty" xml:"type,omitempty"`
Title string `json:"title,omitempty" xml:"title,omitempty"`
Status int `json:"status,omitempty" xml:"status,omitempty"`
Detail string `json:"detail,omitempty" xml:"detail,omitempty"`
Instance string `json:"instance,omitempty" xml:"instance,omitempty"`
}And the shipped JSON codec: package jsoncodec // import "github.com/z5labs/humus/openapi/v3/jsoncodec"
// Decoder and Encoder are T's codecs as application/json. Problem is T's
// encoder as application/problem+json, for openapi.Problem and the types
// that embed it. Pass them uninstantiated:
//
// openapi.Post(items, c.addCartItem, jsoncodec.Decoder, jsoncodec.Encoder)
func Decoder[T any]() openapi.Decoder[T]
func Encoder[T any]() openapi.Encoder[T]
func Problem[T any]() openapi.Encoder[T]
The core module's new dependencies change from swaggest/openapi-go to these two:
In the end user's
|
| Mistake | Result |
|---|---|
| a body or response with no codec | compile error: not enough arguments |
a decoder and an encoder swapped, such as jsoncodec.Encoder, jsoncodec.Decoder |
compile error |
a decoder given to Error, CatchAll or Problems |
compile error |
a non-generic codec of another type, such as a protocodec.Cart for an AddItemRequest |
compile error |
a constrained generic codec given a type outside its constraint, such as a protobuf codec's T proto.Message |
compile error |
a decoder given to Get or Delete |
compile error |
Error(404, jsoncodec.Problem) without [E] |
compile error: cannot infer E |
no Problems or no catch-all for NewAPI |
compile error |
a generic codec that can't handle its type, such as jsoncodec given a channel or an io.Reader |
NewAPI error, from the codec's Schema |
| two encodings, or two types, at one status | NewAPI error, naming both |
a nil codec function |
NewAPI error |
| a request with another Content-Type | a 415, in the operation's Problems encoding |
| an error value that fails to encode | the catch-all answers it. If its own encoder fails too, the status alone is sent |
a generic codec whose Schema doesn't check its type |
found on a request |
| a codec that says it streams but buffers | not checkable: it's the codec's contract |
Errors
Revision 1's model stands, with three changes:
- Every answer names its encoder:
Error[E](status, enc),CatchAll(status, wrap, enc), andProblems(enc)for the package's own problems. - Embedding
Problemdecides nothing. The encoder decides:jsoncodec.Problemwritesapplication/problem+json, andjsoncodec.Encoderwritesapplication/json. - The package's own problems are
NewAPI's second argument, as the catch-all is its third. So an API can't be built without both.
The document
The package writes the document itself. Under each body's media type, it lists the schema the codec's Schema returns:
SchemaOf's reflection, named by the codec's own struct tag, such asxml, with the codec's additions such as OpenAPI'sxmlobject;- a description, such as a protobuf message's name;
- nothing, for raw bytes.
swaggest/openapi-go's operation reflector isn't used. It assumes every body is JSON: it writes any other request body as {"type":"string"}, and reads only json tags. SchemaOf reflects with swaggest/jsonschema-go.
A schema is inline rather than in components, because one Go type can have a different schema under each codec. The 3.2 validation of revision 2 covers every media type the tests generate, XML and CSV included.
Testing
These are added:
- Each mistake in the table above.
- The compile errors have files behind a build tag, which a test builds to check that each one fails.
- The
NewAPIerrors are table cases.
- Openness. A codec in a test package of its own, using only
openapi's exported API, serves XML and problem+xml. - Streaming, with the two tests above that deadlock if a body is buffered, and a stream that fails mid-way cutting its response short.
jsoncodec:- trailing data and a wrong type are 400s;
- an unknown field is ignored;
- a body over
http.MaxBytesHandler's limit is a 413; - a channel, a function or an
io.Readertype is aNewAPIerror.
Migration from today
These rows replace revision 1's:
| Before the reset | With this proposal |
|---|---|
ReadJSON[T] |
jsoncodec.Decoder, given to the operation |
WriteJSON[Resp](status) |
jsoncodec.Encoder, and openapi.Status |
ReadBinary(ct), WriteBinary(status, ct) |
a decoder or encoder of io.Reader in that media type, the end user's own |
ErrorJSON[E](status, ep) |
openapi.Error[E](status, jsoncodec.Problem), or jsoncodec.Encoder for application/json |
CatchAll[E](status, wrap, ep) |
openapi.CatchAll(status, wrap, jsoncodec.Problem) |
a bad parameter or body: {"param", "error"} |
a Problem, in the encoding Problems gives |
Design decisions
Revision 1's "The handler's types fix the encodings" is replaced:
- Every body names its codec, and nothing has a default. humus doesn't know what an API's bodies are, so it doesn't guess. Naming the content types on every operation makes the API's code document itself. The cost is that a JSON-only API names
jsoncodecon every operation. - Typed codecs, over the two other designs:
- Codec values on
any, given once toNewAPI(Consumes,Produces,ProducesErrors) and overridden per operation. They read the shortest. But a codec that can't handle its type is found only byNewAPI, and only when the codec checks, never by the compiler. - Body types with methods, such as
json.Unmarshaler's. These make a body with no encoding a compile error. But they cost three methods on every request and response type, or a wrapper with a.Valuefield in every handler. They also give a Go type one encoding unless it is wrapped, and an embedded type's methods silently drop the fields added next to it.
- Codec values on
DecoderandEncoderare separate. With oneCodecinterface for both directions, two generic codecs swapped between the body and response positions compiled. And a codec that only writes, given for a request body, failed every request with a 400. With two, both mistakes are compile errors (checked).- Codecs are passed as functions, so a generic codec needs no type argument. Passing a generic function uninstantiated is a less common Go idiom, and the compiler's messages name both types. A Go 1.27.0 compiler bug crashes when a generic function from another package is given to a struct-literal field. Function arguments compile. So operations take codecs only as arguments, and the docs page says why.
jsoncodecships, opt-in. JSON will be used everywhere, and its details are easy to get wrong. It is a package of its own, named on each operation like any third party's codec, andopenapinever uses it.Problemhasjsonandxmltags, because RFC 9457 defines exactly those two serializations. The tags name members; they choose no encoding.- The package writes the document. The reflector it used assumed JSON.
Out of scope
This is added:
- More than one media type for a body, chosen by
Content-TypeorAccept(content negotiation). An operation's body has one codec. A second media type is a second operation.
References
These are added:
- swaggest/jsonschema-go, for
SchemaOf - RFC 9457 §6 and Appendix B:
application/problem+json,application/problem+xml, and the XML form
The swaggest/openapi-go reference is removed.
Revision 4: a response that fails after its status is loggedRevision 3 says that once the status is written, an error from the encoder's
Those failures are the server's. Revision 3 recorded them on the span alone, and with the default telemetry, which exports no spans, they would be lost. This revision gives them the catch-all's rule. Trailers are left out. They would let a response report a failure after its body, as gRPC reports a stream's status. They are revisited when a user asks for them. Until then, a failure after the status can only cut the response short. Questions 1, 3 and 4 are still open. Each section below replaces or adds to the part of the same name, as revision 3 left it. Proposed API
// Encode prepares v to be written, before the status is. Its error is
// answered by the catch-all. The API then writes the status and calls
// the WriterTo, which may do more work once it has written the body,
// such as recording that a download completed. An error from the
// WriterTo cuts the response short, and is the server's unless the
// request's context is done. An encoder that buffers returns its
// encoded bytes, such as a *bytes.Reader. An encoder that streams
// returns a WriterTo that writes as it goes.
Encode(v T) (io.WriterTo, error)Requests and responsesThe bullet "An encoder that streams" becomes:
For example, an object copied from S3, then recorded in a database: // download is what the handler returns: what to copy, and what to do once
// it has been copied.
type download struct {
body io.ReadCloser // the S3 object's body
done func() error // runs after the last byte is written
}
func (f Files) getFile(ctx context.Context, req openapi.Request[openapi.EmptyBody]) (download, error) {
key := openapi.ParamFrom(req, fileKey)
obj, err := f.S3.GetObject(ctx, &s3.GetObjectInput{Bucket: &f.Bucket, Key: &key})
if err != nil {
return download{}, err // no status yet: answered by Error or CatchAll
}
return download{body: obj.Body, done: func() error {
return f.DB.MarkDownloaded(ctx, key) // ctx is still the request's
}}, nil
}
func objectStream() openapi.Encoder[download] { return downloadEncoder{} }
type downloadEncoder struct{}
func (downloadEncoder) MediaType() string { return "application/octet-stream" }
func (downloadEncoder) Schema() (openapi.Schema, error) { return nil, nil }
func (downloadEncoder) Encode(d download) (io.WriterTo, error) { return d, nil }
func (d download) WriteTo(w io.Writer) (int64, error) {
defer d.body.Close()
n, err := io.Copy(w, d.body)
if err != nil {
return n, err // S3 failed partway: response failed, and the response is cut short
}
return n, d.done() // a failed update: the same
}
openapi.Get(file, f.getFile, objectStream, openapi.OperationID("getFile"))The status is written before the first byte of the body, as HTTP requires and as TelemetryThis row is added to the table of records:
TestingThese are added:
Design decisionsThis is added:
Out of scopeThis is added:
|
Revision 5: the open questions answeredThe three questions still open:
With that, nothing in The proposal has no open questions left. Each section below replaces or adds to the part of the same name, as revisions 1 to 4 left it. Proposed APIThese replace revision 3's // Unmatched answers the requests no operation matches. Its answers aren't
// in the document. NewAPI requires both handlers.
type Unmatched struct {
// NotFound answers a request whose path no operation serves.
NotFound http.Handler
// MethodNotAllowed answers a request whose path an operation serves,
// but not with the request's method. The API sets the Allow header,
// listing the path's methods, before it calls MethodNotAllowed.
MethodNotAllowed http.Handler
}
// RequestError is a request an operation rejected before its handler ran.
type RequestError struct {
// Status is:
// - 400 for a parameter or a body that doesn't decode or validate;
// - 413 for a body over http.MaxBytesHandler's limit;
// - 415 for a Content-Type the operation's decoder doesn't read, or
// none.
Status int
// Param names the parameter that doesn't decode, for a 400 about one,
// and is "" otherwise.
Param string
// Err is why: the parameter's error or the decoder's.
Err error
}
func (e *RequestError) Error() string
func (e *RequestError) Unwrap() error
// RequestErrors answers every RequestError with its Status and with
// wrap's value, written by enc. Every operation documents it: at 400 when
// the operation reads a parameter or a body, and at 413 and 415 when it
// reads a body. NewAPI requires one. Given to an operation, it replaces
// the API's for that operation.
func RequestErrors[E any](wrap func(*RequestError) E, enc func() Encoder[E]) RequestErrorAnswer
// RequestErrorAnswer is what RequestErrors returns. It is an Option, as a
// Fallback is.
type RequestErrorAnswer interface{ Option /* and unexported */ }
// NewAPI returns an API that serves ops:
// - unmatched answers the requests no operation matches;
// - requestErrors answers the requests an operation rejects;
// - catchAll is the catch-all of every operation that doesn't give its
// own;
// - opts are given to every operation.
//
// A nil handler in unmatched is a NewAPI error. The rest is as revision 1
// describes it.
func NewAPI(s *humus.Setup, info Info, unmatched Unmatched, requestErrors RequestErrorAnswer, catchAll Fallback, ops []Operation, opts ...APIOption) (*API, error)
And the opt-in package: package problem // import "github.com/z5labs/humus/openapi/v3/problem"
// Details is a problem detail, RFC 9457. Its tags name its members as RFC
// 9457 does, in both of the serializations the RFC defines: JSON and XML.
// A type that embeds it is a problem detail of its own.
type Details struct {
XMLName xml.Name `json:"-" xml:"urn:ietf:rfc:7807 problem"`
Type string `json:"type,omitempty" xml:"type,omitempty"`
Title string `json:"title,omitempty" xml:"title,omitempty"`
Status int `json:"status,omitempty" xml:"status,omitempty"`
Detail string `json:"detail,omitempty" xml:"detail,omitempty"`
Instance string `json:"instance,omitempty" xml:"instance,omitempty"`
}
// Error returns Detail, or Title when Detail is empty.
func (d Details) Error() string
// FromRequestError returns e as a problem detail: type about:blank, the
// name of e's status as its title, and e's error's text as its detail. It is
// a wrap for openapi.RequestErrors.
func FromRequestError(e *openapi.RequestError) Details
// Handler answers every request with status and a problem detail, written
// by enc. The detail's type is about:blank and its title is the status's
// name. It is a handler for openapi.Unmatched.
func Handler(status int, enc func() openapi.Encoder[Details]) http.Handler
In the end user's
|
| Mistake | Result |
|---|---|
no RequestErrors or no catch-all for NewAPI |
compile error |
a nil NotFound or MethodNotAllowed |
NewAPI error |
| a request with another Content-Type | a 415, answered by the operation's RequestErrors |
Errors
The bullet "The package's own problems are NewAPI's second argument" is replaced:
- The package's own rejections are
*RequestErrors, answered byRequestErrors,NewAPI's fourth argument. The requests no operation matches are answered byUnmatched, its third. So an API can't be built without an answer to each.
Panics
"This is open question 1" and the sentence after it are replaced. The rule covers the operation's codecs as well as its handler:
- A panic of the handler, or of the operation's decoder in
Decodeor encoder inEncode, is answered by the catch-all. All of these run before the status is written. It is recorded as revision 1 records a panic: an exception event with the stack, the span's status set to Error, and one ERROR record. The record is renamedoperation panicked, since it now covers the codecs too. - A panic in the encoder's
WriterTo, after the status, cuts the response short, as its error does in revision 4, and is recorded the same way. - A panic while answering can't be answered by the catch-all again. This covers a panic in a
wrap, or in the encoder of anError, aCatchAllorRequestErrors.httpserverrecords it and aborts the response, as for anyhttp.Handler. So doUnmatched's handlers, which aren't operations. - A handler that panics with
http.ErrAbortHandleris passed through untouched, as before.
The cost stays grpcserver's: whatever the handler left half-changed stays that way, for the requests that follow.
Telemetry
The record handler panicked is renamed operation panicked. It is emitted when the handler, or one of the operation's codecs, panicked.
Testing
These are added:
Unmatched:- each handler is called, the
MethodNotAllowedone withAllowalready set; - a nil
NotFoundorMethodNotAllowedis aNewAPIerror; - the package writes and logs nothing of its own for either.
- each handler is called, the
RequestErrors:- its
Eis documented at 400, 413 and 415 on the operations that can answer with them; - an operation's own replaces the API's;
- its
wrapgets theRequestErrorwith itsStatus, itsParamand the decoder's error.
- its
- Panics:
- a panic in a decoder's
Decodeor an encoder'sEncodeis answered by the catch-all, with oneoperation panickedrecord; - a panic in a
WriterTocuts the response short; - a panic in a
wrapis aborted byhttpserver.
- a panic in a decoder's
problem:FromRequestErrornames the status and carries the error's text;Handlerwrites its status and detail through the encoder it is given;Detailsencodes as RFC 9457's JSON and XML forms, the XML in theurn:ietf:rfc:7807namespace.
Migration from today
These rows replace revision 3's:
| Before the reset | With this proposal |
|---|---|
a bad parameter or body: {"param", "error"} |
a *openapi.RequestError, answered by RequestErrors, such as RequestErrors(problem.FromRequestError, jsoncodec.Problem) |
| chi's plain-text 404 and 405 | the end user's openapi.Unmatched handlers, such as problem.Handler |
| the example's own problem type | problem.Details, embedded |
Design decisions
Revision 1's "A panic is answered by the catch-all" is replaced, and these are added:
- A panic is answered by the catch-all, of the handler and of the operation's codecs. The API keeps the contract its document states.
- The end user answers every error, the 404 and 405 included.
openapiforces no error format, so it has no answer of its own to give. Both handlers are required, so no request is answered by something the end user didn't choose. Unmatchedis a struct, so its two handlers are named and can't be swapped. The cost: a nil field is aNewAPIerror, not a compile error.- Problem details are a package of their own. Nothing in
openapimakes an API use them. Likejsoncodec,problemis opt-in, built onopenapi's exported API, and named where it is used.
Open questions
None.
Closing: the OpenAPI endpoint DSL has landedThe revisions' stories are done:
Revision 5 left no open questions. The package is The rest is out of scope or belongs to other proposals, as "Out of scope" and its revisions list:
|
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Important
Amended by revision 1. The package is
openapi, andNewAPIreturns an*APIthat is anhttp.Handler. Paths are built from typed values rather than templates, an operation is one call with its parts, and the router is chi. It adds a fourth open question.Revision 2 answers question 2. The package is
openapi/v3, and its document follows the latest minor version of OpenAPI 3: 3.2.0 today.Revision 3 replaces how bodies are encoded. Every body names a typed codec, a
Decoder[T]or anEncoder[T], on its operation, and nothing is assumed: no JSON default and no closed set of encodings. humus ships an opt-inopenapi/v3/jsoncodec.Revision 4 logs a response that fails after its status: an ERROR record, and the response cut short. Trailers are out of scope until a user asks for them.
Revision 5 answers questions 1, 3 and 4. A panic is answered by the catch-all, and the example keeps Postgres. The end user answers 404 and 405 with a required
openapi.Unmatched, and the package's own rejections withopenapi.RequestErrors. Problem details move to an opt-inopenapi/v3/problem. No open questions remain.Note
Part of the humus re-evaluation; see Proposal: humus on the bedrock process model, wave 4. It builds on Proposal: HTTP server foundation, which has landed as
httpserverandhttpservertest, and on wave 1. It depends on no other wave-4 proposal. It replaces the REST half ofrest.Runand bedrock'sruntime/http/restDSL, and owns the rebuiltexample/rest/shoppingcart. It blocks Proposal: authentication and authorization (wave 5), which adds security to REST's operations and to its document, and the wave-6 walkthroughs and AI tooling, which are written against this API.Summary
A new package,
rest, in the core module, builds anhttp.Handlerfrom typed endpoints and serves the API's OpenAPI document. It is the DSL that bedrock v0.21.0 shipped asruntime/http/restand that humus'srest.Runwrapped before the reset, kept where it works and changed where it breaks the process model or humus's principles:http.Handler, and its Service ishttpserver's. The end user'sInitcallsrest.NewHandler, hands the handler tohttpserver.NewService, andmainishttpserver.Main. REST adds no Service, noMainand no test package. It inherits TLS, HTTP/2, the drain, the grace period,otelhttpand the server's settings, and its tests usehttpservertest.GET,POST,PUT,PATCHandDELETEtake a typed handler, then parameters, a body reader, a response writer and typed errors are layered on, andCatchAllcompletes it as aRoute. Only aRoutecan be served, so every operation has an answer for every error.net/http'sServeMux, not chi. Path parameters come fromr.PathValue, andServeMuxsetsRequest.Pattern, sohttpserver's spans are named after the route, such asGET /carts/{cartId}.NewHandlerreturns an error for a path parameter that the path and the endpoint disagree on, a response type that doesn't match the handler's, two bodies for one status, and the other mistakes the old DSL found only at request time, or never.ParamFrompanics for a parameter the endpoint didn't read, instead of returning a zero value.application/problem+json, and they are in the document. A panic of the end user's handler is answered by the route's catch-all, and recorded.examplenested module, as the reference service: itsInitsources its settings from theProcess, opens its Postgres pool through theHost, and registers the pool's close withOnStop.Background
runtime/http/restin bedrock v0.21.0 is the inside-out DSL:Endpoint,Route,Request[B], typedParam[T]s,ReadJSON,WriteJSON,ErrorJSON,CatchAll, andBuild, which returns abedrock.Builder[http.Handler]. bedrock v0.22.0 removed it with the rest of the Builder/Runtime layer. humus'srest.Runwrapped it in the server #680 replaced. story(rest): own the OpenAPI endpoint DSL inside humus #620 planned to copy the DSL into humus unchanged, and was closed for this re-evaluation.addcartitem.godeclares each operation in one function, and its document is generated from the same declarations that decode the request. This proposal keeps that.Buildreturns abedrock.Builder, which is gone. The handler logs nothing and records nothing: an error answered by the catch-all disappears, since the example's catch-all drops it. AndReadFormFilecallsr.FormFile, which writes a large upload to a temp file withos.CreateTemp, on the real filesystem rather than theHost's.ParamFromreturns the zero value for a parameter the endpoint never read.Patterndrops a regular expression that doesn't compile, and then every request fails with 400.WriteJSON[Resp]documentsRespwhatever the handler returns.json.NewEncoder(w).Encode(v), its error ignored, after the status is written, so a value that can't be encoded is a 200 with a broken body.ErrorJSONs at one status overwrite each other in the document.writeValidationErrorwith{"param": …, "error": …}and a 400 that no operation documents. The example's own errors are problem details, which it defined itself, and v0.15.0'sresthad aProblemDetailtype (story(rest): Implement RFC 7807 Problem Details for error responses #385).API.Description, andSummary,EndpointDescription,Tags,OperationIDandMarkDeprecatedon an operation, cover what it asked for.http.ServeMuxmatches methods and{name}wildcards, and since Go 1.23 it setsRequest.Pattern, whichhttpserver's instrumentation names spans after.main.main.goreadsos.Getenv("DATABASE_URL"), opens apgxpoolon the real network and closes it withdefer, aroundrest.Run. The charter lists exactly this as a question the process model raises.http.Handler, and its Service is this Service serving that handler. … So REST inherits TLS, HTTP/2, the instrumentation and the panic handling. Its services usehttpserver.Mainandhttpservertest, unless REST's proposal gives it aMainof its own." And: "A protocol that defines an error response, such as REST, recovers in its own handler and writes that response itself."Proposed API
The package is in the core module, as the charter places it. Its one new dependency is swaggest/openapi-go (see The document below). chi is not a dependency.
In the end user's
InitThe shoppingcart's
Init, in full apart from its settings:And one of its operations, which changes from before the reset only in its imports and its not-found type:
resthas noSettings. What an operator changes is the server's, inhttpserver.Settings, and the document's title and version are the end user's code.Routing
NewHandlerregisters each route on a newhttp.ServeMux, asMETHOD path, and returns the mux.ServeMuxreads the same way. A path starts with/, and its parameters are{name}, where the name is a Go identifier, asServeMuxrequires.NewHandlerrejects{name...}and{$}, which OpenAPI has no form for, and a host. A path that ends in/is registered with{$}after it, so/carts/matches/carts/and not every path below it, as OpenAPI means.{name}in the path has aPathParamthe endpoint reads, and everyPathParamit reads is in the path. Before, a parameter missing from either was missing from the document, or read as"".ServeMuxdoes.GET /carts/{id}next toGET /carts/{cartId}matches the same requests, andServeMux.Handlepanics with a message naming both. That is a programming error, reported by the first test or start, asgrpcserverpanics on a service registered twice. OpenAPI forbids the same pair too.ServeMux's answers: 404 for an unknown path, and 405 with anAllowheader for a known path and an unknown method. They are plain text and not in the document.ServeMuxsetsRequest.Patternon the request it is given, andhttpserver'sotelhttpnames the server span from it once the handler returns:POST /carts/{cartId}/items. Before the reset, every span was namedrest.http.Handler. An end user who serves something REST doesn't describe mounts it next to REST's handler on a mux of their own, withmux.Handle("/", api). Those routes aren't in the document.Requests
Parameters. A
Param[T]is declared once, usually as a package-level value, and read by each endpoint that takes it.Tis a string, bool, int, int64 or float64 type, named types such astype Sort stringincluded, so an unsupported type is a compile error. When a request arrives, each parameter the endpoint reads is decoded and validated before the handler runs:r.PathValue, and are always required;An optional parameter that is absent gets its
DefaultValue, orT's zero value. A value that doesn't parse asT, or breaks one of its constraints, is answered with a 400Problemwhosedetailnames the parameter, and the handler isn't called.NewHandlerchecks each parameter's options against its type:Pattern,MinLengthandMaxLengthneed a string type,MinimumandMaximuma number type,Enum,DefaultValueandParamExamplevalues of the parameter's own type, andOptionalisn't allowed on a path parameter.Patterntakes a compiled*regexp.Regexp, so a bad expression is the end user'sregexp.MustCompilepanicking, not every request failing.ParamFrompanics for a parameter the endpoint didn't read. The handler'sRequestholds a value for every parameter its endpoint reads, present or not, so a missing one is a mistake in the endpoint's declaration. Returning the zero value, as before, turned the mistake into a wrong answer.JSON bodies.
ReadJSON[T]decodes the body intoT, before the handler runs:Content-Typeother thanapplication/jsonor a+jsontype is a 415Problem;Problemwith the decoder's message;encoding/jsondoes, so a client that sends a newer field doesn't break;http.MaxBytesHandler, and a body over the limit is a 413Problem.REST doesn't validate a body against its schema: the schema documents the body, and the handler checks what it needs.
Binary bodies.
ReadBinaryhands the handler the request body as anio.Reader, unread, and documents it ascontentType.Responses
WriteJSONencodes before it writes. The response is marshalled into a buffer, and the status and body are written only once that succeeds. A valueencoding/jsoncan't encode is an error, answered by the catch-all, rather than a 200 with a broken body.WriteBinarywrites the status andcontentType, copies the handler'sio.Reader, and closes it if it is anio.Closer.WriteEmptywrites the status and no body, for a 204. Before, the example wrote a 204 withWriteBinary, an emptyContent-Typeand an empty reader, and the document described a binary body.NewHandlerreturns an error whenWriteJSON[Resp]'sRespisn't the handler's response type,WriteBinary's isn't anio.Reader,WriteEmpty's isn'tEmptyBody, orReadJSON[T]'sTisn't the handler's body type. An endpoint with no writer is an error too. Before, a mismatch documented one type and served another.Errors
An endpoint answers its handler's error with the first
ErrorJSONwhose type matches, witherrors.As, and otherwise with its catch-all. That is the old DSL's model, kept as it was, with three changes.REST's own errors are problem details. Before the handler runs, REST answers with a
Problem, asapplication/problem+json, withtypeabout:blankand the status's name astitle, as RFC 9457 says forabout:blank:http.MaxBytesHandler's limitContent-TypeThe content type follows the error's type.
ErrorJSONandCatchAllwrite a type that isProblem, or embeds it, asapplication/problem+json, and any other type asapplication/json. The end user's errors become problem details by embeddingProblem, asNotFounddoes above.One response per status. An operation has one response for each status in the document.
NewHandlerreturns an error for an operation that names a status twice, with different types, or that names 400, 413 or 415, where REST answers with aProblemitself, with a type other thanProblem. A handler that answers its own validation errors with 400 does so withErrorJSON[rest.Problem](http.StatusBadRequest, ep), and the document has one 400 response.The catch-all records what it answers. An error that only the catch-all answered is one the end user didn't plan for. REST records it on the request's span as an exception event, and emits one ERROR record,
request failed, in the request's context, so the record carries the span's trace and span IDs, withhttp.request.method,http.route,error.typeandexception.message. With the default telemetry, which exports no spans, that record is where the error is found. When the request's context is done because the client went away, the error is recorded on the span only.wrapdecides what the client sees, and should not put the error's text in the body: the record has it. The example'sinternalErrorreturns a fixedProblem.Panics
The charter leaves a panic inside one request to each protocol, and #680 leaves REST's to REST. A REST handler never writes to the
http.ResponseWriter: it returns a value that REST encodes. So when it panics, nothing has been written, and REST can answer cleanly:httpserver, which records it and aborts the response.httpserverrecords one: an exception event on the request's span withexception.type,exception.messageandexception.stacktrace, the span's status set to Error, and one ERROR record,handler panicked, in the request's context, under the loggergithub.com/z5labs/humus/rest.wrapof an error that doesn't carry the panic's value. So the response is the one the document lists for the catch-all, and the panic's value never reaches the client.http.ErrAbortHandleris passed through untouched, so it can still abort its response on purpose.This is open question 1. The cost is
grpcserver's: whatever the handler left half-changed stays that way, for the requests that follow.The document
NewHandlerbuilds the document with swaggest/openapi-go'sopenapi3reflector, which generates the JSON Schema of each Go type from its fields andjsontags. Each operation lists its parameters with their constraints, its request body, required, its success response, itsErrorJSONresponses, REST's own problem responses from the table above, and its catch-all. The document is encoded to JSON once, so every request for it gets the same bytes, and a golden test can compare them.openapi31reflector takes the same operation contexts, so the choice changes one reflector.API.OpenAPIPath, withGET, asapplication/json. The end user names the path, so no path is imposed. An empty path serves no document, for an API that keeps its document private.swaggest/openapi-go is the one new dependency, with its own
swaggest/jsonschema-goandswaggest/refl. It was used for the same purpose by humus v0.15.0'srestand by bedrock's DSL. Writing JSON Schema from Go types by hand is most of what a reflector does, and keeping it in step with OpenAPI is not humus's business. Its modern API,NewOperationContextandAddRespStructurewith a content type and a status, replaces the deprecatedSetJSONResponsethe old DSL used, and is what writesapplication/problem+jsonresponses. chi was the old DSL's other dependency, andServeMuxreplaces it.Telemetry
REST takes everything from
s.Telemetryand never touches OTel's globals. It adds no spans or metrics:httpserver'sotelhttpgives every request a server span named after its route, and the HTTP server metrics with the response's status. REST's records go throughs.Telemetry.LoggerProvider.Logger("github.com/z5labs/humus/rest"):request failedhandler panickedA 400, 413 or 415 is the client's mistake: it is answered and not logged, and
otelhttprecords its status. An error answered by anErrorJSONis an answer the API documents, and is not logged either.What waves 5 and 6 build on this
SummaryandTagsdo, so it adds its own, and its stories add whatever it needs here. REST answers no 401 or 403 of its own until then.httpserver's handler chain, afterotelhttp's extraction and before REST's handler, so REST needs nothing for it.rest.NewApiAPI to this one.Design decisions
rest, in the core module. The charter puts REST in the core, and its one dependency is a schema reflector. It doesn't importhttpserver: its handler is anhttp.Handler, and the end user'sInithands it tohttpserver.NewService.Main, no Service and no test package. A REST service is an HTTP service: its drain, grace period and certificate arehttpserver's, and a test of it ishttpservertest.Startandhttpservertest.NewClient. Arest.Mainwould behttpserver.Mainunder another name.NewHandlertakes theSetup, as every humus constructor does, for its logger. The handler is still built and served inInit, from whatInitbuilds.ServeMuxover chi. It does what REST needs, it names the spans, and it is one dependency fewer.NewHandler, or a compile error, never a 400 on every request.ServeMux's own conflict panic stays a panic, asgrpcserver's registration mistakes do.Problemis exported. It is the type of the responses REST writes itself, so it is in REST's API and its document either way, and the end user embeds it rather than copying it, as the old example did.ProblemDetailsErrorHandler, no error handler interface. v0.15.0's per-operation error handlers andHttpResponseWritererrors are replaced byErrorJSONandCatchAll, which the document is generated from.http.MaxBytesHandleralready is one, and REST answers its error with 413.ReadFormFilewrote large uploads to the real temp directory, and nothing that uses the DSL needs it.Testing
These tests are sketches and have not been run. Every test runs inside a
testing/synctestbubble on a humustestHost, with a certificate fromhumustest.NewCA, and asserts withtestify/require, except the example's container tests below.A whole service, through
httpservertest, with its telemetry recorded:requireJSON,spanNames,cart,notFoundandinternalErrorare the test file's own helpers and types.The tests also show the following:
NewHandler's errors. One table case for each mistake it reports: a missing title or version, a path with{name...},{$}or a host, a{name}with noPathParamand aPathParamnot in the path, each type mismatch, each option on the wrong type,Optionalon a path parameter, an endpoint with no writer, two types at one status, a non-Problemtype at 400. Two bad routes are reported in one error.ServeMux's message, naming both patterns.ParamFrompanics for a parameter the endpoint didn't read, naming it.Problemnaming the parameter, with the handler not called; an absent optional parameter is its default; a repeated query parameter is a 400.text/plain, a 400 for an empty body, malformed JSON, a wrong type and trailing data, a 413 behindhttp.MaxBytesHandler, and an unknown field ignored.WriteEmptywrites a 204 with no body and noContent-Type;WriteBinarycloses the handler's reader.wrap's value and produces onerequest failedrecord carrying the request span'strace_idandspan_id, withhttp.routeandexception.message; anErrorJSONerror and a 400 produce no record; an error returned after the client went away is on the span and not logged.handler panickedrecord carries its IDs; nothing is written top.Stderr(); the next request succeeds. A handler panicking withhttp.ErrAbortHandlerhas its response aborted, ashttpserverdoes.OpenAPIPathwithapplication/json, isn't served with an empty path, is the same bytes on every request, and lists REST's 400, 413 and 415 problem responses on the operations that read a parameter or a body.The package is held to the 80% coverage gate, and its example is an
Examplefunction.humustest.TestServiceisn't run: REST adds no Service.The shoppingcart
The example is the reference service, in the
examplenested module,github.com/z5labs/humus/example, as the charter's module layout has it, so pgx, uuid and testcontainers stay out of the core module's graph. Itsgo.modreplacesgithub.com/z5labs/humuswith../, asgrpc's does. CI already lists every module withgo-modules.sh, and already exempts the packages underexample/from the coverage gate.main.goishttpserver.Main(app.Init, humus.WithTelemetry(app.BuildTelemetry)). Nothing in it reads the environment.Initsourceshttpserver.Settings, with the certificate as a PKCS#12 bundle read throughs.Host.FS, andDATABASE_URL, froms.Process.Environment, withconfig.Struct, so a missing variable exits 78. It builds the pool withpgxpool.NewWithConfig, withDialFuncset tos.Host.Dial, andLookupFuncreturning the host unresolved so that theHost's dial resolves it. It registerspool.Closewiths.OnStop, and builds the handler from the six routes of before: create, get and delete a cart, and add, update and remove an item.pgconn.ParseConfigreads the real environment'sPG*variables, and the real~/.pgpass, for whatever the URL leaves out, and pgx has no way to build a config without it. The example's URL sets every field it uses, and the docs page lists the leak, as the lifecycle page's ambient leaks do for others.app_test.goruns the whole service withhttpservertest.Start, on an in-memoryStorethat an unexported seam ofInitswaps in, since Postgres can't run on humustest'sHost. It sends the requests of each endpoint throughhttpservertest.NewClient, and compares the served document with a golden file,testdata/openapi.json.testcontainersbuild tag asqueue/kafka's are, run the PostgresStoreagainstpostgresin a container, and the whole service once onhumus.OSHost(), with the schema applied fromschema.sql. They run in a CI job of their own, which is not a required check, and the Build job's vet step vets them, as it does Kafka's.This is open question 3.
Migration from today
rest.Run(ctx, rest.Title(…), rest.Version(…), rest.Handle(route))httpserver.Main(Init), withInitcallingrest.NewHandler(s, rest.API{…}, routes...)andhttpserver.NewServicebedrockrest.Build(opts...), abedrock.Builder[http.Handler]rest.NewHandler(s, api, routes...), anhttp.Handlerand an errorTitle,Version,APIDescription,SpecPath; defaults"API","0.0.0",/openapi.jsonrest.API's fields;TitleandVersionrequired; no path unlessOpenAPIPathnames oneroute.Route()NewHandlerGET…DELETE,Request,EmptyBody,Summary,Tags,OperationID, …restParamFromreturns the zero value for a parameter the endpoint didn't readPathParam[T any], an unsupportedTfailing every request with 400PathParam[T ParamValue], a compile errorPattern(string), a bad expression failing every requestPattern(*regexp.Regexp)chi.URLParamhttp.ServeMux,r.PathValue; spans named after the routeReadJSON[T]ReadFormFile,ReadFormFieldWriteJSON,WriteBinary, written with the encoder's error ignoredWriteBinary(204, "", …)and an empty readerWriteEmpty(http.StatusNoContent, …)ErrorJSON[E],CatchAll[E], asapplication/jsonapplication/problem+jsonwhenEembedsrest.Problem{"param", "error"}and a 400 outside the documentrest.Problemand a documented 400, 413 or 415RecoverPanicsaround the runtimeotelhttp.NewHandler(h, "rest")on the globals, spans namedresthttpserver's instrumentationexample/rest/shoppingcartin the root module;os.Getenv("DATABASE_URL")andpgxpool.Newinmainexamplemodule;Initsources the URL, the pool dials through theHost, andOnStopcloses itspecs/(a hand-writtenapi.yaml, sequence diagrams)The changes outside
restare this proposal's stories:examplemodule and the shoppingcart, the container tests' CI job, and the vet step;TestDocs_code_samplesrequires;Migration from rest.Runsection of the "Serving HTTP" page, which now points at REST's page;doc.go, the README and the docs site's index.Open questions
httpserverrecords it and aborts the response, as for anyhttp.Handler? Aborting matches Proposal: HTTP server foundation #680's decision for plain HTTP, but a REST client then sees a reset stream where the document promises a status, and nothing has been written yet that answering would contradict. Recommendation: the catch-all, asgrpcserveranswers a panic withInternal.type: [string, "null"]for a pointer. swaggest generates either. Recommendation: 3.0.3.Storeforapp_test.goand container tests in a CI job of their own, as proposed? Or an in-memory store only, which runs anywhere but shows no dependency built inInit? Recommendation: Postgres. The reference service is where the end user sees a real dependency built inInit, dialled through theHostand released withOnStop.Out of scope
http.Handlerthat composes with whatever that proposal provides.httpserver's chain.rest.NewApiAPI: Proposal: walkthroughs and a humus MIGRATING.md (wave 6). Generating REST code: Proposal: AI tooling (wave 6).http.Handlermounted next to REST's handler, outside the document.ServeMux's 404 and 405.References
httpserver.Main,httpservertest, and "What waves 4 and 5 build on this"httpserver's package doc and panic layerruntime/http/rest, the DSLrest/rest.go,example/rest/shoppingcart; story(rest): own the OpenAPI endpoint DSL inside humus #620, story(rest): add options for setting additional openapi values #283, story(rest): Implement RFC 7807 Problem Details for error responses #385net/http'sServeMuxpatterns, andhttp.MaxBytesHandlerAll reactions