Swagger transport

[peterbourgon]

Investigate what's necessary to have first-class support for Swagger definitions.

[emicklei]

As I intend to finish https://github.com/emicklei/swagger2 , an upgrade from https://github.com/emicklei/go-restful/swagger, maybe I can help here.

[peterbourgon]

@emicklei Brilliant, thank you! I expect the actual mechanics of the transport will likely be a layer on top of transport/http.{Client,Server}, and it looks like the linked package could be a good fit there. Can you say a bit more about what you'd propose to do, concretely?

And — does it make sense to go further? From what I can see, this might be a good entrypoint for #70, as I expect parsing a Swagger 2.0 spec file and doing some codegen would be a lot more straightforward than (say) gRPC.

[emicklei]

The swagger2 package is now a separate repository to make it independent from go-restful (as the original should be have been too). In addition to the complete coverage of models as part of the 2.0 spec, I want to port all the logic and tests related to building models from Go structs ; this a valuable part of the original package. See also emicklei/go-restful#157 . I would like swagger2 to be dependent on std. pkgs only. go-restful integration will be part of its own package.

Next, I would like to study the go-kit package more closely and propose solutions for integration. For producing swagger specifications, http route information is needed as well as request and response types. For inspiration, have a look at https://github.com/emicklei/go-restful/blob/master/examples/restful-user-resource.go . I am not suggestion that this should be the way to integrate ; I think it should be more decoupled w.r.t to this design. Let's discus better alternatives.

Code generation is indeed an interesting use of these models, not just for Go but such tool exist for many other languages as well.

[emicklei]

Just a small update on the status.

The author of https://github.com/go-swagger contacted me about collaborating on his project. Based on that discussion, he recently reorganized his project into independent smaller packages such as https://github.com/go-openapi/spec which is exactly the model part of swagger2 (with a few other implementation decisions).

I will have to decide whether to continue swagger2 or drop it in favor of go-openapi. Comments are welcome.

[nelz9999]

I would definitely encourage taking advantage of the work and community around the go-openapi project. I think the community will benefit more from a united front. (go-swagger AND go-kit together - I :swoon:! ❤️ )

[peterbourgon]

While at GlueCon this year I had a number of interesting discussions with the OpenAPI (née Swagger) folks, which gave me a much better understanding of what OpenAPI actually is, and which problems it attempts to solve. They call it an API description language, and although it was originally intended to indeed describe a REST-y API, it's accrued lots of value-add bits and bobs that seem to make it more and more a source of authority: to proscribe an API.

The most interesting bit for us is, I think, the codegen facility. There was already a Go client codegen; I've filed swagger-api/swagger-codegen#2970 for a Go server codegen, and they've started swagger-api/swagger-codegen#2979 in response. Yay.

All that said, I'm not sure exactly what OpenAPI-as-Go-kit-transport would look like. I definitely like the idea of a OpenAPI spec as an ersatz IDL. But the codegen wouldn't define a Go interface that we can implement, in the way that Thrift or gRPC does. I can't figure out how transport/openapi would be much different than transport/http, to be honest.

Given an OpenAPI spec — like a .proto3 or .thrift spec — what would we want Go kit to do?

[casualjim]

for what it's worth you can provide custom templates for go-swagger the codegen tool.

https://github.com/go-swagger/go-swagger/

for a server generator you can provide the following templates: https://github.com/go-swagger/go-swagger/tree/master/generator/templates/server

it does take care of routing etc but I'd be interested in collaborating on a codegen that leverages go-kit.

The models is where most of the work has been going, and my runtimes are interfaces so they can be get different implementations too.

for the client there are these templates: https://github.com/go-swagger/go-swagger/tree/master/generator/templates/client

I'm in the GoBridge slack team as Ivan Porto Carrero, feel free to ping me I also have a channel there for go-swagger

[dobegor]

@emicklei sorry to bother you, but is there any progress or plans of getting it integrated into go-kit?
go-restful is nice and usable, but go-kit has some very well designed abstractions we use to integrate with other non-Go services well.

[emicklei]

@dobegor , I don't think an integration with swagger2 makes sense. The OpenAPI project is more about a contract-first approach whereas the go-restful package (swagger v1) integration is focused on producing the JSON api description from Route,Models and additional "annotations".

I think it would be worth trying to generate go-kit based Go code based on an OpenAPI description, leveraging the code-generator functionality of the project.

[edmund-troche]

Any updates on this effort? I'm more interested in the bottoms-up approach (generate Swagger/OpenAPI from code [Route, Models, etc]).

[krasi-georgiev]

Any updates on this effort? I'm more interested in the bottoms-up approach (generate Swagger/OpenAPI from code [Route, Models, etc]).

I also think that this makes much more sense.

Templates in 'go-swagger' gives you some freedom, but go kit propagates non opinionatedness so bottom up approach fits better with this model.

The most important benefit for me is when I don't have to explain and document how to use the service - just a link to http://petstore.swagger.io/ with the json spec as a parameter and that is it!

for example:
http://petstore.swagger.io/?url=http%3A%2F%2Flocalhost%3A8080%2Fswagger.json

other than that I much prefer if I can use go-kit for anything else.

bottoms-up approach should also allow using other transports I think

[paultyng]

I think the issue with generating a spec from code is that you would have to have a pretty complicated DSL to describe all the possible response status codes, params, authentication etc.

We originally tried a DSL in an internal REST API that we wanted to publish Swagger for, it used go-openapi/spec to build and generate the Swagger JSON from a supplied Go interface and additional supplied metadata. It very quickly became a maintenance nightmare and worse yet, it wasn't fully enforced at runtime as the DSL was almost entirely for generating the spec.

We've since taken an approach similar to test2doc where we actually generate Swagger from observed behavior in unit testing, the added benefit of this is that something has to be minimally tested to show up in the contracts and you can guarantee that there is some case where that endpoint will result in that status code and response data.

It's still a work in progress, and we still had to introduce a layer of REST "primitives" to allow for enough information to be collected to generate the Swagger, but so far it seems to be working out for us.

[peterbourgon]

OK, serious question, which informs decisionmaking, and which I still don't fully grok. Does Swagger (OpenAPI) intend to describe an existing codebase, and thus be generated from it? Or does it intend to define a specification that a codebase should implement?

[basvanbeek]

It started as a way to describe an existing restful API and still allows to do so but in time and rebranding to OpenAPI it tries to be an IDL too.

Taken from the website:

WHAT IS THE OPENAPI SPECIFICATION?

At the heart of the above tools is the OpenAPI Specification (formerly called the Swagger Specification). The specification creates the RESTful contract for your API, detailing all of its resources and operations in a human and machine readable format for easy development, discovery, and integration.

[peterbourgon]

it tries to be an IDL too

Ugh. Can't be all things to all people — gotta pick one. Am I off base here?

[basvanbeek]

No you're on point IMHO. With Go kit we have our service interface as our source of truth and we create our transport/api wrappers around it. Can't have multiple sources of truth.

I'm perfectly happy with Go kit service description being source of truth. If we want swagger support I think the most to be guaranteed would be a description generator / scaffolder which is part of kitgen.

[paultyng]

@peterbourgon I think you are on target as well. Swagger is also so coupled to HTTP semantics and go-kit is so unopinionated, it would be hard to do without either giving up some enforcement/adherence to spec in generated server code (ie. not limiting status codes or response types, limited JSON schema options) or significantly complicating the transport layer as go-kit just doesn't collect that much information about implementations.

JSON schema is also very complex, so you'd be better off integrating an external JSON schema package just to implement this except my understanding is that Swagger only borrows portions of that spec, not the entirety, and adjusts the definitions somewhat.

I'll admit I'm somewhat biased as my experience bridging the gaps between Go and Swagger has been problematic at best.

[alihalabyah]

Was there any progress here to enable the auto generated API documentation?

[krasi-georgiev]

haven't used it but seems that go-swagger already supports auto generation of a swagger.json spec form a go code by using annotations.
https://goswagger.io/faq/faq_spec.html

basically there is no strict binding between the spec and the code , but rather an auto generated swagger spec to document your api.

the only benefit I see is that it will make it less likely that your api docs and the implementation will drift, but still doesn't enforce it unlike the go-swagger client/server generator.

[alihalabyah]

@krasi-georgiev Can you provide an example? looks like this works only with other frameworks not Go Kit

[krasi-georgiev]

AFAIU this works with any go code. the tool just finds and parses the annotations and generates the spec from that.

I remember seeing some good examples , but can't remember where exactly so you need to do some digging.

[alihalabyah]

@krasi-georgiev Appreciate your support, I'll have a deeper look.

[ifraixedes]

@alihalabyah as @casualjim commented #185 (comment) go-swagger allows to define custom code generation templates, so you should be able to generate go-kit services if you create templates which produce a go-kit like application/service.