Generate OpenAPI documentation from EndpointSpec data type

[jdegoes]

Is your feature request related to a problem? Please describe.

The zio.http.api.EndpointSpec contains a declarative description of an API endpoint, while the ServiceSpec data type contains a declarative description of more than one API endpoints.

These declarative descriptions are wholly sufficient to generate extremely good OpenAPI documentation for the endpoints. Yet, this is a feature that is wholly missing in the current version of zio.http.api.

Describe the solution you'd like

Add a method EndpointSpec#toOpenAPI, which translates the route, query parameter, header, input and output, into the zio.http.api.openapi.* family of data types, incorporating the custom documentation that may be included into the EndpointSpec or into one of the HttpCodec values that make up its description of the input.

Describe alternatives you've considered

N/A

Additional context

This ticket does not include translating the OpenAPI values into HTML or JSON, which is considered outside the scope of this ticket. This ticket is merely to translate the EndpointSpec data type into typed OpenAPI values.

A separate ticket will cover the generation of JSON (etc.) from the OpenAPI data types.

[iusildra]

I'll try to do this one

[jdegoes]

A subproblem here is to merely translate HttpCodec into a fragment of OpenAPI.

This can be done by adding def toOpenAPI: zio.http.api.openapi.OpenAPIBase (or similar) to HttpCodec, and then implementing it on a case-by-case basis for all subtypes, by using the most relevant OpenAPI data types.

Work can be checked by, for example, inspecting the JSON output or converting it to PDF or other form intended for human consumption.

[wpoosanguansit]

Hi John, I hope you do not mind providing more detail on the ticket. OpenAPIBase is private so it is not accessible in EndpointSpec. I made it public but EndpointSpec is a case class. I got the error:

class EndpointSpec needs to be abstract. Missing implementation for:
[error] def toOpenApi: zio.http.api.openapi.OpenAPIBase

That would lead to generating the OpenApi as a complete object as:

openapi.OpenAPI.OpenAPI(
    openapi = "",
    info = Info(
      title = "",
      description = Doc.empty,
      termsOfService = URI.create(""),
      contact = None,
      license = None,
      version = "",
    ),
    servers = List.empty[Server],
    paths = Map.empty[Path, PathItem],
    components = None,
    security = List.empty[SecurityRequirement],
    tags = List.empty[Tag],
    externalDocs = None,
  )

In this case, how do I go about extracting the data from EndpointSpec to fill up OpenAPI? It is pretty challenging to extract data from the generics in EndpointSpec[Input, Output].

On the HttpCodec side, I am looking at the most obvious to get started and I do see:

private[api] final case class Body[A](schema: Schema[A]) extends Atom[CodecType.Body, A]

which could be required to return OpenApi's

final case class RequestBody(description: Doc, content: Map[String, MediaType], required: Boolean = false)
extends ResponseOrReference

Can you point me to where I can find out how I can pull out Doc, Map[String, MediaType] and required flag from schema passed in? Thank you for your help.

[987Nabil]

@wpoosanguansit the OpenAPI data structures represent the full open api spec. And it is very likely that currently not all fields can be filled from the data present in EndpointSpec. So it might be, that doc is just Doc.empty.
If you like, we can maybe pair on this. I'll write to you in discord.

[987Nabil]

@jdegoes
@wpoosanguansit and I noticed some edges.

1. OpenApi references documentation of path parameters by the name of its placeholder. But zio http does not offer to explicitly name path params. One could generate names, but I guess this is not desirable by the user that generates API docs. I think we need some API to have named text codecs for path. So instead of EndpointSpec.get("users" / int).out[Int] maybe something like EndpointSpec.get("users" / int("users-per-page")).out[Int]. This is btw no problem to for query params and headers since they are named by default. Or the body since there is only one. So I think this is only a problem of Route. Maybe it just needs a name parameter.
2. Right now, we can add docs to every HttpCodec. So we can write something like EndpointSpec.get((("users" / uuid) ?? Doc.p("Customer UUID")) / int), but does this really makes sense? Why do you combine a literal and a placeholder/var and document this? And when you have two vars EndpointSpec.get(((int / uuid) ?? Doc.p("Whatever"))) and generate docs like open api, do you give the doc to the int or uuid param? Or both? I think we should allow docs only for single placeholders/vars or the entire EndpointSpec. So you can document the Atoms and the whole, but not any arbitrary combination of Atoms.

What do you think?

[fancellu]

FYI, I created some models for openAI
https://github.com/fancellu/openai-scala-models
Have used them from ZIO and Http4s (Look in the README)

[987Nabil]

@fancellu We have already a model for OpenAPI in zio-http at zio.http.api.openapi.OpenAPI. And if you actually suggest to use your lib, we want to keep the dependencies of zio-http as few as possible.

Also I think what needs improvement here is more likely the zio-http internal representation of endpoints. The OpenAPI model should be in an okay(ish) state.

[jdegoes]

/bounty $500

[algora-pbc]

💎 $500 bounty created by jdegoes
🙋 If you start working on this, comment /attempt #1498 to notify everyone
👉 To claim this bounty, submit a pull request that includes the text /claim #1498 somewhere in its body
📝 Before proceeding, please make sure you can receive payouts in your country
💵 Payment arrives in your account 2-5 days after the bounty is rewarded
💯 You keep 100% of the bounty award
🙏 Thank you for contributing to zio/zio-http!

Attempt	Started (GMT+0)	Solution
🔴 @kitlangton	Jul 29, 2023, 5:37:55 PM	WIP
🟢 @987Nabil		#2470

[987Nabil]

I am on it 🙂

[pjfanning]

@987Nabil I don't know if this helps but I maintain https://github.com/swagger-akka-http/swagger-scala-module

This lib supports OpenAPI3 and basically generates Open API models for Scala classes. I have some related libs for non-core Scala classes.

Even if this lib is not useful to you, I would be interested in looking at any alternative solution you use for generating Open API models - as I might be able to learn something that could be useful in swagger-scala-module.

[987Nabil]

Hey @pjfanning! Thanks for the hint. But I think this are two different approaches to the problem. We are actually not analysing the Scala data structures itself, but only a Schema of them based on the zio-schema lib. So we have only a reified version of the data structures. Therefore, the first version I am writing now, focus on the information that are easily accessible by these schemas and the endpoint type it self. And I will most likely make some assumptions and decisions for default behaviour in some cases, that a user might want to configure. But I don't expect that any kind of config will be available in the first impl.
Our goal is as well, to have Endpoint definitions that can not only generate OpenAPI, but also other documentation formats. Therefore I doubt, that zio-http will ever have OpenAPI specific annotations. Maybe we will endup having a config object that you can handover to the generator function. But I am not sure yet about it.

[pjfanning]

@987Nabil Sounds like you have a good idea of how to proceed. It would be nice to be able to provide a partial definition of the OpenAPI doc when requesting the full doc. The EndpointSpec can provide details to generate server details, API details, model details, etc. There is a fair amount of metadata you should include in an OpenAPI doc though - for instance, the 'Info' object in https://swagger.io/specification/.

[987Nabil]

@pjfanning We have already a domain model for the OpenAPI spec in zio-http. Including Info. I expect that you will be able to modify this object manually, if you want to, before you will hand this over to something that will generate for you the actual json/html that you might want to offer via an endpoint.

[fernanluyano]

Any updates on this?

[algora-pbc]

@kitlangton: Reminder that in 7 days the bounty will become up for grabs, so please submit a pull request before then 🙏

[algora-pbc]

💡 @987Nabil submitted a pull request that claims the bounty. You can visit your org dashboard to reward.

[algora-pbc]

🎉🎈 @987Nabil has been awarded $500! 🎈🎊