Provide OpenAPI/Swagger specification out of the box

[simonmit]

Feature request:
An Elasticsearch installation should provide a OpenAPI/Swagger specification without installing any additional plugins/configuration.

Describe the feature:
Currently (5.x) there is no detailed documentation of the REST API of Elasticsearch.
When coding against this API it is very "costly" to revese-engineer the JSON format of the requests and responses.

Example:

• https://www.elastic.co/guide/en/elasticsearch/reference/5.4/indices-get-index.html#indices-get-index (How does the resulting JSON look like?)
• https://www.elastic.co/guide/en/elasticsearch/reference/5.4/modules-snapshots.html#_snapshot (How to parse the resulting JSON to extract the state field and read its value)
• Errors (5xx - or 4xx errors are nicely formatted into a JSON, but how does it look exactly so it can be parsed? Always the same structure?)

A simple way to document the API would be a swagger spec (or OpenAPI nowadays): Usually provide an endpoint where you can find a swagger spec in YAML format.
https://github.com/OAI/OpenAPI-Specification/blob/master/versions/2.0.md
http://swagger.io/

Excpectations:

• Every node can provide a specification that it is currently running.
  • Endpoints
  • Query parameters
  • Request/Response JSON format
  • Status Codes
  • etc. (See Swagger/OpenAPI documentation)
• Provide documentation for all (JSON returning) endpoints that are shipped by default with ES (e.g. do not provide X-Pack specific endpoint documentation)
  • Maybe exclude _cat API which does not return JSON.
• Add the swagger URL to the documentation (e.g. with a concrete link to Swagger UI like http://petstore.swagger.io/?url=http://localhost:9000/_openapi/spec.yaml so I can browse the specification of my locally installed ES)

Remarks:
I did not find any issues mentioning Swagger/OpenAPI
I did not find any information about API documentation on the Elasticsearch Documentaion.

[javanna]

Relates to #24065 .

[jpountz]

Removing the discuss label since this is something that we eventually want.

[dgaedcke]

From reading this, it seems that it's not currently possible to:

• Acquire an OpenApiSpec for the ES (not ECE) Rest API
• Use that spec to generate a 3rd party client for ES

Am I reading that correctly?
Any tips or links on other ways to accomplish the above?

[stephanebastian]

Has anyone on the ES team started to work on providing an OpenApi/Swagger specification?
If no, I can work on it and try to provide a first draft of the ES OpenApi spec.
Would you guys be interested ?

[lcawl]

@stephanebastian I've been looking at the documentation side (i.e. generating asciidoc from the swagger output), so I'm happy to help from that perspective.

[stephanebastian]

@lcawl Thx for your help.
Out of curiosity, where in ES are you generating asciidoc from a swagger spec?
Also do you know which version of swagger do you use ? is it 2.0 (OpenApi 3.0) ?

[lcawl]

@stephanebastian At this point I think the only documentation that is generated using the swagger specification is for Cloud. For example: https://www.elastic.co/guide/en/cloud-enterprise/current/create-es-cluster.html

I believe they use version 2.0.

So far I've just been playing with generating JSON files from existing APIs (e.g. using api2swagger) then generating ASCIIDOC files from those JSON (e.g. using swagger2markup or https://github.com/elastic/swagger2asciidoc). The missing piece for me is generating the JSON from annotations directly in the elasticsearch and X-Pack APIs.

[Mpdreamz]

We have made attempts over the years to create a JSON schema specification for the API's but quite a few area's can not be expressed properly using it. Most common issues are around having variable keys of which only one can exist and other parts too loosely typed to codify. We found that we needed quite a few features from later JSON Schema drafts but support for these later schemas is/was not very widespread. See also: #8965.

Swagger and related specs work great when the server is written from the get go with it in mind, as is the case with the ECE API.

That said the whole idea of a client request and body specification is not completely dead. Rather than encoding the spec in json schema directly we are researching publishing a reference implementation in typescript the AST of which can be used to generate a domain model where one can easily iterate over all the endpoints and all the types in order to feed a generator. Swagger/JSON schema would just be one of the outputs.

One major major benfit here is that that typescript compiler services are very portable (hostable from many platforms so spec could potentially be part of the elasticsearch gradle build ) and errors in the specification would yield actual compiler errors that are easy to fix. Maintaining a json schema by hand vs a reference implementation through terse typescript interfaces is also a factor. Some parts of the API such as aggregations and suggest are irregular and clients are expected to write custom code to convert these. You'd still have the reference types though to know what to convert to however.

I've opened up our repository for folks to track, although we are not seeking active PR's at the moment https://github.com/elastic/elastic-client-generator. e.g an initial outdated spec can be found here: https://github.com/elastic/elastic-client-generator/tree/master/specification/specs. The work around utilizing the typescript AST to create a readable model that will also yield human readable compiler errors is done. We still need to go back and update the spec that was dumped from NEST's c# types and do manual touch ups after.

We'll resume work on this in the very near future as we'll work on moving the .NET client over from manual mapping to generated request and response classes (and serializers).

[stephanebastian]

@Mpdreamz Thanks a bunch for such a detailed explanation

I looked at #8965 where @clintongormley and others explained the shortcomings of JsonSchema. However this issue is quite old and a some progress has been made by JsonSchema, especially to express some fairly complex constraints.
At the time you looked at this issue, do you know if the 'dependencies´ keyword was already supported by JsonSchema ? It's a very neat and concise way to express conditional constraints (if the property A is defined then property B, C and D must be defined as well, etc..) ?
http://json-schema.org/draft-04/json-schema-validation.html#rfc.section.5.4.5

Concerning the TypeScript solution, this is a clever idea.
Having said that, the ES API such like most APIs is pretty static and I am wondering if TypeScript is not an overkill.
After all the only missing part from JsonSchema back in 2015 was the ability to express validation constraints ? Am I right ?
Is there anything else that JsonSchema could not handle at that time?

[renicon]

would it be possible provide a plugin like this one or to pick this one up and update it for current version? https://github.com/timschlechter/swagger-for-elasticsearch

@timschlechter stopped updating it due to too many changes so not sure if there is enough value vs. the effort of updating it. Maybe someone who has used this in the previous versions and found it useful can comment more.

[maciejmakowski]

Could someone please provide an ETA for when this could be made available? The most optimistic one is fine.

We are about to start work on a service to connect to our elastic search cluster, and our live would be way easier if we had a Swagger definition from which we can automatically generate a retrofit client, instead of having to code things by hand.

[tomcallahan]

@maciejmakowski We recognize that this is an issue for a lot of folks and are discussing it, but don't have anything further to share yet. We know that we want to do this (or something like it). I'm afraid that as a matter of course we do not share ETA's on specific features.

[rodripf]

In the meantime I solved the need of the swagger spec using the project elastic-swagger-generator_aspnetcore to generate the json from the ES provided json definitions. You can find the generated output on this gist.
I hope someone may find it useful.

[jkone27]

any updates on this? Thanks. this would also make possible to query elastic-search from TypeProvider via SwaggerTypeProvider (F#)

[phalverson]

Is there even an informal description of ES request and response formats?

[jpountz]

Currently the closest thing we have to that are examples in the documentation.

[dzmitry-lahoda]

after forward proxy of es from my kube i just want to shell into es with all intellisence, preferably in pwsh

[codebrain]

Is there even an informal description of ES request and response formats?

Yes, here; https://github.com/elastic/elastic-client-generator

Although this is very much an alpha.

[fbaligand]

Thanks for this link and this project!
Very promising!
The major missing point at this time is the API body specification.
But I noted that it is an alpha state ;)

[Kidlike]

Is there even an informal description of ES request and response formats?

Isn't this a formal description of the ES api? or am I missing something?
https://github.com/elastic/elasticsearch/tree/master/rest-api-spec

@elasticsearch guys, any updates on this? Are we going to get an OpenAPI spec? 🤩

[justindonnaruma]

Looks like it is available, just not on our private instances yet. https://cloud.elastic.co/api/v1/api-docs/spec.json

[fbaligand]

Thanks for the share @justindonnaruma but it seems to only include Elastic Cloud APIs.

For instance, /_search API is not described

[networkinss]

Is there even an informal description of ES request and response formats?

Isn't this a formal description of the ES api? or am I missing something?
https://github.com/elastic/elasticsearch/tree/master/rest-api-spec

@elasticsearch guys, any updates on this? Are we going to get an OpenAPI spec? star_struck

I am new to Elasticsearch. The JSON provided there are indeed interesting,
but it is very cumbersome to use that, since it is not OpenAPI.

It was said that JSON did not yet provide all what is needed.
However, to have one document in OpenAPI and having a text description for that what cannot
described as OpenAPI will be very very helpful.

I think it is not difficult to even make an autogenerator for existing indexes.

[elasticmachine]

Pinging @elastic/clients-team (Team:Clients)

[philkra]

As @Mpdreamz stated in #25152 (comment) , OAS and/or RAML are not expressive enough to describe the Elasticsearch's request and responses. As example, aggregations have variable keys, this can not be declared properly with these spec formats.

That being said, you can find a complete specification of request & responses bodies in https://github.com/elastic/elasticsearch-specification - more precise, please consult this file: schema.json. This schema provides the baseline for the type definition of the Java Client as well as the types in the JavaScript Client. The structure is described here.

Closing this issue, as OAS output can't achieve in a complete way. Please open a ticket in https://github.com/elastic/elasticsearch-specification if you have follow up question, you want to contribute or we can assist you in any form using the spec.

[dzmitry-lahoda]

@philkra, sorry for off topic. where can i look at example of aggregations have variable keys requested responses? i want to check if gql supports these

[philkra]

Hi @dzmitry-lahoda , sorry, I don't have an answer for this specific question. Please post this question on https://discuss.elastic.co/ for a broader knowledge transfer, thank you

[honza-zidek]

Well, the issue is marked as Closed ("philkra closed this as completed").
My local Elastisearch is running on localhost:9200.
On which address is then the OpenAPI available?
The standard address http://localhost:9200/swagger-ui.html returns error.

[honza-zidek]

@philkra I tried to import https://raw.githubusercontent.com/elastic/elasticsearch-specification/main/output/schema/schema.json into POSTMAN, but all I get is

Error while importing: format not recognized

[justindonnaruma]

https://www.elastic.co/guide/en/cloud/current/ec-openapi-specification.html

[fbaligand]

Well, this is Elastic Cloud API specification, not Elasticsearch API specification.

[stefnestor]

For future searchers finding this thread: you can load Elasticsearch Serverless (not general Elasticsearch)'s OpenAPI from elasticsearch-specification via ./output/openapi/elasticsearch-serverless-openapi.json. (Kindly still note per above in 2021 that "OAS and/or RAML are not expressive enough to describe the Elasticsearch's request and responses".)

E.g. loaded via Swagger: