Epic 3: Document API with an OpenAPI schema

[JamesMGreene]

Description

Mission statement: Users can read the “raw” OpenAPI docs since they’re written in YAML, but for a better experience, we want to build a Pages site that intakes the raw files and outputs formatted, easy-to-read information.

Context
We currently document our API with manually updating Wiki pages, which are prone to inaccuracy and to becoming outdated. 😕

One alternative to consider would be documenting the API with a central OpenAPI schema file.

Side benefit: if desired, we can also then use additional Express middleware tooling for request validation. I have mixed feelings about this in practice based on past experiences but it is not unlikely that there are better tools available these days than a few years back. 🤷🏻‍♂️

With either of those approaches, we can then generate the documentation for it using the existing OpenAPI/Swagger tooling (see example demo page).

Spec

SMART exit criteria:

• Specific – API documentation
• Measurable – API-related PRs include schema changes rather than requiring manual updates to the documentation
• Assignable – I'm open to it unless someone else would like to jump in
• Realistic – auto-generated documentation based on developer-led API changes at the code/schema level
• Time-related – Probably a week's worth of intermittent work

Links/Attachments for Reference

• OpenAPI
• SwaggerUI demo page
• Express OpenAPI Validator middleware (one option of many 🤷🏻‍♂️)

Considerations

• Since our API is only callable by our own frontend, is this just creating unnecessary work for us in the long term?
  • Similar question for our Wiki-based documentation, though, to be honest? 🤷🏻‍♂️
  • I think given the nature of our project and how contributors may be transient, this could still be a valuable idea. 🤔

Exit Criteria Implementation notes

• Create OpenAPI schema file and document the API in it
• Add Actions workflow to generate Swagger UI documentation from the OpenAPI schema and store it into the gh-pages branch
• Enable GitHub Pages to host the documentation from the gh-pages branch
• Potentially add Express middleware for schema-based validation

Hack out Tasks (9:15-10:30):

• Initial setup: Can we can just use the steps contained in this repo https://github.com/peter-evans/swagger-github-pages#steps-to-manually-configure-in-your-own-repository to generate and publish our “pretty” documentation on a GitHub Pages site.
• OpenAPI overall workflow and schemas
• API testing: Document how it works and how to do it. But that's outside the scope of the API specifications, so where are we putting that, and how are we getting it there? We only had API documentation before, so we've been solving for that problem, but now we have different kinds.
• Codespaces addition with OpenAPI plugin?
• Actions optimizations like Swagger UI?

[manishapriya94]

@JaymePerlman @momartinjazz for visibility

[manishapriya94]

@JaymePerlman:
API in Wikis is very complete

Clarify:
Which APIs are they going to use?
What are parameters you can modify?
Are new APIs anticipated to be added?

• Meta documentation: define a process for complete steps
  • Ex:
    • Code, integrate, PR, then the documentation
      ABC API and I’ve gotten to step 7
      Can someone help me to step 7-9
  • What is the process to adding to the OpenAPI
    • What happens for non devs: instructions to support an issue that devs can enable in OpenAPI?

Potential videos:

• I'm new and need to set up the codespace environment

[JaymePerlman]

Capturing notes here.

Mission statement

Users can read the “raw” OpenAPI docs since they’re written in YAML, but for a better experience, we want to build a human-readable set of docs. There are tools that consume the raw files and output formatted, easy-to-read information.

Documentation should cover any HTTP error responses that are likely to happen and include instructions on how to remedy it. And some kind of message for the “200 success” code.

Potential jump-start

As I suspected, we’re far from the first to try and solve this problem. For initial setup, we can just use the steps contained in this repo to generate and publish our “pretty” documentation on a GitHub Pages site. However, this site contains only one specification file, and we might end up needing more than one.

As a bonus, it even contains a publish Action that we can modify to suit. Right now it runs an update every 10 hours (cron job) but we can probably change that to have it run when triggered instead of on a regular cadence. We could also change it to run more frequently, or hey, worst case it’s completely functional as-is.

Having this basic infrastructure already in place frees up the hackathon to work on other things.

Technical References

If folks are working in a codespace they can add the OpenAPI plugin.

Here’s a good basic structure documentation site, but the official specification is pretty easy to follow.

The root file must contain a few objects:

• The root OpenAPI object which also contains a
• general Info object that contains a title, version, licensing, contact information, and a link to the Pages site (externalDocs), also ToS or other stuff.
• Server URL/base path configuration
• And then each API has paths /root/user and those paths have actions root/user/get

We probably want to have as many re-usable parts as possible. We should define a list of them, but there are definitely a few that leap to mind:

• Schemas - contain the reusable ones in one place, like Address, every API probably needs the user’s address, or username.
• Responses - generic “success” or “404” or other messages we don’t want to write out 5000 times
• Error - something went wrong and we don’t know what. Used when other, specifically written error messages do not apply. (case-else)

I don’t know enough to know if we need to define a security scheme (authentication/authorization), or if we’re even using any - that might be handled completely elsewhere, like by GitHub itself.

Thoughts and TBDs

Can we / do we document APIs we are using, not writing?
If so, they may have their own OpenAPI files already available, so we can potentially copy their stuff into our stuff rather than write it all up from scratch. Not sure what the plan is to keep all that up to date, though. If they change stuff that we rely on, how do we know?

Regarding API testing, we have a responsibility to document how to do it. But that's outside the scope of the API specifications, so where are we putting the SBS procedures for testing, and how are we getting it there? We only had API documentation before, so we've been solving for that problem, but now we have different kinds.

OpenAPI is a specification for writing APIs, not just documenting them; are the devs on board with writing to this specification?

Most parsers can handle one root file, and if we only document the APIs we write, we should only have one root file. If we decide to / can include APIs that have different root server sources, we'll need different files to capture them all. There are multiple documentation-generation tools available, but installing/configuring them may require some expert/developer time. The only one that seems to work with Jekyll, which means it's compatible with GH Pages, is jekyll-openapi But there are much more advanced options such as API explorer which includes an interactive API explorer. Awesome, if we can get it to work, but overkill?

[momartinjazz]

Epic Notes

Introductions and setting expectations

@JaymePerlman introduced her research around this Epic 3

• Need to get developer buy-in to tag team on the development effort and the supporting documentation.
• Are we creating or using APIs which impacts what we document?
• Do we agree we step back to decide a path? Discussed options and path forward. Looked at Node.js info.
• Need to set up environment to use Swagger.
• Can use YAML with both OpenAPI and Swagger tool which uses OpenAPI to compile. Use tools to create JSON and YAML files.
  • Node documents inline with comments. Uncertain how YAML files fit. This alleviates the need to have a schema file.
  • Does a YAML file make the OpenAPI portable?

Actions or impacts

• Manually move API documentation that exists to .js document.
• Swagger JS tool iterates and generates OpenAPI YAML.
• Action run Swagger command line and generate files for the UI.
  • Manual document endpoints still need validation.
  • Automatic enablement would be an Action.
• Identify a structure to contribute.
  • Add contribution guide and code of conduct.
  • How to test and validate
    • Test to ensure documentation conforms to JS.
    • Enable validation for PRs that new routes are documented.
      Note: Create Action on push that looks for new endpoints not documented; run Swagger express. This would open PR with descriptions to edit and merge into the PR. Drop comments. Opening PR against a developer's branch.
• Action to push to GitHub Pages.

Actions to get started

• Share steps with developer team for review and comment.
• Find a tool to generate .js document automatically first.
• Find standards.

Next steps

• Epic 3 team solidify steps above. Join Tuesday sync.
• Go to developer team for comments.

References

• https://oai.github.io/Documentation/start-here.html
• https://sphinxcontrib-openapi.readthedocs.io/
• https://github.com/swagger-api/swagger-node
• Sample JS project Swagger installed and it looks like the yaml is the contents: https://github.com/azizkale/Swagger-in-NodeJS/blob/master/Routes/posts.js
• https://github.com/fliptoo/swagger-express#readme
• VS plugin: https://github.com/ProgramEquity/amplify/tree/main/server/routes/api
• https://marketplace.visualstudio.com/items?itemName=brikev.express-jsdoc-swagger-snippets
• https://cs.github.com/?scopeName=All+repos&scope=&q=swagger