Automatically expose documented API based on the defined Operations

[Martinsos]

We could use Swagger or Openapi for this, or something similar!

It would also make sense to first work more on allowing users to define which API is public, should not change, allow them to customize it, ... , before we let them expose it like this.

EDIT: For a way to manually add Swagger docs at the moment, check how @Genyus did it lower in this issue: #863 (comment) .

[Martinsos]

Btw Rails has an option to print all the routes, maybe that is good first start.

Btw this is also about us promising to not change how routes work without it being a breaking change.

[Martinsos]

Related to #268

[Martinsos]

@breadchris expressed interest in this.

[Martinsos]

NOTE: We are now using superjson for serializing / deserializing RPC calls, so if we will be exposing them as an API, we need to look into making that work with superjson.

[Martinsos]

From some feedback we got, this would help with allowing building mobile and other clients that can then use existing Wasp's backend.

[Martinsos]

Btw they could create their own API by writing their own api declarations and reusing the logic they use in Operations, but that is quite some work and would be much nicer to just be able to get that API for free.

[FrancisVarga]

openapi would be the best since it has newest standard...
is there any progress on this?

[Martinsos]

openapi would be the best since it has newest standard... is there any progress on this?

While we will certainly be doing this, we don't yet have a specific date for it, other stuff is being prioritized at the moment! But it will be happening.

[tito300]

Is there an update on this feature? it would help with apps that expose API endpoints for external use.

[Martinsos]

Is there an update on this feature? it would help with apps that expose API endpoints for external use.

Nothing yet, we are focusing on 1.0 now and this will probably be coming after 1.0! Are you looking to use externally the Operations (Queries and Actions) or Custom API endpoints? All details are appreciated, it will help us when we will be designing this feature.

[Martinsos]

Besides nicely documented REST API, another solution that makes sense next to this one, as an additional layer, is also an SDK for calling RPC! We could produce one for JS for sure, maybe also one for python, go, ... . We get into RPC terittory then, might make sense to use somwthing like grpc in the background then? Hm.

[tito300]

for my use case, I was looking to expose custom REST api endpoints specifically. My application will allow clients to call api endpoints to send realtime events. Also, maybe off topic for this ticket but I also needed to be able to provide api keys or tokens to use for these endpoints since this will be server to server communication.

I wasn't planning on exposing RPC but not opposed to it. It would be cool if the sdk can also interact with REST endpoints!

[infomiho]

@Genyus documented his attempt at adding Swagger from user-land here: https://ptb.discord.com/channels/686873244791210014/1349675223477125171/1350496573259513897

[Genyus]

I attempted a few different approaches to integrate Swagger:

1. Implementing directly. This involved installing the swagger-jsdoc and swagger-ui-express packages, creating a new api namespace with middleware for the Swagger UI at the desired path. I also found I needed to install helmet so that I could disable CSP and HSTS at that path, for the docs to be rendered.

However, the conclusion was that because Wasp generates the final routes, there's no way to define the JSDoc comments in a persistent way in your source code, so the output was always empty (I did verify that adding comments to the built code works, but obviously that’s not maintainable).

2. I then tried to use express-oas-generator, but they want you to hook into specific points of the express server lifecycle, which Wasp doesn't expose, so I left that one alone.

3. Finally, I attempted using swagger-autogen and while it can generate the basic route definitions, I was faced with the same problem as attempt 1 above, where there's no way to define the necessary comments in the generated code.

Happy to be corrected by a member of the team, but as far as I can tell, it's not currently possible practical.

[Martinsos]

for my use case, I was looking to expose custom REST api endpoints specifically. My application will allow clients to call api endpoints to send realtime events. Also, maybe off topic for this ticket but I also needed to be able to provide api keys or tokens to use for these endpoints since this will be server to server communication.

I wasn't planning on exposing RPC but not opposed to it. It would be cool if the sdk can also interact with REST endpoints!

Thanks for sharing! I believe you already managed to do this via https://wasp.sh/docs/advanced/apis ?

[Martinsos]

I attempted a few different approaches to integrate Swagger:

1. Implementing directly. This involved installing the swagger-jsdoc and swagger-ui-express packages, creating a new api namespace with middleware for the Swagger UI at the desired path. I also found I needed to install helmet so that I could disable CSP and HSTS at that path, for the docs to be rendered.

However, the conclusion was that because Wasp generates the final routes, there's no way to define the JSDoc comments in a persistent way in your source code, so the output was always empty (I did verify that adding comments to the built code works, but obviously that’s not maintainable).

2. I then tried to use express-oas-generator, but they want you to hook into specific points of the express server lifecycle, which Wasp doesn't expose, so I left that one alone.
3. Finally, I attempted using swagger-autogen and while it can generate the basic route definitions, I was faced with the same problem as First version of informative README. #1 above, where there's no way to define the necessary comments in the generated code.

Happy to be corrected by a member of the team, but as far as I can tell, it's not currently possible practical.

Thanks @Genyus for all the awesome effort here!

So main blocker seems to be pushing comments to the generated code? That is interesting! I wonder how we could potentially support that in the future -> some kind of hooking up into Wasp's generation process. Of course first-level support for Swagger would be even better. Btw I don't know a lot about Swagger -> why are these comments needed, what kind of comments exactly is it? The idea is that you define a comment per each route in Express and Swagger picks that up from the code, or how does that work?

As for hooking into express lifecycle, that Wasp doens't expose -> if you let us know which part specifically, we might look into exposing that!

[Genyus]

why are these comments needed, what kind of comments exactly is it? The idea is that you define a comment per each route in Express and Swagger picks that up from the code, or how does that work?

Yes, the typical implementation for JS/TS projects is via JSDoc comments on the route definitions, e.g.:

/**
 * @swagger
 * /api/job-listings/by-url:
 *   get:
 *     summary: Retrieves a user job listing by URL
 *     description: Fetches a user job listing by the job listing URL for the authenticated user
 *     tags:
 *       - Job Listings
 *     security:
 *       - bearerAuth: []
 *     parameters:
 *       - in: query
 *         name: url
 *         required: true
 *         schema:
 *           type: string
 *         description: The URL of the job listing to retrieve
 *     responses:
 *       200:
 *         description: User job listing retrieved successfully
 *         content:
 *           application/json:
 *             schema:
 *               type: object
 *               properties:
 *                 id:
 *                   type: string
 *                 userId:
 *                   type: string
 *                 jobListingId:
 *                   type: string
 *                 status:
 *                   type: string
 *                 documents:
 *                   type: array
 *                   items:
 *                     type: object
 *                 jobListing:
 *                   type: object
 *                 match:
 *                   type: object
 *                 user:
 *                   type: object
 *       400:
 *         description: URL parameter is missing
 *         content:
 *           application/json:
 *             schema:
 *               type: object
 *               properties:
 *                 error:
 *                   type: string
 *       401:
 *         description: Unauthorized - User is not authenticated
 *       404:
 *         description: Job listing or user job listing not found
 *       500:
 *         description: Server error
 */
router.get(
  '/api/job-listings/by-url',
  [auth, ...getUserJobListingByUrlApiMiddleware],
  handleRejection(
    (
      req: Parameters<typeof _waspgetUserJobListingByUrlApifn>[0] & { user: AuthUserData | null },
      res: Parameters<typeof _waspgetUserJobListingByUrlApifn>[1],
    ) => {
      // <snip>
    }
  )
)

which then generates output like this:

As for hooking into express lifecycle, that Wasp doens't expose -> if you let us know which part specifically, we might look into exposing that!

For a tool like express-oas-generator, it wants its functions to be added as the first and last middleware. So having some ability to control the ordering/priority of middleware functions might suffice here.

[Martinsos]

@Genyus I think you should be able to modify global middleware at the moment, check out https://wasp.sh/docs/advanced/middleware-config#default-global-middleware- , it explains a bit, and then there is a section about customizing global middleware. You can take the existing one and add to it at the start and at the end, I believe.

Btw I am guessing alternative solution might be something like https://www.npmjs.com/package/swagger-routes-express or using https://www.npmjs.com/package/swagger-ui-express directly, where routes are described in a standalone yaml file? But that doesn't sound great hm, I would much rather prefer the descriptions stand next to the route code, as you went for.

But for https://github.com/Surnet/swagger-jsdoc that you used -> How we generate the project is a bit complicated, we link to your user code from the generated server code, so that might make it tricky, but I wonder if the problem might come down to not correctly setting that path in options.apis in

const swaggerJsdoc = require('swagger-jsdoc');

const options = {
  failOnErrors: true, // Whether or not to throw when parsing errors. Defaults to false.
  definition: {
    openapi: '3.0.0',
    info: {
      title: 'Hello World',
      version: '1.0.0',
    },
  },
  apis: ['./src/routes*.js'],
};

const openapiSpecification = swaggerJsdoc(options);

I am not sure off the bat what the path should there be, but what have you tried? Are you sure you can't set that path to something that works?

[Genyus]

I am not sure off the bat what the path should there be, but what have you tried? Are you sure you can't set that path to something that works?

@Martinsos Thanks for this! 💡 I always thought the Swagger comments worked like Java annotations and had to be applied directly to the routes they describe, but I had a look at the package and realised it's just a simple regex-based parser, so they can live anywhere. I moved them to my *.wasp.ts files (which seemed like the most logical place as that's where the routes are defined) and updated the apis path as you suggested. Works perfectly now! The JSDoc comments can be maintained in the source files and get picked up at runtime.

I've created a Gist and will share my updated findings in the original Discord thread

[Martinsos]

Awesome @Genyus, thanks for all the effort you put into this! This will be a nice starting point for when we decide to implement this as a feature!

[Martinsos]

I wonder if https://typespec.io project wuold be interesting for Wasp Operations - it is worth chekcking it out a bit.