Create (server) API for Studio

[magicmatatjahu]

In #80 proposal was proposed to create a (server) API that would later be used by the Studio for things that cannot be done in the browser. Most of our tools are isomorphic, but e.g. generator still need to run a NodeJS environment. Issue is opened to discuss how this API should look like, what it should offer, in which technology/framework it should be written and in which way it should be distributed.

@jonaslagoni @derberg @fmvilas @smoya You should be interested in this topic :)
@BOLT04 If you want also add comment, feel free, but I'm not forcing anything.

Closed issues (related to old implementation of the Studio) but are related to the server API and should be consider to implement:

• Increase payload size #9
• Support base URL using request headers #12
• Move plugins to their own repos #8 - plugins for server API
• Create a Google Analytics plugin #14 - Google Analitycs plugin - mainly in front-end app, but here it can be also used.

[smoya]

Do we have a list of those actions the server should perform, e.g. generation of templates?
In terms of infrastructure estimation costs, what is the estimated traffic for that API? Would it be similar to current playground?

[magicmatatjahu]

@smoya

Do we have a list of those actions the server should perform, e.g. generation of templates?

At the moment in plans is implement things from playground, so generating and downloaded result of templates. In future we can implement (as @jonaslagoni suggest) the model generation, optimizations etc, but a lot of things we can perform in browser, because our libraries are isomorphic, so the most important thing is the generation of templates. The list is open and we can add to it whatever we/community need.

In terms of infrastructure estimation costs, what is the estimated traffic for that API? Would it be similar to current playground?

It depends in which way we will go - in which form we will distribute that API - because we have several options:

• treat API as separate package that anyone can use to set up a server at their servers.
• publish api with (e.g.) api.asyncapi.com domain, that everyone will be able to request.
• merging of two above - everyone can be able to set up new server, but also be able use server (by free) hosted by AsyncAPI Initiative

Of course, for first option, the costs are 0, for second it depends on traffic 😅 So at the moment I don't know. We should check what traffic we have on https://playground.asyncapi.io/ and by this estimate costs. But we must also remember that the community will grow, so costs also. We can also limit hourly/IP requests to reduce costs etc.

[BOLT04]

Hi @magicmatatjahu 🙂, in regards to the technology/framework I think Node.js & Express.js w/ TypeScript is fine. I think it would be cool to have a version of the API deployed in DigitalOcean, for example, but have it rate-limited like you mentioned, to reduce costs.

For the API design I think for now we would have endpoints that is the equivalent of the markdown route:

studio/src/routes/markdown.js

Line 14 in 06cd3b1

router.post('/generate', async (req, res) => {

To generate and download the result of templates. I just don't know if it would only be markdown or more templates. If we want the @asyncapi/nodejs-ws-template template to be downloaded from the UI, then the template package name could be parameter in the HTTP request, and then be validated with a list of valid templates 🤔.

What other endpoints does everyone think the API needs? @jonaslagoni suggestion for the optimizer is pretty cool, can we use the package https://github.com/asyncapi/optimizer on the client-side, or do we need an endpoint on the API?

[smoya]

We can also limit hourly/IP requests to reduce costs etc.

Perhaps not to go in very detail on this, as I think it's not the most important part about this issue:

I would say the only (strong) reason for adding a rate limiter are attacks. So I would say, with lower priority, we could add nginx in front of all our HTTP services (meaning current playground, hub, etc), which can implement a simple leaky bucket algorithm based on headers and/or IP. This could be automated by the installation scripts (Terraform).

[derberg]

Perhaps not to go in very detail on this, as I think it's not the most important part about this issue:

Definitely let us not overcomplicate. We do not create a commercial API, but free service that can be improved basing on how we learn it is used. Rate limiting can be added once we see someone abuses the API, and doesn't make much sense to focus on all these tiny details IMHO (better first connect google analytics to playground to see how ofter people actually click download button - but again if that really makes sense)

More important. And I have a pretty strong opinion here 😄 We should follow API first approach here, so really looking forward to seeing the API designed with latest OpenAPI and some code generated with it 😄

[magicmatatjahu]

More important. And I have a pretty strong opinion here 😄 We should follow API first approach here, so really looking forward to seeing the API designed with latest OpenAPI and some code generated with it 😄

Fran once suggested to go with the GraphQL - it would make sense in case we expand the API, but if we end up with just a generator then I think simple REST is enough.

[magicmatatjahu]

@BOLT04

To generate and download the result of templates. I just don't know if it would only be markdown or more templates. If we want the @asyncapi/nodejs-ws-template template to be downloaded from the UI, then the template package name could be parameter in the HTTP request, and then be validated with a list of valid templates 🤔.

Exactly, I also thought about that in this way, that you will have possibility to generate not only markdown/html but also another our templates. Most likely we won't go for custom template generation due to security issues.

What other endpoints does everyone think the API needs? @jonaslagoni suggestion for the optimizer is pretty cool, can we use the package https://github.com/asyncapi/optimizer on the client-side, or do we need an endpoint on the API?

We will use optimizer in the client-side, but maybe we will also create endpoint for that in API. Currently only the generator cannot be used on the client side and it is a priority, but if need be we will do endpoints for other things too :)

[derberg]

Fran once suggested to go with the GraphQL - it would make sense in case we expand the API, but if we end up with just a generator then I think simple REST is enough.

even if we expand to more, like optimizations, diff, etc. I honestly do not see a benefit of using GraphQL. This is going to be an API that you use for one thing at a time IMHO. REST should be grant

[fmvilas]

I suggested GraphQL when it was only a server for the frontend but since it's going to be an open API that anyone would use, I'd go for a REST API instead. Usually, I go for GraphQL when I control the API and the client(s). If we let anyone call our API, we don't control the clients anymore and therefore I think it would be better to bring a REST API.

[magicmatatjahu]

I would be very willing to start writing the discussed API app, of course only the /generate path for markdown-template and html-template. Most of the stuff for the new Studio is already merged on the spa-scratch branch, and before we release the new Studio we should also support templating generations like the old playground. But the question is where to write the API app? In the studio repository (and move to the new one in the future), or immediately in the new one?

cc @derberg @jonaslagoni @smoya?

[jonaslagoni]

I would say immediately in a new repo, and ensure you design the API first with OpenAPI 😆 (no joke).

[magicmatatjahu]

About OpenAPI I know 😄 So then, what name of new repo? asyncapi-api 😆 ?

[smoya]

Isn't asyncapi-minion the name you were looking for? hahaha. At the end, is an API that does all the work for you.

[magicmatatjahu]

What about api?

The name can always be changed, but what about the tech stack? @BOLT04 suggested to use Express + Typescript. Do you agree @jonaslagoni @smoya @derberg? The app will be small at first, so we can always switch to another framework in next iteration if we want.

[smoya]

What about api?

The name can always be changed, but what about the tech stack? @BOLT04 suggested to use Express + Typescript. Do you agree @jonaslagoni @smoya @derberg? The app will be small at first, so we can always switch to another framework in next iteration if we want.

Im not very up to date with node web frameworks but i recently read about Koa which seems to be like a modern version of ExpressJS (and built by the same team) that supports async functions.
Woud it make sense to use it due to the nature of the work we might wanna do from the controllers? I just saw you can achieve the same with ExpressJS, so nvm :)

Otherwise, ExpressJS seems like the standard.

[magicmatatjahu]

Ok, I will create server-api repo and then we can rename it :)

[fmvilas]

I'd consider using https://github.com/anttiviljami/openapi-backend as the tech stack.

[magicmatatjahu]

We have such a project in our organization (https://github.com/asyncapi/server-api) and we are very close to integrate first path (/generate - #229). I close it, please reopen if you need.