Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Swagger docs #892

Closed
wants to merge 42 commits into from
Closed

Swagger docs #892

wants to merge 42 commits into from

Conversation

depau
Copy link
Contributor

@depau depau commented May 4, 2021

Type of change

Type A:

  • Bug fix (non-breaking change which fixes an issue)
  • Refactor/Code Cleanup
  • Docs
  • Capability implementation for existing core capability
  • New robot implementation

This PR introduces Swagger API docs.

Docs are generated at build-time and added to client/swagger.json, which I gitignored and should be force-added when needed.

An alternative to adding it to the client could be to add them to the docs website. However we still need to add the swagger "client" libraries to the UI because CORS etc.

Copy link
Contributor

@alexkn alexkn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wouldn't it be better to generate the swagger.json in the CI and release job?

@depau
Copy link
Contributor Author

depau commented May 4, 2021

Wouldn't it be better to generate the swagger.json in the CI and release job?

Mmh yes indeed, considering that it will also lint the swagger definitions.

Let's see what @Hypfer thinks

@Hypfer
Copy link
Owner

Hypfer commented May 4, 2021

Are there any downsides?

@depau
Copy link
Contributor Author

depau commented May 4, 2021

Are there any downsides?

Swagger compilation takes < 1s, I would add it to the build command (and therefore to CI jobs).

This also implies that it is validated on every build: I pulled in a swagger validator that is run after generating swagger.json

@depau depau marked this pull request as ready for review May 4, 2021 10:14
@depau
Copy link
Contributor Author

depau commented May 4, 2021

Ready for review as soon as the build stops failing.

Still left to do (not by me :) ):

  • Add annotations to all state attributes
  • Change annotations of state endpoints to something like
    type: array
    items:
      anyOf:
        - $ref: "#/components/schemas/BlaBlaStateAttr...
        [...]
  • Same as above for the map children
  • Evaluate ways to properly document SSE endpoints

Btw, the swagger endpoint is /swagger.

@Hypfer
Copy link
Owner

Hypfer commented May 4, 2021

I do have to say that writing YAML inside a jsdoc-style comment is insanely painful and I have absolutely no idea how you've managed to document 80% of the whole application without throwing your computer out of the window.

Can we do something so that this becomes less of a pain?
Like maintaining seperate files where the IDE can at least reduce the overall amount of YAML headache?

What I also strongly dislike is that apparently there's no way to reuse the existing types in the swagger docs which leads to enums being duplicated for no good reason.

@Hypfer
Copy link
Owner

Hypfer commented May 4, 2021

So i just found this error by actively looking at the generated swagger.json and not because anything failed or warned me or was visible via syntax highliting anywhere else

image

And as it turns out, the indentation is wrong.
Thanks YAML for making that completely invisible
image

Also, what is this object schema view? Why is everything spaced half a screen apart
image

Why do we use jsdoc when it doesn't even care about the already existing and defined types?
What is the point? Without inferring anything from the file it is stored in, what are the upsides of having it there in the first place instead of just having one single huge yaml file?
How can this be considered an improvement?

I've been spending the last hour to research if we're doing something wrong here.
Apparently not.
Apparently, this is the best it will get when using swagger which is just outrageous.

How is that even possible? How can anything this developer-hostile the best thing currently available in the market?
Does everyone doing web development hate themselves?

While I really appreciate your effort here, I'm afraid that I can't merge this or anything that adds swagger support, since I can guarantee that I will loose any motivation to maintain this project the moment 1/3 of every file becomes comment YAML and every time I look at the API I'm greeted with "Swagger by $BRAND_NAME".

Did you know $BRAND_NAME? Come join our Webinar. Go watch a Video Tutorial because we're so bad at providing docs about our product that is about docs that we need to resort to videos. Also, buy $BRAND_NAME merch today and while you're at it, add a $BRAND_NAME sticker to your macbook so you can show everyone that you just love to suffer.

It seems to me that OpenAPI prioritizes the API-user experience over the API-developer experience, which of course makes sense if you're trying to sell a product. With Valetudo however, there's no such incentive and therefore I'd strongly prefer developer experience over anything else.

Personally, I'd instantly walk away from a project when confronted with that yaml hell and a growing list of docs and other things that you'll have to update manually when changing basically anything about the project.

I don't want the Valetudo PR template to become a list of checkboxes that one has to read and check every single time just because the tooling is so bad that things need constant reminding instead of just working as intended.

Anyways, sorry for the rant and your wasted time.
I didn't expect things to be this bad when I said sure, go ahead

@Hypfer Hypfer closed this May 4, 2021
@depau
Copy link
Contributor Author

depau commented May 5, 2021

I've been spending the last hour to research if we're doing something wrong here.
Apparently not.
Apparently, this is the best it will get when using swagger which is just outrageous.

How is that even possible? How can anything this developer-hostile the best thing currently available in the market?
Does everyone doing web development hate themselves?

I'd love to tell you there's a better option but no. This is the best you can get with JavaScript at the current state of the art (assuming we can call it "art" cause I don't think it's very appropriate). This is the same rant I've always had at work and this is the reason why I'd write a web application in C rather than using JavaScript. If that's the environment I have to work in, I'd rather work on memory corruption bugs.

When I had used it at work I mostly worked on Java and with Java everything works waaaay more smoothly, basically the Swagger implementation can infer pretty much everything that's required from the source code.

What I also strongly dislike is that apparently there's no way to reuse the existing types in the swagger docs which leads to enums being duplicated for no good reason.

With TS you can do it to some extent, but with that being said (and also mentioning that the guys at the company are actually very good and really against this type of duplication) we'd still add the yaml definition on top, even on TS. And considering that TS is imo absolutely not worth the pain to begin with, 🤷‍♂️

On top of that, Express has an horrendous API and there's basically nothing you can do to automatically infer the request structure etc, as opposed to stuff like SpringBoot or even just Flask.

and every time I look at the API I'm greeted with "Swagger by $BRAND_NAME".

That triggers me as well and I was going to add some CSS to remove it 🤣


Now, a few counterarguments:

I do have to say that writing YAML inside a jsdoc-style comment is insanely painful and I have absolutely no idea how you've managed to document 80% of the whole application without throwing your computer out of the window.

I think there is an extension for VSCode. I found nothing for WebStorm but you get the hang of it, "space space" instead of pressing tabs and Alt+J for changing indentation for multiple lines.

Also, for the most part you can just copy-paste from other endpoints, most of them have the same "layout".

Not ideal or particularly "fun", but also not as bad as you say.

Personally, I'd instantly walk away from a project when confronted with that yaml hell and a growing list of docs and other things that you'll have to update manually when changing basically anything about the project.

Mmh I think you're overwhelmed by the size of the PR. I don't think it's particularly unmanageable. It's pretty bad, but it could have been much much worse.

Overall, what has to be documented with the @swagger tags is requests and objects sent into requests.
In practice, the list contains:

  • All routers
  • All serializable entities

While I'd love the second to not be the case, there's not much I can do. But on the other hand, the standard used for those is JSONschema, which isn't (imo) bad at all. It is a separate standard from Swagger, fyi.

For the routers, the REST APIs really need some sort of formalization (see again my comment on Express) and I think that this type of comment does the job just fine. All the time someone will spend adding the definition for their new or updated REST endpoint is time well spent and it is 1/4 the time some person wanting to use the REST API will spend finding the endpoint's source code and jumping around it to find all the allowed payloads and evaluating all the ways it works.

Can we do something so that this becomes less of a pain?
Like maintaining seperate files where the IDE can at least reduce the overall amount of YAML headache?

I don't think having separate files is a good idea since new stuff will never be documented. Having the docs close to the code they document adds to the guilt of doing things halfway.

I don't want the Valetudo PR template to become a list of checkboxes that one has to read and check every single time just because the tooling is so bad that things need constant reminding instead of just working as intended.

Are you sure? The code under webserver hasn't really seen a lot of particularly notable changes since the caps system refactor.


Conclusions

I totally understand your concerns and your outrage was exactly the same I felt when I first used Swagger with TS at work.

However, after letting the outrage boil down and after accepting that if you work with JS and Express this is what you have to deal with, I think that the overall advantages are more than the disadvantages. The disadvantages also do not make the code unmaintainable. It's just that they could have thought it a lot more thoroughly.

I think this for many other JS libraries that I hate: Express is one of them, another one is Jest. I'm not going to go into exactly why but while I think Express is one of the worst libraries ever written - only Jest is worse than it - we can still agree that having shared code for managing bug-prone HTTP stuff and that having tests is better than not.

Finally, here's some quotes from screenshots I have from when I used to work with TS, translated and censored:

expect("THIS BULLSHIT").toBe("THIS BULLSHIT"); // This test would fail

and from my shell config

# Because JavaScript
alias [swear word in italian]="npm run build; npm run test"
alias [swear word in italian]="npm run build"
alias [swear word in italian]="npm run start"

tl;dr: You used JavaScript, what did you expect? :P

When you decide to port the backend to golang or rust, hit me up :P

@alexkn
Copy link
Contributor

alexkn commented May 5, 2021

I didn't tried it yet, but what about using express-jsdoc-swagger?
That looks more like normal JSDoc.

@Hypfer
Copy link
Owner

Hypfer commented May 5, 2021

Thank you for taking the time to respond to that!

That triggers me as well and I was going to add some CSS to remove it 🤣

Yes please

So there are a few reasons for why I don't like the comments as they are now:

  • They are terrible to write due to yaml. Even with extensions, the * at the start of the line break everything
  • They bloat the file
  • They drastically reduce the quality of search results

Therefore, I've decided to rework that and instead add a doc folder to each folder where stuff needs documentation and have files named Classname.swagger.json in there. It doesn't matter anyways, since there's literally no connection to the comment in the js-file whatsoever

This way, they are still right next to the files that they should document while not bloating them and requiring us to strip them from the binary in the build process.
Otherwise, I can already smell the "we need to minify the code because it's just too large" PR.
WebDev has become disease where you have to use C to mitigate the negative effects of B which mitigates the negative effects of A and so on. It's terrible and I'd very much like to avoid that vicious cycle.

Also, I can easily exclude *.swagger.json from my search function.

Furthermore, it's not YAML. That alone should be more than enough reason to do it

Funfact:
image

This needs the empty line in YAML because there's no line-break otherwise. Two is one and one is none. \n gets ignored.
Thanks YAML, you god-awful piece of garbage devops tech.

@Hypfer
Copy link
Owner

Hypfer commented May 5, 2021

@alexkn

https://brikev.github.io/express-jsdoc-swagger-docs/#/requestBody

Hmm. That actually looks like it could re-use the existing stuff which would be even better

@depau
Copy link
Contributor Author

depau commented May 6, 2021

https://brikev.github.io/express-jsdoc-swagger-docs/#/requestBody

Hmm. That actually looks like it could re-use the existing stuff which would be even better

I like this one, thanks for the suggestion :)

add a doc folder to each folder where stuff needs documentation and have files named Classname.swagger.json

That's also a valid compromise, I'm not a huge fan but it works. Let's make it still YAML though, JSON would be unreadable.

Otherwise, I can already smell the "we need to minify the code because it's just too large" PR.

That's still a good improvement imo. Something like this:

Hypfer pushed a commit that referenced this pull request Jun 1, 2021
Hypfer added a commit that referenced this pull request Jun 8, 2021
* Import/convert from #892

* docs(swagger): Add swagger definitions for SystemRouter

* Swagger is an optional feature and should be treated as such

* Remove $brand

* docs(swagger): Add swagger definitions for NTPClientRouter

* chore: Fix package-lock

* chore: Fix package-lock

* docs(swagger): Document State Attributes and Map Layer + Map Entities

* Use tagged version

* fix pkg build

* add schema validation

* remove unnecessary custom jsdoc tag

Co-authored-by: Davide Depau <davide@depau.eu>
@github-actions github-actions bot locked as resolved and limited conversation to collaborators May 7, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants