Improved Crystal API model mapping

[straight-shoota]

#4746 added JSON serialization for Crystal API components like types, methods, constants etc.
The implementation uses custom #to_json methods to extract the data from the compiler models.

However, it would be great if there was a way to also deserialize the JSON data into Crystal models. This is necessary for external doc generators to hook in. It also offers possibility for other uses of the generated API model.

This doesn't necessarily need to be part of the stdlib, but given that the serialization part is required for the compiler anyway, I figure it would make sense to also add the deserialization so both can be kept in sync.

The best way to do this would be to implement simple data model types with JSON serialization. The compiler types would just create such a model instead of serializing directly with custom #to_json methods.
A rough draft of the API model can be seen here: https://gist.github.com/straight-shoota/d6b783dd4c973712aa6a2183153fd391

[straight-shoota]

Building on that idea, I'd like to propose refactoring the docs generator, to more effectively separate it into two stages:

• extracting a documentation model from the parsed program
• processing that data into display format (HTML)

The interface between both stages would be a structured format which can easily be serialized to JSON and would also serve as an external interface which can be used for customized doc generators and other purposes.

We've discussed about potentially removing the HTML-generation from the compiler entirely. This proposal doesn't go this far, but it would be a preparation for that. But it's a good improvement anyway to refactor and separate concerns.

[jkthorne]

would removing the html generation be a step towards removing markdown?

[straight-shoota]

Markdown is only required for generating the HTML, so it would remove that dependency entirely from the compiler.

But that's talk about following steps and out of scope here.

[crysbot]

This issue has been mentioned on Crystal Forum. There might be relevant details there:

https://forum.crystal-lang.org/t/the-weight-of-compiler-tools/6888/1

[devnote-dev]

I have created Redoc which covers most of the docs JSON models and their attributes. I think at some point this could be used for documentation generation, I've already integrated it into Docr to power its info and search commands.