Add versioning to the documentation

[oltarasenko]

Looks like it's quite hard to maintain the documentation in a good shape if you have multiple versions with different settings. (And especially different (slightly diverging) tutorials).

We need to have versioning.

I am slightly biased against the standard Elixir style of documenting the code (e.g. having very large docstrings makes the code unreadable, at least to me).

I would try to add an index page to docsify, and would store stand alone copies for different versions (in case of major updates in API)

[Ziinc]

I usually collapse the docstrings with vscode. I was also wondering why you hadn't used ExDoc.

The developer docs is lacking, I still don't quite understand some parts of crawly, and i had to carefully follow function names when trying to understand the pipelines logic. I think using exdoc for a set of dev docs would be good.

i personally find that separating code and documentation (dev or user) makes it harder to keep them in sync.

[oltarasenko]

Wow, that's a valuable feedback! Let me try to improve that part. Could you please raise an issue with things which was problematic in middlewares so I can create in improvement PR?

I would still separate the functions documentation with things like getting started/tutorials/examples/settings description (well I don't quite know if it's possible to describe settings properly with ExDoc) etc.

[Ziinc]

It's more to do with not having a single set of dev docs to look through and reference to know what each module and function is doing.

Actually, if I'm not wrong, hexdocs already automatically generates a set of docs using ExDoc:
https://hexdocs.pm/crawly/api-reference.html

I also notice that this ranks higher on google than the docsify page. I think its attributable to the JS only approach of docsify.

When it comes to ExDoc, I like how phoenix has done their guides section,which was user-friendly enough to get started, and is organized with separate markdown files in their repo. As for how dev docs is organized, I think the Elixir lang or Ecto module sections are the gold standard. Just the right level of depth with source code links to check implementation.

ExDoc already has inbuilt versioning, so i wouldn't worry about that.

[oltarasenko]

I would give it a try. @Ziinc maybe you could point me to the guides implementation, e.g. I don't quite see where their sources are. (E.g. how can we integrate it into the root of the hex docs page)

[Ziinc]

Oops I didn't copy paste it correctly before.

The guides on hexdocs:
https://hexdocs.pm/phoenix/overview.html#content

and on github:
https://github.com/phoenixframework/phoenix/tree/master/guides

[oltarasenko]

@Ziinc I have made the first step towards hexdocs. I kind of liked what I am getting after all, so I think I will shut down the current version of static docs in a while (in favor of the hexdocs).

Could you please have a glance on: #32

[oltarasenko]

Done in PR #32