Split docs out of main repo and use ReadTheDocs.org

[cweagans]

There's a few benefits here:

• Documentation editing access can be handed out more freely than core ctags access without worry
• We can set up our ReadTheDocs project on docs.ctags.io
• Per-version documentation can be easily viewed (for instance, docs might vary between whatever is in master and our future 1.0.0 release)
• ReadTheDocs provides a really nice reading/search interface for our docs

I'll take on doing the move, but I wanted to run it by everyone else first.

[cweagans]

Note that this means the the docs directory in the ctags repo won't exist anymore and would live in universal-ctags/documentation.

[cweagans]

Some other benefits:

• Translatable documentation
• Epub and HTML export

[masatake]

Thank you. It seems that you know well in this area.

Could you tell me how our workflow for coding and writing documents changes?

Consdier I add an option to ctags. I also update docs/*.rst and make the changes
both code and documentation into a single pull request.
I think it is nice that I can do it in the unified source tree and repository.

[cweagans]

The documentation would live in a separate repo, so that wouldn't be possible anymore. The ctags repo would be code only, and the docs repo would be documentation only. I'd be interested in maintaining the documentation, so if you wanted to just put some bullet points in your PR and @ mention me, I can take that info and make sure it gets into the docs.

[masatake]

Thank you for the offering. However, I cannot image how we work together.

Look at docs/.rst. The most of all contents are for developers. So I cannot image well that someone who doesn't touch the code of ctags writes/updates documents in docs/.rst.

What kind of documents do you image?

On my side, I have some things I would like you to do about documentation: updating ctags.1.
Aboud ctags.1 can be managed under ReadTheDocs.org.

---

docs/*.rst are for us, developers. ctags.1 is for users.

e.g. I introduced some options to ctags. I wrote it to docs/*.rst. I didn't write it to ctags.1.
There are two reasons I did so.

1. Tecnical reviewing is needed. The new features including the options are not evaluated well.
   It can be immature. So I didn't write it to ctags.1. Once an immature option becomes popular
   it is difficult to withdraw the option. optlib related options should not go ctags.1. It is immature.
2. The quality of English is law. You may know what I woud like to say:-P.

---

ctags.1. is very related to making a release. As I wrote I don't have much interests about making a release because for making a release we have to update ctags.1. And updating ctags.1 requires reviewing features I randomly added. Which kind of tasks do you like: adding a new feature or reviewing added features?

I have many things I want you to know. But it is difficult to write down at once.
So could you do the following things?

1. Read the 4 documents:
   http://ctags.sourceforge.net/FORMAT, ctags.1, EXTENDING.html, docs/*.rst
2. Consider which one should be maintained at ReadTheDocs.org.
   I thinks docs/*.rst is for developers. Therefore it should be maintained at github.
   (optional) 3. Tell me kind of document do you want as a user and as a developer of ctags itself?

[masatake]

(#365, I reorganized docs/*.rst.)

[masatake]

It seems that I was wrong.
Can I ask you to rework on this topic again?
I recognized the overhead of mine is rather smaller things than losing chances for getting more volunteers.
I will follow your instruction.

[ffes]

Hi guys,

It has been a while... Life sometimes takes a lot more time than expected.

But on the subject: I recently converted the documentation of my NppTags plugin from DocBook to reStructuredText (far from perfect yet) and put them up on http://npptags.readthedocs.org

There is no need to split the sources and the docs. It is all in one repo.

All it takes is to make sure the docs can be properly generated used Python Sphinx. If we put a proper conf.py and a new Makefile in the docs directory, RTD will do the rest for you. We probably need to make separate sections for user documentation and the technical hacking docs. But that is simple as well. Just a well organized index.rst (or whatever name is chosen) and maybe put the rst files in two separate subdirectories of docs.

Maybe we should convert the ctags.1 to a rst file as well and let Sphinx generate the ctags.1 for us. It would make keeping the man page up to date much easier. Writing groff is hard, everyone can write rst.

But first I will try this week to convert my conf.py and see if I can generate the docs local.

[cweagans]

rst is pretty painful, IMO. I'd happily convert things to Markdown, which is much more widespread, especially on Github.

+1 to generating ctags.1 from a better format.

[ffes]

According to RTD co-founder Eric Holscher Markdown is more of HTML replacement and RST is a semantic language. For instance, It allows for proper references in docs.

Very interesting talk by him. IMHO 30 minutes well spend.
https://youtu.be/eWNiwMwMcr4 (from around 5 minutes it is about rst, 8 min = markdown vs rst)

And a handy cheatsheet.
http://sphinx-tutorial.readthedocs.org/cheatsheet/

[ffes]

I am preparing a PR that would solve the first part of this issue: Make sure the docs can be generated with Python Sphinx

But first #671 needs to be merged. I need to add anchors to of the some of rst files, I would like to avoid merge conflicts.

Shall I merge #671

[cweagans]

Last night, I started reorganizing the docs and converting them to Markdown. We really don't need all the features the RST provides. We get a lot out of switching to markdown, too. The process for working on the docs locally is really straightforward (1. install mkdocs - pip install mkdocs, 2. run mkdocs serve from the root of the ctags repo and open localhost:8000 in a browser - it even live updates as you save changes to markdown files in the docs/ directory), and the tools for doing markdown -> groff is pretty simple as well.

The biggest thing, though, is that Markdown is simple enough that you can't really do anything complicated. Given that docs are the place where people are least likely to contribute, it should be as simple as possible. Markdown files are simple enough to understand, and RTD adds a nice layer over the top to help navigate them (and references on RTD are very straightforward - I've got an example already done).

I'm willing to put in the effort to reorganize the docs into a nice structure and get things up and running on RTD, but I don't want to do it in RST because almost everything else I do is in Markdown (Slack, Github, Blog posts for Jekyll, just to name a few). As I said before, I'm even willing to document new features and such that are added through PRs if you just @ me when things need updated.

I've seen the talk. I just don't agree with it :)

[ffes]

I really think that rst is better for docs. So let's agree to disagree on md vs rst. ;)

Very curious what format @masatake wants the docs to be in. He writes most of the code and the docs. His opinion matters!

We both have a PR ready that allows us to put the docs on RTD.
We only need to decide on the format. Do we keep rst or convert to md?

[cweagans]

@ffes That's fair :)

I've said this a few times before, but just want to have it out here separate from my longer comment above:

I am willing to maintain all of the existing documentation + write new docs when needed so that you guys can focus on writing code :)

[cweagans]

FYI, here's what I've been working on so far: http://ctags.readthedocs.org/en/docs/

It's the docs branch in my fork.

[masatake]

Guys, thank you very much. I'm very proud of working with sincere hackers for the same goal.

Can I ask you to continue using rst format?

[ffes]

Good to have the ctags project name of RTD!

I see you have taken another approach then I have done so far. I just made sure the current "hacking guide" can be shown on RTD. Didn't do any of the other stuff yet.

[cweagans]

Yeah, my larger goal is to reorganize everything so that it's ready for public consumption when we release 1.0.0. The developer docs are nice, but we need user docs too, especially if we're going to generate ctags.1 from rst.

@ffes Can you open a PR with your changes?

[ffes]

You are absolutely right that we need user documentation as well. That is very important!

But my first goal was to get the "hacking docs" we have right now on RTD. They need to be reorganized to integrate the hacking doc and the user doc. With the conversion of ctags.1 to rst we have largely what we need.

FYI, I am working on converting ctags.1.in to ctags.1.rst.in but that take a bit more work. I could not find a decent tool to do the conversion for me so I'm doing it manually :-S

[cweagans]

@ffes I can do the man page conversion if that would help. Getting the docs structure in place is more important right now. FYI, varnish/hitch@d949327 might be a good starting point for an automated RST->groff conversion.

[masatake]

Following 3 docs are the most important.

1. format.rts, which is for people who writes a tool reading tags file
2. ctags.1, which is for people who runs ctags.
3. old-docs/EXTENDING.html, which is for peeple who writes a parser.
   (You can change the file name. e.g. tags-format.5.)

Consider all docs/.rst is just information source for making about 3 documents.
I will add more things docs/.rst. So I would like you to update 1. and 2.
If you need for updating the above important documents, you can tell me a topic
which I should add to docs/*.rst. I will make more garbage.

   3. is important but we may not have enough time for working on it.

[ffes]

@cweagans Any progress on setting up RTD, so we can close this one?

[cweagans]

@ffes http://docs.ctags.io :) I don't have access to set up a webhook in this repository, though. Can you do that? The URL to POST to is http://readthedocs.org/build/ctags.

[ffes]

I added the webhook "Service" for ReadtheDocs.

At this moment the hook has not been triggered yet. I will close this issue when it appears to work as expected.

All we need to do now is restructure and improve those docs ;-)

[masatake]

@ffes, I would like to see your ctags.rst.1.in.

[ffes]

The WIP version of November 2015 is here https://dl.dropboxusercontent.com/u/25917447/ctags.1.rst

I need to updated it with the current ctags.1.in. That appears to be only one commit.

[masatake]

@ffes, thank you very much. I would like to take over your work. I will integrate your ctags.1.rst to the source tree.

[ffes]

@masatake That's fine with me. Note that the file is WIP. Some changes from the the current master need to be merged in and I am not sure about the end result of the man page. It probably needs some tweaking as well. It was the first (and so far only) time I have been using rst to create a man page.

[masatake]

@ffes. I see. When merging ctags.1.rst, I would like you to review its "commit log".

[masatake]

Thank you guys.
I"m updating ctags.1.rst.in now.