Moving the manual to Markdown

[MikeInnes]

As the last part of the docapocalypse / docapocalopalypse I'd like to move the manual to the much nicer Markdown format and into its own repo. Right now that work is over at JuliaManual.jl (happy to move to JuliaLang and bikeshed the name as appropriate*), and the markdown manual is auto-generated into the /doc directory.

Now that we have a shiny new markdown manual it's just a case of figuring out how to host it. How important are things like versioning and search? Can we live without versioning, and forward search to google? If so, it's a simple case of generating HTML for github pages. If not, things are more complicated and we need to figure out a sane route forward.

^{* JuliaMAN.jl might be complementary to our recent package additions}

[StefanKarpinski]

Docnarok?

[tkelman]

How important are things like versioning and search?

Very

Can we live without versioning

No

forward search to google

Maybe

simple case of generating HTML for github pages. If not, things are more complicated

Does mkdocs do versioning?

[ivarne]

Search is essential

[IainNZ]

By versioning do you mean 0.3 vs 0.4 docs? Because definitely need that. Search via Google could work, not sure though.

[MikeInnes]

Ok, it should be easy enough to do different minor versions of the docs. re searching,

https://encrypted.google.com/search?hl=en&q=site%3Adocs.julialang.org%2Fen%2Flatest%20iobuffer#hl=en&q=site:docs.julialang.org%2Fen%2Frelease-0.3+iobuffer

Just having a search box that redirects to that doesn't seem too awful. Other suggestions are welcome, of course.

@shashi How about Escher-backed interactive docs? We could just implement search in Julia.

[StefanKarpinski]

https://cse.google.com/cse/ ?

[shashi]

I'm all for using Escher for this, since this is a good real world use case it means to address. Escher can compile to HTML as of now, though a couple of features need to be figured out for this purpose e.g. a table of contents tree explorer. If run using a Julia server it can indeed be interactive. @one-more-minute @tkelman @IainNZ We can discuss this tomorrow if you guys are around.

[ViralBShah]

We can deploy the rich interactive documentation on JuliaBox. Heck, if we combine with Homework.jl, it can also become a Julia certification system!

[tkelman]

I'm around tomorrow, but I'm pretty sure the last web page I made was in FrontPage 97 so I doubt I'll be of much use here.

[jiahao]

What's the story with equations and references now? We have some in the manual and standard library reference and I would hate to lose them in the Markdown transition.

[tkelman]

@jiahao a lot of them were lost last night in the from-rst to to-rst transition

[jiahao]

That's really not the answer I was hoping to hear. I spent quite a lot of time writing those docs, and I had warned repeatedly about the lossy format transition. I'm really not pleased about losing them.

[MikeInnes]

@tkelman I haven't seen any missing equations and most references have formats like f() which are really easy to recognise and turn into links. Can you give more specific examples?

[tkelman]

By references you mean cross-references between different functions right? I think the hope was to possibly make them automatic somehow? Don't shoot the messenger.

There are some bibliographic citations which weren't lost, but the formatting was a bit disrupted. See #11906 for things that @hayd and I noticed while skimming through the super long diff. Most or all of the tables have been fixed already, code quoting and lists and citation formatting and x-refs are still to-be-fixed I believe. And restoring large lost chunks of math.rst

[MikeInnes]

Either way, I understand that a lot of work went into things like cross-references and will do my best to make sure we lose as little as possible in the transition. I can pull them out of the old RST files reasonably easily.

The issues are due to the (somewhat mysterious) conversion process used when the original helpdb.jl was generated, so we might have more luck generating the docstrings directly from the RST files.

[jiahao]

@tkelman that wasn't my intention.

@one-more-minute look for cross-references of the form

:xxx: func(examplevar) <func>

ReST supports arbitrary text in the link to some other function :func: or object :obj: or module :mod: etc, and I had found it necessary to use this form of cross-referencing in a few places.

I had also inserted many cross-references of the form :obj:SomeType which are cross-references to SomeType and do not render as SomeType(), but just SomeType. All of these cross-references appear to be gone now.

[mauro3]

Off topic: @tkelman that pond sure made my evening! Presumably the "living koi" are by now also "deceased koi", I wonder what their cause of death was...

[hayd]

Could we use https://github.com/MichaelHatherly/Lexicon.jl cc @MichaelHatherly

Some discussion: MichaelHatherly/Lexicon.jl#6

(xlink to my previous comment about alternative way to get help jl files)

[MichaelHatherly]

Could we use https://github.com/MichaelHatherly/Lexicon.jl

Lexicon won't be able to help with a rst->md conversion much. Also, I still hadn't gotten to sorting out how xrefs would work, so it's unlikely it would be that useful for what we need here. From the looks of it @one-more-minute's JuliaManual.jl is progressing quite rapidly already.

[cormullion]

Can you write definition lists (<dd><dl>) in Markdown? I was writing some Markdown the other day and couldn't work out how to do it. (I think the manual uses them extensively.)

[mauro3]

Does closing this issue mean that the manual stays in RST?