Documentation System #8791

Merged
merged 15 commits into from Nov 10, 2014

Projects

None yet
@MikeInnes
Member

A metadata system for global objects + pretty markdown parsing and display (see the readme). For example:

@doc """
... flowing rivers of technical poetry ...
""" ->
function foo() ...

Documenting multiple methods of a function will concatenate the doc strings together, in the order the methods were originally defined, and this is overloadable.

By default @doc treats unadorned strings as equivalent to md"" (i.e. markdown syntax) but you can actually put anything before the arrow to associate it with the function.

Docs can be queried with e.g.

doc(Docs.doc)
@doc Docs.doc
@doc @doc # For macros
@doc text""

(and you can try these out on this branch – the repl help key ? will work too)

General Todos:

  • Prettier syntax ("..." -> @doc doc"..." ->)
  • Add Docs.@init call to module init code
  • REPL ? integration
  • Make sure exports are present and correct
  • Move away from the current help system and deprecate it
  • Fix the segfault when Text docs are uncommented (!?)
@MikeInnes MikeInnes added the doc label Oct 23, 2014
@ViralBShah
Member

It seems to me that not all the TODOs need to be done before we merge this. It would be great if we can have something that exists along with the existing help for a bit, while we transition and fix everything and deprecate the existing system.

@MikeInnes
Member

I agree completely – the TODOs are more general things that would be nice to have eventually, but sure, this is basically usable as is.

@mauro3
Contributor
mauro3 commented Oct 24, 2014

I found that it was quite easy to just parse the old helpdb.jl file, put its contents into the new help storage and have the new help-system displaying 99% of what there was before:
https://github.com/mauro3/julia/blob/metadata/base/help.jl#L89

@hayd hayd referenced this pull request Nov 7, 2014
Closed

Fix help for .+ in the REPL #8922

@ViralBShah
Member

I suggest merging this and then taking it from there.

@MikeInnes
Member

+1

@johnmyleswhite
Member

Merging something (or the threat of merging something) does seem like the best way to make progress.

@StefanKarpinski
Member

I'm a little concerned about packages starting to use this in a way that doesn't degrade gracefully, but I can get over that fear for the sake of making forward progress. Go for it.

@ViralBShah
Member

This is exactly what we want to sort out in the middle of the dev cycle. @one-more-minute Please merge when you are ready.

We may want to start using in a few packages and then improve from there.

@ViralBShah
Member

Perhaps squash some of the commits too?

@timholy
Member
timholy commented Nov 10, 2014

I would say squashing isn't necessary for merging this. (If there were intermediate breakages I might think differently, but that's a lot of green checkmarks above.)

@MikeInnes
Member

That's OK, I was due to learn some git rebasing anyway. The history should be a lot cleaner now.

@MikeInnes MikeInnes merged commit d43dfa8 into master Nov 10, 2014

1 check was pending

continuous-integration/travis-ci The Travis CI build is in progress
Details
@MikeInnes
Member

🎆

Ok, try ?@doc at the REPL and then get going writing some great docs ;)

@mauro3
Contributor
mauro3 commented Nov 10, 2014

YEAH!

@tonyhffong tonyhffong referenced this pull request in tonyhffong/Lint.jl Nov 10, 2014
@tonyhffong tonyhffong update lint of at-doc syntax 18379a2
@timholy
Member
timholy commented Nov 10, 2014

Without a doubt, this is one of the best things about julia 0.4. 🍰

@ivarne
Contributor
ivarne commented Nov 10, 2014

@timholy How can that be? It isn't even mentioned in NEWS.md.

[I hope the irony is obvious]

@MikeInnes
Member

#8962 :)

@milktrader
Member

I love the description of the foo function in @doc @doc. I always wondered what the main purpose of that function was. 😃

@MichaelHatherly
Member

Awesome! 📚

@stevengj stevengj added a commit that referenced this pull request Nov 10, 2014
@stevengj stevengj added NEWS for #8791 54c54a2
@MichaelHatherly
Member

@one-more-minute do you happen to know how much of http://spec.commonmark.org/0.11/ you've covered so far? (If the intention is to follow that spec?)

@ntessore
Contributor

Great, this was one of the mostly missed features for me. One question: does the first item in the todo list mean we will get rid of the -> at some point?

Edit: I see now this is the point of #8965, never mind.

@MikeInnes
Member

@MichaelHatherly Definitely, although most likely we'll have some documentation-specific extensions such as LaTeX and tables. It'd be great to get a test suite for CommonMark running, though.

Basically, at the moment basic stuff (headers, emphasis, links) is well supported, but more unusual things like nested lists / the lazy rule for block quotes aren't. It should be fine for basic uses, though.

Markdown.jl is meant to be easy to hack on and extend – if you're interested, PRs would be very welcome, and I'll make sure Base stays in sync with the main repo.

@MichaelHatherly
Member

@one-more-minute great, I'll probably get a chance to in a few weeks time.

@MikeInnes
Member

Awesome, can't wait to see what you come up with

@StefanKarpinski
Member

This is very exciting. Thank you all for pushing this through.

@milktrader
Member

I guess now is a good time to bring up the matter of standardizing headers. For comparison, R has the following ...

Description 
...
Usage
...
Arguments
...
Details
...
References
...
See Also
...
@stevengj
Member
@MikeInnes
Member

@StefanKarpinski Glad to help!

@MikeInnes
Member

I added some really basic documentation examples in basedocs.jl – hope that's OK. It's not meant to be final in any way so much as to get something going, and give people something to try out in the repl (e.g. you can do ?fft, @time, ?r"" etc. now) to get a feel for it.

@mauro3
Contributor
mauro3 commented Nov 11, 2014

I'm using it over in https://github.com/mauro3/Traits.jl, works a treat!

@MikeInnes
Member

That's great to see 👍

@timholy
Member
timholy commented Nov 12, 2014

One of the really sweet things @MichaelHatherly added to Lexicon was the ability to dispatch on help. I confess that due to other commitments I haven't followed this effort at all; is that functionality supported?

@MikeInnes
Member

I actually just added this (3c2e2c4):

julia> @doc "foo1" foo() = 1
foo (generic function with 1 method)

julia> @doc "foo2" foo(x) = 2
foo (generic function with 2 methods)

help?> foo()
  foo1

help?> foo(2)
  foo2
@MikeInnes
Member

Ok, I see there's also the idea of having a searching version of @which, which is pretty neat. How well does it work? I'd definitely be interested in integrating something like that.

@timholy
Member
timholy commented Nov 12, 2014

I haven't tested it extensively (I'm only using Docile/Lexicon in a subset of my private code tree, basically the stuff that was developed after these packages became available). But everything I've tried so far suggests it's the cat's meow.

Specifically, I was wondering about this for ? or @doc, not for @which (although implementing for @which would be awesome too).

@MikeInnes
Member

I only mention @which because it seems like a good idea to implement a general @whiches (or whatever) which returns the array of methods which fuzzy-match – then adding support to ? in the repl should be pretty easy.

That's only an implementation detail, though – I'm definitely on board with fuzzy searching for docs in general. @MichaelHatherly So you have this to working in Docile/Lexicon already?

@MichaelHatherly
Member

The query(::Function, (DataType...)) method in Lexicon does left-to-right partial matching, since I found doing an unordered match on the type signatures typically gave more results than was helpful. The @query macro does exact method matching in the same way as @which.

Having a more intelligent fuzzy-matcher could be quite nice. Not sure what it would look like though.

@sglyon sglyon referenced this pull request in QuantEcon/QuantEcon.jl Nov 19, 2014
Closed

Support for reducible Markov chains #20

@waTeim waTeim added a commit to waTeim/julia that referenced this pull request Nov 23, 2014
@stevengj @waTeim stevengj + waTeim added NEWS for #8791 8d62ccc
@gummif gummif referenced this pull request in JuliaDSP/Roadmap Nov 24, 2014
Closed

Commenting & documentation #6

@MikeInnes MikeInnes deleted the omm/doc-system branch Dec 10, 2014
This was referenced Dec 20, 2014
@ViralBShah
Member

Pushing forward on completing the transition - it seems that the next step is to transition help to the new documentation system.

Does this mean that our standard library documentation should be translated to Markdown first to leverage the new system best? This may not work best with readthedocs - does it have to use rst?

@MikeInnes
Member

I think readthedocs requires RST, yes. That said, we already have our own docs template and website, so I'm not sure readthedocs really buys us that much – @MichaelHatherly has done a lot of work on generating great documentation pages and might be interested in helping to move away from it entirely.

OTOH, it wouldn't be hard to generate RST manual pages from the inline docs using Markdown.jl, so that's another option.

The important thing is to have some way to share content between Base docs and the manual – otherwise there'll be a lot of repeated work.

@MichaelHatherly
Member

There's also http://www.mkdocs.org/, which @tshort has been using recently (https://tshort.github.io/Sims.jl/) in conjunction with Docile.

@ViralBShah
Member

I was thinking the same about readthedocs. The manual and Base docs should just be equally accessible, but they don't share much of anything. They can be done in different ways without impacting anything, I feel.

mkdocs looks really cool!

@tshort
Member
tshort commented Jan 11, 2015

Embedded Docile/Base 0.4 docs can be used with Mkdocs and published on either readthedocs or GitHub Pages. It's really easy.

Github Pages works great with Mkdocs. I haven't tried it, but one advantage of readthedocs is it provides online searching.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment