Transform the current example gallery into a more docs-integrated example list

[eteq]

The sphinx example gallery, while pretty and useful in concept, has caused a fair amount of trouble. In particular it:

• Has led to various build problems due to bugs and/or version incompatibilities with sphinx or matplotlib
• Blurs the distinction between "tutorials" and "docs" in a way that leads to repeated questions of "where does this thing go?"
• Is not that "discoverable" for people who find their way to the "regular" part of the documentation (i.e., those that don't land directly on the example gallery)

So at the 2017 astropy coordination meeting we decided the way forward would be to change the example gallery into a list of pointers to examples that live elsewhere in the docs. In the process we would move the examples already there into either a relevant place in the docs, or out of the docs completely and into the Astropy tutorials.

I've already started on investigating how this might be done with some test code here: https://github.com/eteq/sphinx-example-docs-testing. However, the fact that I haven't gotten to it in several months demonstrate that I am more than happy to see someone else take it on (and am happy to consult if it would be helpful).

A bit more detail on this is available here, as it is being proposed as a project idea for GSoC 2018.

[eteq]

Cross-posting an additional suggestion from @sosey in #6589 (which we closed as these are duplicates):

@kelle @eteq You could put a link in the sphinx docs directly to a notebook with the example you want to show, this will pop it up in colab, which might be an interesting option for people:

https://colab.research.google.com/github/astropy/astropy-workshop

More info:
https://colab.research.google.com/github/googlecolab/colabtools/blob/master/notebooks/colab-github-demo.ipynb

[kelle]

@jonathansick

[jonathansick]

Got it! The GSoC description is pretty useful. Would you want the examples to be scoped within a reStructuredText directive? Or are you thinking more that it'd be a super reference target that marks the beginning of the example?

The first would look like:

.. astropy-example:: Demonstration of a tutorial
   :tags: tag-a, tag-b

   Tutorial content is here.

   1. A
   2. B
   3. C

versus

.. astropy-example:: Demonstration of a tutorial
   :tags: tag-a, tag-b

Tutorial content is here.

1. A
2. B
3. C

The nice thing about scoping content inside the directive is that we might be able to auto-select a figure that would populate a gallery view. I think that's what you folks are talking about when you say "gallery" right?

[kelle]

Jonathan and I had a lovely meeting where we talked this out and I think he has a pretty good idea of the vision. Goal is to have a MVP (minimally viable product) by Thurs, Apr 11 where I could come down to STScI to hash out with him and @eteq. Mark your calendars! :)

[jonathansick]

@eteq , @kelle, and I discussed the example gallery project further on 2019-04-11.

We discussed and agreed on the intent and design of the example gallery:

• The documentation consists of conceptual guides and references into addition to examples. Examples are actionable snippets that show how to accomplish different goals with Astropy. Our goal is to surface these examples to users more directly, rather than only having examples something that you stumble upon in the course of reading the full documentation.
• Examples are identified in the regular documentation text and cross-posted into the example gallery, as opposed to making the example gallery something that links into the original documentation. This does two things:
  • Existing documentation doesn't need to be substantially rewritten or reorganized to optimize example-oriented documentation. The existing documentation will remain as it is. The example gallery becomes a new view on that existing documentation.
  • The full example content is presented as a page in the example gallery so that the reader can focus on the example, and not get distracted by other content.
• Not every piece of documentation that involves a code block in the Astropy documentation makes sense as an example. We're fine with that, and won't force all content into the example gallery. At the same time, many pieces of content can be lightly rewritten or adjusted to become great examples.
• The examples don't need to be completely self-sufficient or runnable (for example, because there were imports earlier on the original documentation page). The extracted example code won't be tested. (We may want to put a notice on the examples encouraging readers to click through to the original documentation page if they do need that full context. We may need to think through the messaging and UX for this.)
• The example gallery itself will present all these examples from a single page (the index). If the example yields a plot, a small thumbnail of that plot should be included in the index page item/card for that example.
• Examples should be tagged and those tags should be surfaced on the index page to enable faceted browsing.

In terms of implementation, the following components need to be built:

• A restructured text directive that contains the content of the example in its original location in the documentation. The directive passes the content back into the original page, but also copies that content into Sphinx state, available to be inserted into the example gallery pages. This directive would also generate the reference target so that example pages can link back to the example.
• A Sphinx extension that generates the standalone example pages. We could discuss the implementation more specifically, but I think an approach that similar to automodapi might make sense here.
• A reStructuredText directive that inserts the example content into the example's page. This directive would also need to "clean" the example content of any target elements so that there aren't duplicate reStructuredText link targets.
• A Sphinx directive that populates the example index page with example items/cards. This would also need to generate the tag-based faceted browsing, but I think we all agree that beyond the MVP needed to prove the concept.
• Developer (contributor) documentation for how to mark up examples in documentation with these directives.
• As part of the development work, a few pages will be marked up using this system to demonstrate the concept.

I'm assuming that this code will go into the astropy-sphinx repository.

In terms of timeline, I'll be able to start this project in June. End date is September.

[pllim]

What is the status on this? Can we retire the example gallery in this repo? cc @saimn

[pllim]

Another year passed and I have not seen any progress over here. We should seek alternate approach. Example Gallery is rotting.