Improve documentation

[Technologicat]

Over time, the feature set of unpythonic has become larger than can be briefly explained in a simple README. The documentation needs rethinking.

EDIT: As of v0.15.0, some of this has been done. See at the end of this thread for the current TODOs.

---

• Can we shorten the text without losing important information? Especially this is a #helpwanted.
• Within the explanation of each feature, the main text could use copyediting. Important points first, details after.
  • E.g. as of 0.14.1, in the documentation of fupdate, the fup function, which is the recommended everyday API, is currently only mentioned as an afterthought, after explaining the low-level fupdate API in detail. (Well, it came first, but documentation should follow logic, not history.) This particular point is fixed in 0.14.2, but more similar issues may remain.
• Make docstrings and the separate documentation complement each other.
  • Right now there's quite a lot of duplication between them, often with one of them being an older revision of the text in the other one (often the separate documentation is more polished than the docstring).
  • A policy of orthogonal roles might help, such as:
    • A docstring is brief, to the point. Its role is to give the semantic and usage details of a particular feature (with examples), briefly explain any caveats, and mention any relevant see-alsos (at least when not obvious).
    • The separate documentation is the user manual, with the usual bells and whistles such as a TOC with section links, and pretty formatting (as far as reasonably possible in Markdown).
      • The current documentation is already a rough attempt at this.
      • Ideally, we could use a narrative format, like Pyramid's?
      • How to optimize the documentation for discoverability of features?
      • How far to go in explaining the use cases of each feature?
        • The whole point of unpythonic is to do unpythonic things, pythonically. Some of the features may seem weird to readers not familiar with the particular sources of inspiration, until given the context: what the problem solved by a particular feature is.
        • Most features are useful in production, while there are a few that are primarily useful for teaching concepts (continuations, lazify). These should probably be approached differently.
          • The teaching features aim at robustness, too, but I'm not completely sure I want a production codebase that claims to be Python, yet uses call-by-need semantics. Forming an informed opinion on things like this first requires some prototyping with smaller projects.
        • On the other hand, for example the general promotion of FP is beyond the scope of unpythonic, even though it matters; particularly, FP enables building complex functionality by composition, and precisely this is where curry comes in useful.

---

Previous, already done items:

• DONE: README should play the role of a short tour-type overview. Done in 0.14.2.
  • Currently both READMEs are long, which makes them scary. But that's because they contain the motivation and examples for each and every feature, no matter how minor.
  • The macro tour could live in a separate README (just like the macro documentation already does), so users who don't want to depend on MacroPy don't have to wade through what is easily 50% of the total documentation. Just mention it at the beginning, like we already do.
  • Thanks to aisha-w for moving the pure-Python API docs to doc/features.md, which is a much better place for that much detail. Version 0.14.2 adds a short demo to the README to give a short overview.
• DONE: The detailed documentation could live in separate files. Contributed by @aisha-w, see Separate documentation files. #28.
  • This would allow having a relatively stable front-page tour, only needing an update when new major features are added.
  • Keep the authoring light; just use .md or .rst.
• DONE: Clean up the history clutter. The 0.14.1 release has been out for a while. Contributed by @aisha-w, see Separate documentation files. #28.
  • Integrate the information from the "changed in vx.yy" notes into the main text.
  • Drop the "added in vx.yy" notes.
  • Add one short note at the start of the README, documenting this change. Not really needed.

[Technologicat]

(As for good first issue, obviously every small bit helps. This needs to be split into sub-issues when tackled, anyway.)

[Technologicat]

Updated TODO list in the initial comment above.

[Technologicat]

Updated again. Cleaned-up version of remaining TODOs:

• Within the explanation of each feature, the main text could use copyediting. Important points first, details after.
• We need a clearer division between what goes into docstrings and what goes into the separate documentation. Right now there's quite a lot of duplication between them.
• Can we shorten the text without losing important information? Especially this is a #helpwanted.

[Technologicat]

There are four kinds of documentation:

• Tutorials
  • Learning-oriented, practical steps, most useful when studying
• How-to guides
  • Problem-oriented, practical steps, most useful when working
• Explanation
  • Understanding-oriented, theoretical knowledge, most useful when studying
• Reference
  • Information-oriented, theoretical knowledge, most useful when working

Keeping these separate benefits clarity, makes it easier for the reader to find what they need, and makes for faster reading.

As of 0.15.0, the user manuals of unpythonic (mainly doc/features.md and doc/macros.md) mix several of these kinds in one document. In the future, we should look into detangling the different kinds.

• We definitely need an API reference.
  • This could be generated with Sphinx, by reformatting all docstrings (probably >10k lines total) into Sphinx format. (NOTE: Remember to update the contribution guidelines accordingly.)
  • Using Sphinx motivates making the docstrings the definitive reference resource, avoiding duplication.
  • NOTE: As of 0.15.0, for some features the docstring contains the most readable language, but for other features the better one could be in the narrative in the user manual. Need to check each and every one of them, and get the most readable one into the docstring, before removing reference-like text from the rest of the docs.
• Since unpythonic is a kitchen-sink language extension, and one of its goals is teaching and learning, we should have an explanation of the concepts foreign to Python.
• Since the features are conceptually separate, and most of them are simple to use, the need for how-to guides is minor, but still there:
  • How to achieve modular error handling with the conditions-and-restarts system.
  • How to combo the block macros (the xmas tree combo).
  • How to make a dialect using mcpyrate.
  • Anything I've missed?
• Do we need tutorials?