Big docs reorganization

This is not as big a change as it looks like, because it's taking the way the
docs were already organized by audience and making it explicit by creating
top-level dirs for each audience and moving all the docs into them.
Also cleans up a couple of loose ends where particular files didn't fit into
the organization-based scheme, fit them in. It then has to touch a lot of files
in doc/ to update Sphinx cross-references.

Create 8 top-level directories, one for each different audience:
User guide, sysadmin guide, maintainer's guide, API guide, extending guide,
theming guide, contributing guide and changelog.
(User guide, sysadmin guide and changelog are actually just files, not
directories, because they only need one file each.)

- New overview on docs front page explains the high-level structure

- Added a section to the documentation guidelines explaining the new structure

- Lots of feature pages moved into new maintaining/ dir

- Configuration.rst moved into maintaining/

- getting-started.rst moved into maintaining/

- Move Solr multicore appendix page into maintaining/,
  remove the appendix section itself as no longer needed

- Remove the "Features" index page, no longer needed as the pages just go into
  the "Maintainer's guide" now

- Create doc/maintaining/installing/ subdir and move the installing and
  deployment docs into it

- Create doc/maintaining/upgrading/ subdir and move the upgrading docs into
  it

- Move api.rst and legacy-api.rst into new api/ dir

- Move test.rst into contributing/

- Fix Sphinx references all over the place to point to the new file locations

CHANGELOG.rst

@@ -188,7 +188,7 @@ Note: This version requires a database upgrade
 
 Note: This version requires a Solr schema upgrade
 
-Organizations based authorization (see :doc:`authorization`):
+Organizations based authorization (see :doc:`/maintaining/authorization`):
  CKAN's new "organizations" feature replaces the old authorization system
  with a new one based on publisher organizations. It replaces the "Publisher
  Profile and Workflow" feature from CKAN 1.X, any instances relying on it will
@@ -200,7 +200,7 @@ Organizations based authorization (see :doc:`authorization`):
  * New authorization ini file options
 
 
-New frontend (see :doc:`theming/index`):
+New frontend (see :doc:`/theming/index`):
  CKAN's frontend has been completely redesigned, inside and out. There is
  a new default theme and the template engine has moved from Genshi to
  Jinja2. Any custom templates using Genshi will need to be updated, although

ckan/logic/__init__.py

@@ -526,8 +526,8 @@ def side_effect_free(action):
     '''A decorator that marks the given action function as side-effect-free.
 
     Action functions decorated with this decorator can be called with an HTTP
-    GET request to the :doc:`Action API </api>`. Action functions that don't
-    have this decorator must be called with a POST request.
+    GET request to the :doc:`Action API </api/index>`. Action functions that
+    don't have this decorator must be called with a POST request.
 
     If your CKAN extension defines its own action functions using the
     :py:class:`~ckan.plugins.interfaces.IActions` plugin interface, you can use
@@ -559,9 +559,9 @@ def auth_sysadmins_check(action):
     to call an action function.
 
     Normally sysadmins are allowed to call any action function (for example
-    when they're using the :doc:`Action API </api>` or the web interface),
-    if the user is a sysadmin the action function's authorization function
-    will not even be called.
+    when they're using the :doc:`Action API </api/index>` or the web
+    interface), if the user is a sysadmin the action function's authorization
+    function will not even be called.
 
     If an action function is decorated with this decorator, then its
     authorization function will always be called, even if the user is a

ckan/plugins/interfaces.py

@@ -770,12 +770,12 @@ def read_template(self):
 
         If the user requests the dataset in a format other than HTML
         (CKAN supports returning datasets in RDF or N3 format by appending .rdf
-        or .n3 to the dataset read URL, see :doc:`/linked-data-and-rdf`) then
-        CKAN will try to render
-        a template file with the same path as returned by this function,
-        but a different filename extension, e.g. ``'package/read.rdf'``.
-        If your extension doesn't have this RDF version of the template
-        file, the user will get a 404 error.
+        or .n3 to the dataset read URL, see
+        :doc:`/maintaining/linked-data-and-rdf`) then CKAN will try to render a
+        template file with the same path as returned by this function, but a
+        different filename extension, e.g. ``'package/read.rdf'``.  If your
+        extension doesn't have this RDF version of the template file, the user
+        will get a 404 error.
 
         :rtype: string
 

doc/api.rst → doc/api/index.rst

@@ -1,6 +1,9 @@
-============
-The CKAN API
-============
+=========
+API guide
+=========
+
+This section documents CKAN's API, for developers who want to write code that
+interacts with CKAN sites and their data.
 
 CKAN's **Action API** is a powerful, RPC-style API that exposes all of CKAN's
 core features to API clients. All of a CKAN website's core functionality
@@ -39,8 +42,8 @@ code that calls the CKAN API.  For example, using the CKAN API your app can:
 
    CKAN's FileStore and DataStore have their own APIs, see:
 
-   * :doc:`filestore`
-   * :doc:`datastore`
+   * :doc:`/maintaining/filestore`
+   * :doc:`/maintaining/datastore`
 
 .. note::
 

doc/legacy-api.rst → doc/api/legacy-api.rst

@@ -370,7 +370,7 @@ The ``Dataset`` and ``Revision`` data formats are as defined in `Model Formats`_
 
  Only public datasets can be accessed via the legacy search API, regardless of
  the provided authorization. If you need to access private datasets via the
- API you will need to use the `package_search` method of the :doc:`api`.
+ API you will need to use the `package_search` method of the :doc:`index`.
 
 
 **Resource Parameters**

doc/contents.rst

@@ -7,15 +7,9 @@ Full table of contents
    index
    user-guide
    sysadmin-guide
-   installing
-   upgrading
-   getting-started
-   features
+   maintaining/index
+   api/index
    extensions/index
    theming/index
-   api
    contributing/index
-   test
-   configuration
    changelog
-   appendices/index

doc/contributing/architecture.rst

@@ -41,7 +41,7 @@ Action functions are exposed in the API
 ---------------------------------------
 
 The functions in ``ckan.logic.action`` are exposed to the world as the
-:doc:`/api`.  The API URL for an action function is automatically generated
+:doc:`/api/index`.  The API URL for an action function is automatically generated
 from the function name, for example
 ``ckan.logic.action.create.package_create()`` is exposed at
 ``/api/action/package_create``. See `Steve Yegge's Google platforms rant
@@ -50,7 +50,7 @@ interesting discussion about APIs.
 
 **All** publicly visible functions in the
 ``ckan.logic.action.{create,delete,get,update}`` namespaces will be exposed
-through the :doc:`/api`. **This includes functions imported** by those
+through the :doc:`/api/index`. **This includes functions imported** by those
 modules, **as well as any helper functions** defined within those modules.  To
 prevent inadvertent exposure of non-action functions through the action api,
 care should be taken to:

doc/contributing/documentation.rst

@@ -122,17 +122,48 @@ Once your docs are ready to submit to the CKAN project, follow the steps in
 
 .. _structure:
 
--------------------------
-2. Structure and audience
--------------------------
-
-:doc:`/index` describes the overall structure of the docs, and the intended
-audience for each part. This structure is intended to be clear, simple and
-extendable.  If you're adding a new section to the docs, try to fit it into
-this structure.
+-----------------------------
+2. How the docs are organized
+-----------------------------
+
+It's important that the docs have a clear, simple and extendable structure
+(and that we keep it that way as we add to them), so that both readers
+and writers can easily answer the questions:
+If you need to find the docs for a particular feature, where do you look?
+If you need to add a new page to the docs, where should it go?
+
+As :doc:`/index` explains, the documentation is organized into several guides,
+each for a different audience: a user guide for web interface users, an
+extending guide for extension developers, a contributing guide for core
+contributors, etc. These guides are ordered with the simplest guides first,
+and the most advanced last.
+
+In the source, each one of these guides is a subdirectory with its own
+``index.rst`` containing its own ``.. toctree::`` directive that lists all of
+the other files in that subdirectory. The root toctree just lists each of these
+``*/index.rst`` files.
+
+When adding a new page to the docs, the first question to ask yourself is: who
+is this page for? That should tell you which subdirectory to put your page in.
+You then need to add your page to that subdirectory's ``index.rst`` file.
+
+Within each guide, the docs are broken up by topic. For example, the extending
+guide has a page for the writing extensions tutorial, a page about testing
+extensions, a page for the plugins toolkit reference, etc. Again, the topics
+are ordered with the simplest first and the most advanced last, and reference
+pages generally at the very end.
+
+:doc:`The changelog </changelog>` is one page that doesn't fit into any of
+the guides, because it's relevant to all of the different audiences and not
+only to one particular guide. So the changelog is simply a top-level page
+on its own. Hopefully we won't need to add many more of these top-level
+pages. If you're thinking about adding a page that serves two or more audiences
+at once, ask yourself whether you can break that up into separate pages and
+put each into one of the guides, then link them together using `seealso`_
+boxes.
 
 Within a particular page, for example a new page documenting a new feature, our
-suggestion for what sections the page should have is:
+suggestion for what sections the page might have is:
 
 #. **Overview**: a conceptual overview of or introduction to the feature.
    Explain what the feature provides, why someone might want to use it,
@@ -154,16 +185,15 @@ suggestion for what sections the page should have is:
 Subdirectories
 ==============
 
-Many sections of the docs are organized into subdirectories. For example,
-there's a ``doc/extensions`` subdirectory with
-:doc:`its own index file </extensions/index>` and table of contents, grouping
-all the docs about extensions together into one topic. The
-:doc:`contributing docs </contributing/index>`, theming docs, and other
-sections of the docs are the same.
+Some of the guides have subdirectories within them. For example
+:doc:`/maintaining/index` contains a subdirectory
+:doc:`/maintaining/installing/index`
+that collects together the various pages about installing CKAN with its own
+``doc/maintaining/installing/index.rst`` file.
 
-While using subdirectories is useful, we recommend that you
-**don't put further subdirectories inside the subdirectories**, keep it to
-one level of subdirectories inside the ``doc`` directory, keep it simple,
+While subdirectories are useful, we recommend that you **don't put further
+subdirectories inside the subdirectories**, try to keep it to at most two
+levels of subdirectories inside the ``doc`` directory. Keep it simple,
 otherwise the structure becomes confusing, difficult to get an overview of and
 difficult to navigate.
 
@@ -441,13 +471,13 @@ Cross-referencing to CKAN config settings
 -----------------------------------------
 
 Whenever you mention a CKAN config setting, make it link to the docs for that
-setting in :doc:`/configuration` by using ``:ref:`` and the name of the config
+setting in :doc:`/maintaining/configuration` by using ``:ref:`` and the name of the config
 setting::
 
   :ref:`ckan.site_title`
 
 This works because all CKAN config settings are documented in
-:doc:`/configuration`, and every setting has a Sphinx label that is exactly
+:doc:`/maintaining/configuration`, and every setting has a Sphinx label that is exactly
 the same as the name of the setting, for example::
 
     .. _ckan.site_title:
@@ -464,7 +494,7 @@ the same as the name of the setting, for example::
     This sets the name of the site, as displayed in the CKAN web interface.
 
 If you add a new config setting to CKAN, make sure to document like this it in
-:doc:`/configuration`.
+:doc:`/maintaining/configuration`.
 
 
 Cross-referencing to a Python object
@@ -649,8 +679,8 @@ seealso
 =======
 
 Often one page of the docs is related to other pages of the docs or to external
-pages. A `seealso <http://sphinx-doc.org/markup/para.html?highlight=seealso#directive-seealso>`_
-block is a nice way to include a list of related links::
+pages. A `seealso block <http://sphinx-doc.org/markup/para.html?highlight=seealso#directive-seealso>`_
+is a nice way to include a list of related links::
 
     .. seealso::
 

doc/contributing/index.rst

@@ -1,13 +1,13 @@
-====================
-Contributing to CKAN
-====================
+==================
+Contributing guide
+==================
 
 .. _CKAN repo on GitHub: https://github.com/okfn/ckan
 .. _docs.ckan.org: http://docs.ckan.org
 
 CKAN is free open source software and contributions are welcome, whether
 they're bug reports, source code, documentation or translations. The following
-documents will walk you through our processes for making different kinds
+sections will walk you through our processes for making different kinds
 of contributions to CKAN:
 
 .. If editing this, note that these are roughly ordered with the most simple
@@ -20,6 +20,8 @@ of contributions to CKAN:
    issues
    i18n
 
+   test
+
    commit-messages
    pull-requests
    reviewing

doc/contributing/release-process.rst

@@ -7,7 +7,7 @@ new CKAN release.
 
 .. seealso::
 
-   :doc:`/upgrading`
+   :doc:`/maintaining/upgrading/index`
      An overview of the different kinds of CKAN release, and the process for
      upgrading a CKAN site to a new version.
 

doc/contributing/upgrading-dependencies.rst

@@ -50,7 +50,7 @@ work with CKAN:
 #. Install the development dependencies: ``pip install -r
    dev-requirements.txt``
 
-#. Run the tests to make sure everything still works (see :doc:`/test`).
+#. Run the tests to make sure everything still works (see :doc:`test`).
 
    - If not, try to fix the problem. If it's too complicated, pinpoint which
      dependency's version broke our tests, find an older version that still

doc/extensions/index.rst

@@ -1,26 +1,30 @@
-=======================
-Writing CKAN extensions
-=======================
+===============
+Extending guide
+===============
 
-CKAN can be modified and extended using extensions. Some **core extensions**
-come packaged with CKAN. Core extensions don't need to be installed before you
-can use them as they're installed when you install CKAN, they can simply be
-enabled by following the setup instructions in each extension's documentation
-(some core extensions are already enabled by default). For example, the
-:doc:`datastore extension </datastore>`, :doc:`multilingual extension
-</multilingual>`, and :doc:`stats extension </stats>` are all core extensions,
-and the :doc:`data viewer </data-viewer>` also uses core extensions to enable
-data previews for different file formats.
+The following sections will teach you how to customize and extend CKAN's
+features by developing your own CKAN extensions.
 
-**External extensions** are CKAN extensions that don't come packaged with
-CKAN, but must be downloaded and installed separately. A good place to find
-external extensions is the
-`list of extensions on the CKAN wiki <https://github.com/okfn/ckan/wiki/List-of-extensions>`_.
-Again, follow each extension's own documentation to install, setup and use the
-extension.
+.. seealso::
 
-This document covers everything you need to know to write your own CKAN
-extensions.
+    Some **core extensions** come packaged with CKAN. Core extensions don't
+    need to be installed before you can use them as they're installed when you
+    install CKAN, they can simply be enabled by following the setup
+    instructions in each extension's documentation (some core extensions are
+    already enabled by default). For example, the :doc:`datastore extension
+    </maintaining/datastore>`, :doc:`multilingual extension
+    </maintaining/multilingual>`, and :doc:`stats extension
+    </maintaining/stats>` are all core extensions, and the :doc:`data viewer
+    </maintaining/data-viewer>` also uses core extensions to enable data
+    previews for different file formats.
+
+.. seealso::
+
+    **External extensions** are CKAN extensions that don't come packaged with
+    CKAN, but must be downloaded and installed separately. A good place to find
+    external extensions is the `list of extensions on the CKAN wiki
+    <https://github.com/okfn/ckan/wiki/List-of-extensions>`_.  Again, follow
+    each extension's own documentation to install, setup and use the extension.
 
 .. toctree::
    :maxdepth: 2

doc/extensions/testing-extensions.rst

@@ -2,7 +2,7 @@ Testing extensions
 ==================
 
 CKAN extensions can have their own tests that are run using ``nosetests``
-in much the same way as running CKAN's own tests (see :doc:`/test`).
+in much the same way as running CKAN's own tests (see :doc:`/contributing/test`).
 
 Continuing with our :doc:`example_iauthfunctions extension <tutorial>`,
 first we need a CKAN config file to be used when running our tests.
@@ -18,8 +18,8 @@ your CKAN source directory, relative to your ``test.ini`` file).
 
 The ``test.ini`` file is a CKAN config file just like your |development.ini|
 and |production.ini| files, and it can contain any
-:doc:`CKAN config file settings </configuration>` that you want CKAN to use
-when running your tests, for example::
+:doc:`CKAN config file settings </maintaining/configuration>` that you want
+CKAN to use when running your tests, for example::
 
     [app:main]
     use = config:../ckan/test-core.ini