[Docs] Reorganise and add sections from wiki

- Change the layout and contents of docs to be better organised and
  follow ideas from: https://www.divio.com/blog/documentation/
- Use markdown for non-technical documents to speed up writing.
- Added new sections and imported documents from Trac wiki.

Build fixes:

- Added a patch to fix recommonmark 0.4 and doc referencing:
    readthedocs/recommonmark#93
- Set docs build in tox to Py2.7 since there are problems with autodoc
  mocking multiple inheritance on Python 3 resulting in metaclass errors.
- Supressed warning about `modules.rst` not in the toctree by creating
  a static `modules.rst` with `:orphan:` file directive and add to git.
  Also skip creating this toc file with sphinx-apidoc in setup and tox.
- Simplified finding exported RPC and JSON API methods by adding an
  autodoc custom class directive. Removed unneeded __rpcapi.py.

.gitignore

@@ -2,7 +2,7 @@
 build
 .cache
 dist
-docs/source/modules
+docs/source/modules/deluge*.rst
 *egg-info
 *.egg
 *.log

deluge/ui/web/json_api.py

@@ -615,6 +615,7 @@ def _on_torrent_status(self, torrent, d):
 
     @export
     def get_torrent_status(self, torrent_id, keys):
+        """Get the status for a torrent, filtered by status keys."""
         main_deferred = Deferred()
         d = component.get('SessionProxy').get_torrent_status(torrent_id, keys)
         d.addCallback(self._on_torrent_status, main_deferred)
@@ -694,6 +695,7 @@ def get_torrent_info(self, filename):
 
     @export
     def get_magnet_info(self, uri):
+        """Parse a magnet URI for hash and name."""
         return get_magnet_info(uri)
 
     @export
@@ -915,14 +917,17 @@ def get_plugins(self):
 
     @export
     def get_plugin_info(self, name):
+        """Get the details for a plugin."""
         return component.get('Web.PluginManager').get_plugin_info(name)
 
     @export
     def get_plugin_resources(self, name):
+        """Get the resource data files for a plugin."""
         return component.get('Web.PluginManager').get_plugin_resources(name)
 
     @export
     def upload_plugin(self, filename, path):
+        """Upload a plugin to config."""
         main_deferred = Deferred()
 
         shutil.copyfile(path, os.path.join(get_config_dir(), 'plugins', filename))

docs/source/conf.py

@@ -16,13 +16,28 @@
 from datetime import date
 
 import pkg_resources
+from recommonmark.states import DummyStateMachine
 from recommonmark.transform import AutoStructify
+from sphinx.ext.autodoc import ClassDocumenter, bool_option
 
 try:
     from ...version import get_version
 except ImportError:
     get_version = None
 
+
+# Monkey patch
+class PatchDummyStateMachine(DummyStateMachine):
+    """Fix recommonmark 0.4 doc reference issues."""
+
+    def run_role(self, name, *args, **kwargs):
+        if name == 'doc':
+            name = 'any'
+        return DummyStateMachine.run_role(self, name, *args, **kwargs)
+
+
+DummyStateMachine = PatchDummyStateMachine
+
 # Must add these for autodoc to import packages successully
 __builtin__.__dict__['_'] = lambda x: x
 __builtin__.__dict__['_n'] = lambda s, p, n: s if n == 1 else p
@@ -271,9 +286,18 @@ def __or__(self, __):
 # If false, no module index is generated.
 # latex_use_modindex = True
 
+# Register an autodoc class directive to only include exported methods.
+ClassDocumenter.option_spec['exported'] = bool_option
+
+
+def maybe_skip_member(app, what, name, obj, skip, options):
+    if options.exported and not (
+        hasattr(obj, '_rpcserver_export') or hasattr(obj, '_json_export')
+    ):
+        return True
+
 
 def setup(app):
-    app.add_config_value(
-        'recommonmark_config', {'auto_toc_tree_section': 'Contents'}, True
-    )
+    app.connect('autodoc-skip-member', maybe_skip_member)
+    app.add_config_value('recommonmark_config', {}, True)
     app.add_transform(AutoStructify)

docs/source/contributing/code.md

@@ -0,0 +1,115 @@
+# Contributing code
+
+## Basic requirements and standards
+
+- A [new ticket](http://dev.deluge-torrent.org/newticket) is required for bugs
+  or features. Search the ticket system first, to avoid filing a duplicate.
+- Ensure code follows the [syntax and conventions](#syntax-and-conventions).
+- Code must pass tests. See [testing.md] for
+  information on how to run and write unit tests.
+- Commit messages are informative.
+
+## Pull request process:
+
+- Fork us on [github](https://github.com/deluge-torrent/deluge).
+- Clone your repository.
+- Create a feature branch for your issue.
+- Apply your changes:
+  - Add them, and then commit them to your branch.
+  - Run the tests until they pass.
+  - When you feel you are finished, rebase your commits to ensure a simple
+    and informative commit log.
+- Create a pull request on github from your forked repository.
+  - Verify that the tests run by [Travis-ci](https://travis-ci.org/deluge-torrent/deluge)
+    are passing.
+
+## Syntax and conventions
+
+### Code formatters
+
+We use two applications to automatically format the code to save development
+time. They are both run with pre-commit.
+
+#### Black
+
+- Python
+
+#### Prettier
+
+- Javascript
+- CSS
+- YAML
+- Markdown
+
+### Common
+
+- Line length: `79` chars.
+- Indent: `4 spaces`, no tabs.
+- All code should use `'single quotes'`.
+
+### Python
+
+We follow [PEP8](http://www.python.org/dev/peps/pep-0008/) and
+[Python Code Style](http://docs.python-guide.org/en/latest/writing/style/)
+which is adhered to with Black formatter.
+
+- Code '''must''' pass [flake8](https://pypi.python.org/pypi/flake8) (w/[https://pypi.python.org/pypi/flake8-quotes flake8-quotes]), [https://pypi.python.org/pypi/isort isort] and [http://www.pylint.org/ Pylint] source code checkers.
+
+  flake8 deluge
+  isort -rc -df deluge
+  pylint deluge
+  pylint deluge/plugins/\*/deluge/
+
+- Using the [http://pre-commit.com/ pre-commit] app can aid in picking up
+  issues before creating git commits.
+
+#### Strings and bytes
+
+To prevent bugs or errors in the code byte strings (`str`) must be decoded to
+strings (unicode strings, `unicode`) on input and then encoded on output.
+
+_Notes:_
+
+- PyGTK/GTK+ will accept `str` (utf8 encoded) or `unicode` but will only return
+  `str`. See [http://python-gtk-3-tutorial.readthedocs.org/en/latest/unicode.html GTK+ Unicode] docs.
+- There is a `bytearray` type which enables in-place modification of a string.
+  See [Python Bytearrays](http://stackoverflow.com/a/9099337/175584)
+- Python 3 renames `unicode` to `str` type and byte strings become `bytes` type.
+
+### Javascript
+
+- Classes should follow the Ext coding style.
+- Class names should be in !CamelCase
+- Instances of classes should use camelCase.
+
+### Path separators
+
+- All relative path separators used within code should be converted to posix
+  format `/`, so should not contain `\` or `\\`. This is to prevent confusion
+  when dealing with cross-platform clients and servers.
+
+### Docstrings
+
+All new docstrings must use Napoleon
+[Google Style](http://www.sphinx-doc.org/en/stable/ext/napoleon.html)
+with old docstrings eventually converted over.
+
+Google Style example:
+
+    def func(arg):
+       """Function purpose.
+
+       Args:
+           arg (type): Description.
+
+       Returns:
+          type: Description. If the line is too, long indent next
+             line with three spaces.
+       """
+       return
+
+See complete list of [http://www.sphinx-doc.org/en/master/usage/extensions/napoleon.html#docstring-sections supported headers].
+
+Verify that the documentation parses correctly with:
+
+    python setup.py build_docs

docs/source/contributing/documentation.md

@@ -0,0 +1,14 @@
+# Documentation contributions
+
+## Build
+
+We use Sphinx to create the documentation from source files and docstrings in code.
+
+    pip install -r requirements-docs.txt
+    python setup.py build_docs
+
+The resulting html files are in `docs/build/html`.
+
+## man pages
+
+Located in `docs/man`

docs/source/contributing/index.md

@@ -0,0 +1,9 @@
+# Development & community
+
+Deluge is an open-source project, and relies on its community of users to keep
+getting better.
+
+- [Code contributions](code.md)
+- [Running tests](testing.md)
+- [Documentation contributions](documentation.md)
+- [Translation contributions](translations.md)

docs/source/contributing/testing.md

@@ -0,0 +1,64 @@
+# Running tests
+
+Deluge testing is implemented using Trial which is Twisted's testing framework
+and an extension of Python's unittest.
+
+See Twisted website for documentation on [Twisted Trial](http://twistedmatrix.com/trac/wiki/TwistedTrial)
+and [Writing tests using Trial](http://twistedmatrix.com/documents/current/core/howto/testing.html).
+
+## Testing
+
+The tests are located in the source folder under `deluge/tests`.
+The tests are run from the project root directory.
+View the unit test coverage at: [deluge-torrent.github.io](http://deluge-torrent.github.io)
+
+### Trial
+
+Here are some examples that show running all the test through to selecting an
+individual test.
+
+    trial deluge
+    trial deluge.tests.test_client
+    trial deluge.tests.test_client.ClientTestCase
+    trial deluge.tests.test_client.ClientTestCase.test_connect_localclient
+
+### Pytest
+
+    pytest deluge/tests
+    pytest deluge/tests/test_client.py
+    pytest deluge/tests/test_client.py -k test_connect_localclient
+
+### Plugin
+
+Running the tests for a specific plugin (requires [pytest](https://pypi.python.org/pypi/pytest)):
+
+    pytest deluge/plugins/<name-of-plugin>
+
+## Tox
+
+All the tests for Deluge can be run using [tox](https://pypi.python.org/pypi/tox)
+
+#### See available targets:
+
+    tox -l
+    py27
+    py3
+    lint
+    docs
+
+#### Run specific test:
+
+    tox -e py3
+
+#### Verify code with pre-commit:
+
+    tox -e lint
+
+## Travis-ci
+
+Deluge develop branch is tested automatically by [Travis].
+When creating a pull request (PR) on [github], Travis will be automatically run
+the unit tests with the code in the PR.
+
+[travis]: https://travis-ci.org/deluge-torrent/deluge
+[github]: https://github.com/deluge-torrent/deluge/pulls

docs/source/contributing/translations.md

@@ -0,0 +1,65 @@
+# Translation contributions
+
+## Translators
+
+For translators we have a [Launchpad translations] account where you can
+translate the `.po` files.
+
+## Marking text for translation
+
+To mark text for translation in Python and ExtJS wrap the string with the
+function `_()` like this:
+
+    torrent.set_tracker_status(_("Announce OK"))
+
+For GTK the text can also be marked translatable in the `glade/*.ui` files:
+
+    <property name="label" translatable="yes">Max Upload Speed:</property>
+
+For more details see: [http://docs.python.org/library/gettext.html Python Gettext]
+
+## Translation process
+
+These are the overall stages in gettext translation:
+
+`Portable Object Template -> Portable Object -> Machine Object`
+
+- The `deluge.pot` is created using `generate_pot.py`.
+- Upload `deluge/i18n/deluge.pot` to [Launchpad translations].
+- Give the translators time to translate the text.
+- Download the updated `.po` files from translation site.
+- Extract to `deluge/i18n/` and strip the `deluge-` prefix:
+
+       rename -f 's/^deluge-//' deluge-*.po
+
+- The binary `MO` files for each language are generated by `setup.py`
+  using the `msgfmt.py` script.
+
+To enable WebUI to use translations update `gettext.js` by running `gen_gettext.py` script.
+
+## Useful applications
+
+- [podiff](http://puszcza.gnu.org.ua/projects/podiff/) - Compare textual information in two PO files
+- [gtranslator](http://projects.gnome.org/gtranslator/) - GUI po file editor
+- [Poedit](http://www.poedit.net/) - GUI po file editor
+
+## Testing translation
+
+Testing that translations are working correctly can be performed by running
+Deluge as follows.
+
+Create an `MO` for a single language in the correct sub-directory:
+
+    mkdir -p deluge/i18n/fr/LC_MESSAGES
+    python msgfmt.py -o deluge/i18n/fr/LC_MESSAGES/deluge.mo deluge/i18n/fr.po
+
+Run Deluge using an alternative language:
+
+    LANGUAGE=fr deluge
+    LANGUAGE=ru_RU.UTF-8 deluge
+
+Note: If you do not have a particular language installed on your system it
+will only translate based on the `MO` files for Deluge so some GTK
+text/button strings will remain in English.
+
+[launchpad translations]: https://translations.launchpad.net/deluge/