Merge branch 'develop' of http://github.com/mytardis/mytardis into de…

…velop

* 'develop' of http://github.com/mytardis/mytardis: (35 commits)
  encode utf-8 for haystack indexing
  3.7 release
  docs requirements.txt update
  docs link fix
  misplaced text
  build.sh update
  docs restructure and update
  After the Facility Overview's latest datasets are initially sorted by dataset id descending, this ordering was being overwritten by sorting in the groupBy methods.  This commit addresses this by cloning the data before sorting in the groupBy methods.
  Add SFTP_USERNAME_ATTRIBUTE to default_settings so that it's documented.
  Use SFTP_USERNAME_ATTRIBUTE setting for login name in Cyberduck screenshot.
  Adding libfreetype6-dev to Ubuntu dependencies so Pillow builds with support, fixes /sftp_access Cyberduck image.
  preview_image_par should be initialized.
  If RENDER_IMAGE_SIZE_LIMIT is 0 or missing from settings, it should not be applied.
  Better use of the RENDER_IMAGE_SIZE_LIMIT setting.  If we have generated a (small) preview image using a filter and saved it in a DatafileParameter, then there is no need to exclude that DataFile from the preview images, even if the original DataFile's file size is large.  Instead of excluding it, we just need to ensure that DataFile's get_image_data method doesn't return a file object for a large file.  It should either return a file object for a small image file, or None.
  Bug fix - correctly process some RAPID_CONNECT_* settings, raise an exception if misconfigured.
  default limit on carousel image number
  Max number of images in carousel is now configurable.
  get_images.count can be slow for large datasets.
  Casting size to BIGINT is no longer necessary.
  Addressing the part of Issue #595 which relates to counting and loading of preview images.  (Not addressing the is_online part of that issue.)  For a dataset with 125,550 datafiles, running exists() on the QuerySet takes a fraction of a second, whereas testing if get_images.count() > 0 takes several minutes.
  ...

README.rst

@@ -1,19 +1,28 @@
-README for MyTARDIS
-===================
+MyTardis
+========
 
 .. image:: https://readthedocs.org/projects/mytardis/badge/?version=develop
    :target: http://mytardis.readthedocs.org/en/develop/?badge=develop
    :alt: Documentation Status
 
-.. image:: https://badges.gitter.im/Join%20Chat.svg
-   :alt: Join the chat at https://gitter.im/mytardis/mytardis
-   :target: https://gitter.im/mytardis/mytardis?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge
 
-MyTARDIS is a multi-institutional collaborative venture that facilitates the
+.. image:: https://semaphoreapp.com/api/v1/projects/5d21cc89-645b-4793-bd78-cf525a0dcce2/345561/shields_badge.svg
+   :target: https://semaphoreapp.com/mytardis/mytardis
+   :alt: Semaphore build status
+
+.. image:: https://www.codacy.com/project/badge/c5899f09f2c545edaaf6d474e9e5e11e
+   :target: https://www.codacy.com/public/grischa/mytardis
+   :alt: Codacy Badge
+
+.. image:: https://coveralls.io/repos/mytardis/mytardis/badge.svg?branch=develop
+  :target: https://coveralls.io/r/mytardis/mytardis?branch=develop
+  :alt: Coveralls Badge
+
+MyTardis is a multi-institutional collaborative venture that facilitates the
 archiving and sharing of data and metadata collected at major facilities such
 as the Australian Synchrotron and ANSTO and within Institutions.
 
-An example of the benefit of a system such as MyTARDIS in the protein
+An example of the benefit of a system such as MyTardis in the protein
 crystallography community is that while the model coordinates and (less often)
 the structure factors (processed experimental data) are stored in the
 community Protein Data Bank (PDB) the raw diffraction data is often not
@@ -28,6 +37,19 @@ summarised as:
    archives are typically volatile).
 
 
+Find out more
+-------------
+
+Project homepage http://mytardis.org
+
+The source code is hosted at https://github.com/mytardis/mytardis
+
+Documentation at http://mytardis.readthedocs.org includes
+
+- User documentation
+- Administrator documentation
+- Developer documentation
+
 Releases
 --------
 
@@ -40,54 +62,35 @@ included. It will move to newer versions automatically. Follow this branch
 if you want to stay up to date in a production environment.
 
 Each version has its own branch named by version number. At the time of
-writing this is ``3.5``. Follow this branch for your production installation and
+writing this is ``3.7``. Follow this branch for your production installation and
 perform version upgradres manually.
 
 Each bugfix or set of fixes bumps the minor version and each new release is
-tagged, eg. ``3.5.2``. Use tagged releases if you are paranoid about changes to
+tagged, eg. ``3.7.2``. Use tagged releases if you are paranoid about changes to
 the code you have not tested yourself.
 
 To follow development, please see the contributing section below.
 
+
 Reporting Bugs
 --------------
 
 Bug reports and feature requests can be made via our `public issue tracker`_.
 
 .. _`public issue tracker`: https://github.com/mytardis/mytardis/issues
 
+
 Contributing
 ------------
 
-The ``develop`` branch is the cutting edge code base that all development is
-based upon.
-
-Send wishes, comments, etc. to tardis-devel@googlegroups.com.
-
-New contributors are welcome, however all developers should review the
+New contributors are always welcome, however all developers should review the
 `pull-request checklist`_ before making pull requests.
 
-.. _`pull-request checklist`: https://github.com/mytardis/mytardis/blob/master/CONTRIBUTING.rst
-
-
-Code Health Status
-------------------
-
-Automatic build, once a day:
-
-.. image:: https://semaphoreapp.com/api/v1/projects/5d21cc89-645b-4793-bd78-cf525a0dcce2/345561/shields_badge.svg
-   :target: https://semaphoreapp.com/mytardis/mytardis
-   :alt: Semaphore build status
+For any wishes, comments, praise etc. either open a GitHub issue or contact us.
 
+Active developers are also welcome to join our Slack team.
 
-Code analysis (just testing this out at the moment):
+Contact details can be found on `mytardis.org`_.
 
-.. image:: https://www.codacy.com/project/badge/c5899f09f2c545edaaf6d474e9e5e11e
-   :target: https://www.codacy.com/public/grischa/mytardis
-   :alt: Codacy Badge
-
-Code coverage
-
-.. image:: https://coveralls.io/repos/mytardis/mytardis/badge.svg?branch=develop
-  :target: https://coveralls.io/r/mytardis/mytardis?branch=develop
-  :alt: Coveralls Badge
+.. _`mytardis.org`: http://mytardis.org
+.. _`pull-request checklist`: https://github.com/mytardis/mytardis/blob/master/CONTRIBUTING.rst

build.sh

@@ -4,12 +4,7 @@ echo This is a guide only, please either edit or run appropriate commands manual
 exit
 
 # for Ubuntu 14.04
-# sudo apt-get update
-# sudo apt-get install python-pip git libxml2-dev libxslt1-dev python-dev zlib1g-dev python-wand
-# sudo apt-get install python-virtualenv virtualenvwrapper python-psycopg2 python-yaml ipython
-# sudo apt-get install python-anyjson python-bs4 python-billiard python-feedparser python-html5lib
-# sudo apt-get install python-httplib2 python-pystache python-crypto python-flexmock python-dateutil
-# sudo apt-get install libldap2-dev libsasl2-dev libssl-dev
+# sudo bash install-ubuntu-requirements.sh
 # # optionally:
 # # sudo apt-get install memcached python-memcache
 

docs/admin.rst

@@ -1,80 +1,14 @@
-====================
-Database Maintenance
-====================
-
-Initialising
-------------
-
-When creating a new database the ``syncdb`` command will need to be
-called to initialise the schema and insert the initial data fixtures.
-
-Usage
-~~~~~
-``python mytardis.py syncdb --noinput --all``
-
-Migrating
----------
-
-Some of the upgrades to MyTARDIS will require that the database schema
-be upgraded to match the internal data model. This tool migrates data
-from old database schemas to the current one. It detects which version
-of the database you are currently running and will automatically
-migrate to the current version. If you decided not to use South for
-migrating the data models and have removed it from the list of
-INSTALLED_APPS in the settings.py file, you could skip this step and
-go on initiating an administrator account with Django createsuperuser
-which is described in more detail below.
-
-In certain cases it is also necessary to update the permissions table.
-
-Usage
-~~~~~
-
-Intial migration after ``syncdb --all``::
-
-  python mytardis.py migrate --fake
-
-Then::
-
-  python mytardis.py migrate
-
-If the model changes require it, run::
-
-  python mytardis.py update_permissions
-
-
-creating superuser
-------------------
-
-After success of database initialization or migration, please use a
-command line utility called ``createsuperuser`` to create an
-administrator account using the admin site which is hooked to the URL
-/admin/.
-
-Usage
-~~~~~
-
-``python mytardis.py createsuperuser``
-
-Backup
-------
-
-The backupdb command allows to backup and to restore of the MyTARDIS
-database.  The command uses the corresponding database tool to
-facilitate this task. Currently implemented are PostgreSQL and
-MySQL. In case of backup, a directory called ``backups`` is created
-(if it does not exists) in the working directory.  In case of restore,
-the database for storing the tablespace must already exist before
-loading the backup file into the database.
-
-Usage
-~~~~~
-``python mytardis.py backupdb``
-
-.. option:: -r FILE, --restore=FILE
-.. option:: -v VERBOSITY, --verbosity=VERBOSITY
-.. option:: --settings=SETTINGS
-.. option:: --pythonpath=PYTHONPATH
-.. option:: --traceback
-.. option:: --version
-.. option:: -h, --help
+================================
+Configuration and Administration
+================================
+
+.. toctree::
+  :maxdepth: 2
+
+  admin/install
+  admin/authentication
+  admin/filters
+  admin/database
+  admin/metadata
+  admin/searchsetup
+  admin/storage

docs/ref/auth_httpbasicendpoint.rst → docs/admin/auth_httpbasicendpoint.rst

@@ -1,11 +1,11 @@
 .. _ref-httpbasicendpoint_auth:
 
-:py:mod:`tardis.tardis_portal.auth.httpbasicendpoint_auth` -- HTTP Basic Endpoint Authentication
-================================================================================================
+HTTP Basic Endpoint Authentication
+==================================
 
 
-.. py:module:: tardis.tardis_portal.auth.httpbasicendpoint_auth
-.. moduleauthor:: Tim Dettrick <t.dettrick@uq.edu.au>
+:py:mod:`tardis.tardis_portal.auth.httpbasicendpoint_auth`
+
 
 HTTP Basic Endpoint authentication uses access to a HTTP resource (or endpoint)
 to determine if authentication should be allowed.
@@ -26,10 +26,9 @@ name. eg.
 On each request, MyTardis will attempt to use basic authentication with the
 provided credentials to access the HTTP resource. If it fails, access is denied.
 
-.. attribute:: tardis.default_settings.LDAP_TLS
 
-    Endpoint to use in HTTP Basic Endpoint Auth. eg.
+Endpoint to use in HTTP Basic Endpoint Auth. eg.
 
-    .. code-block:: python
+.. code-block:: python
 
-        HTTP_BASIC_AUTH_ENDPOINT = 'https://test.example/endpoint'
+   HTTP_BASIC_AUTH_ENDPOINT = 'https://test.example/endpoint'

docs/ref/auth_ldap.rst → docs/admin/auth_ldap.rst

@@ -1,12 +1,10 @@
 .. _ref-ldap_auth:
 
-:py:mod:`tardis.tardis_portal.auth.ldap_auth` -- LDAP Authentication
-====================================================================
+LDAP Authentication
+===================
 
+:py:mod:`tardis.tardis_portal.auth.ldap_auth`
 
-.. py:module:: tardis.tardis_portal.auth.ldap_auth
-.. moduleauthor:: Gerson Galang <gerson.galang@versi.edu.au>
-.. moduleauthor:: Russell Sim <russell.sim@gmail.com>
 
 To enable LDAP you'll need to specify which components of the LDAP
 authentication backend are enabled. In the *settings.py* add the
@@ -70,9 +68,3 @@ initialising the LDAP backend.
    " + LDAP_BASE*
 
 
-:class:`LDAPBackend` Objects
-----------------------------
-.. autoclass:: LDAPBackend
-   :members:
-   :undoc-members:
-

docs/ref/auth_tokenauth.rst → docs/admin/auth_tokenauth.rst

@@ -1,11 +1,9 @@
 .. _ref-tokenauth:
 
-:py:mod:`tardis.tardis_portal.auth.token_auth` -- Temporary Token Authentication
-====================================================================
+Temporary Token Authentication
+==============================
 
-
-.. py:module:: tardis.tardis_portal.auth.token_auth
-.. moduleauthor:: Ryan Braganza <ryan@intersect.org.au>
+:py:mod:`tardis.tardis_portal.auth.token_auth`
 
 To use token authentication, you'll need to specify the following
 *settings.py*
@@ -24,7 +22,7 @@ bin/django createtokenuser
 
 Cleaning up
 ------------------------------------------
-bin/django cleanuptokens
+``bin/django cleanuptokens``
 
 It is recommended that you schedule regular purging of expired tokens.
 Set a cronjob to run bin/django cleanuptokens
@@ -34,7 +32,7 @@ Expiry
 --------------------------------
 Token auth works by hijacking the group provider system.
 
-MyTARDIS groups are calculated and cached when a user logs in.
+MyTardis groups are calculated and cached when a user logs in.
 
 This means that if a session is active, and a token becomes in valid (either through deletion or expiry) that access will still be granted.
 To mitigate this, when a token user logs in, an explicit expiry is set on their session - the earlier of 4am the next day, or the session expiry date (the end of the day)

docs/admin/authentication.rst

@@ -0,0 +1,46 @@
+
+.. _dev-authentication:
+
+Authentication Methods
+======================
+
+
+Users
+-----
+
+MyTardis supports several sources of authentication and identity, referred to
+as user providers.
+
+In the *settings.py* user providers are activated by specifying them
+within the **USER_PROVIDERS** variable::
+
+   USER_PROVIDERS = ('tardis.tardis_portal.auth.localdb_auth.DjangoUserProvider',)
+
+Groups
+------
+
+MyTardis also supports several sources for group membership information,
+referred to as group providers.
+
+In the *settings.py* group providers are activated by specifying them
+within the **GROUP_PROVIDERS** variable::
+
+   GROUP_PROVIDERS = ('tardis.tardis_portal.auth.localdb_auth.DjangoGroupProvider',
+                      'tardis.tardis_portal.auth.vbl_auth.VblGroupProvider',)
+
+
+Included Auth Plugins
+---------------------
+
+.. toctree::
+   :maxdepth: 1
+
+   auth_httpbasicendpoint
+   auth_ldap
+   auth_tokenauth
+
+* :py:mod:`tardis.tardis_portal.auth.ip_auth`
+
+The :doc:`../pydoc/tardis.tardis_portal.auth` module contains the authentication
+specific code.
+