1.0 release

* Add functions for forward-compatibility * Enable new release process including automatic deployment of updates * Clean up documentation
dials · Feb 16, 2019 · 077af66 · 077af66
2 parents 7f67937 + 63ca7a6
commit 077af66
Show file tree

Hide file tree

Showing 16 changed files with 219 additions and 85 deletions.
diff --git a/.travis.yml b/.travis.yml
@@ -1,5 +1,8 @@
 language: python
 
+branches:
+  except: /^v[0-9]/ # Don't build on release tags
+
 install: pip install -U tox-travis
 script: tox
 

diff --git a/.travis/update-package-version b/.travis/update-package-version
@@ -1,9 +1,35 @@
 #!/bin/bash
 
-VERSION=$(grep version dials_data/__init__.py | cut -d'"' -f 2)
+set -e
+
+RESTRICT_REPOSITORY="dials/data"
+if [[ "${TRAVIS_REPO_SLUG}" != "${RESTRICT_REPOSITORY}" ]]; then
+  echo Skipping step outside of ${RESTRICT_REPOSITORY} repository
+  exit 0
+fi
+
+VERSION=$(grep "^__version__" dials_data/__init__.py | cut -d'"' -f 2)
 BASETAG=v${VERSION}
-DEPTH=$(git rev-list ${BASETAG}..HEAD --count)
+git tag --list | grep "^${BASETAG}$" || {
+  echo Searching for tag on remote
+  if git fetch origin tag ${BASETAG}; then
+    echo Tag found on remote. Deepening checkout...
+    git fetch --shallow-exclude ${BASETAG}
+    git fetch --deepen 1
+  else
+    echo "Tag not found on remote. Let's make a new one!"
+    git tag ${BASETAG}
+    # assume git credentials were previously set in update-hashinfo
+    git push origin ${BASETAG}
+  fi
+}
+
+DEPTH=$(git rev-list ${BASETAG}..HEAD --count --first-parent)
 REAL_VERSION=${VERSION%.0}.${DEPTH}
-sed -i "s/^__version__.*/__version__ = \"${REAL_VERSION}\"/" dials_data/__init__.py
+echo Setting package version from ${VERSION} to ${REAL_VERSION}
+sed -i "s/^__version__ =.*/__version__ = \"${REAL_VERSION}\"/" dials_data/__init__.py
 sed -i "s/^\(\s\+\)version=\".*\",$/\1version=\"${REAL_VERSION}\",/" setup.py
-echo Set package version from ${VERSION} to ${REAL_VERSION}
+
+COMMIT=$(git rev-parse --verify HEAD)
+echo Setting package commit to ${COMMIT}
+sed -i "s/^__commit__ =.*/__commit__ = \"${COMMIT}\"/" dials_data/__init__.py
diff --git a/CONTRIBUTING.rst b/CONTRIBUTING.rst
@@ -23,17 +23,11 @@ If you are reporting a bug, please include:
 * Any details about your local setup that might be helpful in troubleshooting.
 * Detailed steps to reproduce the bug.
 
-Fix Bugs
-~~~~~~~~
+Add Datasets
+~~~~~~~~~~~~
 
-Look through the GitHub issues for bugs. Anything tagged with "bug" and "help
-wanted" is open to whoever wants to implement it.
-
-Implement Features
-~~~~~~~~~~~~~~~~~~
-
-Look through the GitHub issues for features. Anything tagged with "enhancement"
-and "help wanted" is open to whoever wants to implement it.
+DIALS data was planned to support a more or less arbitrary number of datasets.
+You can contribute by adding more.
 
 Write Documentation
 ~~~~~~~~~~~~~~~~~~~
@@ -98,31 +92,27 @@ Pull Request Guidelines
 
 Before you submit a pull request, check that it meets these guidelines:
 
-1. The pull request should include tests.
-2. If the pull request adds functionality, the docs should be updated. Put
+1. The pull request should include tests, unless you are adding or updating
+   a dataset.
+2. If you add or update a dataset then make individual pull requests for each
+   dataset, so that they can be discussed and approved separately.
+3. If the pull request adds functionality, the docs should be updated. Put
    your new functionality into a function with a docstring, and add the
-   feature to the list in README.rst.
-3. The pull request should work for Python 2.7, 3.4, 3.5 and 3.6, and for PyPy. Check
-   https://travis-ci.org/dials/data/pull_requests
-   and make sure that the tests pass for all supported Python versions.
-
-Tips
-----
-
-To run a subset of tests::
-
-$ py.test tests.test_dials_data
+   feature to the list in HISTORY.rst.
+4. The pull request should work for all supported Python versions. Check
+   https://travis-ci.com/dials/data/pull_requests
 
 
 Deploying
 ---------
 
-A reminder for the maintainers on how to deploy.
-Make sure all your changes are committed (including an entry in HISTORY.rst).
-Then run::
+Any commit on the master branch is now automatically deployed to PyPI, so there
+is no need to play around with tags or version numbers on a regular basis.
+
+For slightly larger changes make sure that the entry in HISTORY.rst is updated,
+and then run::
 
-$ bumpversion patch # possible: major / minor / patch
-$ git push
-$ git push --tags
+$ bumpversion minor # possible: major / minor, do not use patch
 
-Travis will then deploy to PyPI if tests pass.
+Travis will then automatically tag the commit once it hits the master branch
+and the tests pass, and then deploy to PyPI as usual.
diff --git a/HISTORY.rst b/HISTORY.rst
@@ -2,6 +2,12 @@
 History
 =======
 
+1.0 (2019-02-16)
+^^^^^^^^^^^^^^^^
+
+* Add functions for forward-compatibility
+* Enable new release process including automatic deployment of updates
+
 0.6 (2019-02-15)
 ^^^^^^^^^^^^^^^^
 

diff --git a/LICENSE b/LICENSE
@@ -1,4 +1,4 @@
-Copyright (c) 2018 Diamond Light Source.
+Copyright (c) 2018-2019 Diamond Light Source.
 All rights reserved.
 
 Redistribution and use in source and binary forms, with or without

diff --git a/dials_data/__init__.py b/dials_data/__init__.py
@@ -1,6 +1,9 @@
 # -*- coding: utf-8 -*-
 
-"""Top-level package for DIALS Regression Data."""
+"""
+DIALS Regression Data Manager
+https://github.com/dials/data
+"""
 
 from __future__ import absolute_import, division, print_function
 
@@ -10,7 +13,9 @@
 __all__ = ["pytest_addoption", "dials_data"]
 __author__ = """Markus Gerstel"""
 __email__ = "dials-support@lists.sourceforge.net"
-__version__ = "0.6.0"
+__version__ = "1.0.0"
+__commit__ = ""
+__version_tuple__ = tuple(int(x) for x in __version__.split("."))
 
 
 def pytest_addoption(parser):
@@ -36,7 +41,11 @@ def dials_data(request):
         pytest.skip("Test requires --regression option to run.")
     df = DataFetcher()
 
-    def skip_test_if_lookup_failed(result):
+    def skip_test_if_lookup_failed(result, dials_data_too_old=False):
+        if dials_data_too_old:
+            pytest.skip(
+                "Test requires a newer version of dials_data (v%s)" % dials_data_too_old
+            )
         if not result:
             pytest.skip(
                 "Automated download of test data failed. Download manually using dials.data"

diff --git a/dials_data/cli.py b/dials_data/cli.py
@@ -69,6 +69,10 @@ def cli_list(cmd_args):
 
 
 def main():
+    if dials_data.__commit__:
+        version = dials_data.__version__ + "-g" + dials_data.__commit__[:7]
+    else:
+        version = dials_data.__version__ + "-dev"
     parser = argparse.ArgumentParser(
         usage="dials.data <command> [<args>]",
         description="""DIALS regression data manager v{version}
@@ -77,7 +81,7 @@ def main():
    list     List available datasets
    get      Download datasets
 """.format(
-            version=dials_data.__version__
+            version=version
         ),
         formatter_class=argparse.RawTextHelpFormatter,
     )

diff --git a/dials_data/download.py b/dials_data/download.py
@@ -101,19 +101,31 @@ def fetch_dataset(
     verbose=False,
     pre_scan=True,
 ):
-    """Return a the location of a local copy of the test dataset repository.
-       If this repository is not available or out of date then attempt to download/update it transparently.
+    """Check for the presence or integrity of the local copy of the specified
+       test dataset. If the dataset is not available or out of date then attempt
+       to download/update it transparently.
 
        :param verbose:          Show everything as it happens.
        :param pre_scan:         If all files are present and all file sizes match
                                 then skip file integrity check and exit quicker.
        :param read_only:        Only use existing data, never download anything.
                                 Implies pre_scan=True.
+       :returns:                False if the dataset can not be downloaded/updated
+                                for any reason.
+                                True if the dataset is present and passes a
+                                cursory inspection.
+                                A validation dictionary if the dataset is present
+                                and was fully verified.
     """
+    if dataset not in dials_data.datasets.definition:
+        return False
+    definition = dials_data.datasets.definition[dataset]
+
     target_dir = dials_data.datasets.repository_location().join(dataset)
+    if read_only and not target_dir.check(dir=1):
+        return False
     target_dir.ensure(dir=1)
 
-    definition = dials_data.datasets.definition[dataset]
     integrity_info = definition.get("hashinfo")
     if not integrity_info or ignore_hashinfo:
         integrity_info = dials_data.datasets.create_integrity_record(dataset)
@@ -207,25 +219,51 @@ def __repr__(self):
             self._target_dir.strpath,
         )
 
-    def result_filter(self, result):
+    def result_filter(self, result, **kwargs):
         """
         An overridable function to mangle lookup results.
         Used in tests to transform negative lookups to test skips.
+        Overriding functions should add **kwargs to function signature
+        to be forwards compatible.
         """
         return result
 
-    def __call__(self, test_data):
+    def __call__(self, test_data, **kwargs):
+        """
+        Return the location of a dataset, transparently downloading it if
+        necessary and possible.
+        The return value can be manipulated by overriding the result_filter
+        function.
+        :param test_data: name of the requested dataset.
+        :param min_version: minimum required version of dials_data.
+        :return: A py.path.local object pointing to the dataset, or False
+                 if the dataset is not available.
+        """
         if test_data not in self._cache:
-            if self._read_only:
-                data_available = fetch_dataset(test_data, pre_scan=True, read_only=True)
-            else:
-                with download_lock(self._target_dir):
-                    # Need to acquire lock as files may be downloaded/written.
-                    data_available = fetch_dataset(
-                        test_data, pre_scan=True, read_only=False
-                    )
-            if data_available:
-                self._cache[test_data] = self._target_dir.join(test_data)
+            self._cache[test_data] = self._attempt_fetch(test_data, **kwargs)
+        return self.result_filter(**self._cache[test_data])
+
+    def _attempt_fetch(self, test_data, min_version=None):
+        if min_version:
+            if isinstance(min_version, tuple):
+                min_version_tuple = min_version
             else:
-                self._cache[test_data] = False
-        return self.result_filter(self._cache[test_data])
+                min_version_tuple = tuple(int(x) for x in min_version.split("."))
+            if dials_data.__version_tuple__ < min_version_tuple:
+                return {
+                    "result": False,
+                    "dials_data_too_old": ".".join(map(str, min_version_tuple)),
+                }
+
+        if self._read_only:
+            data_available = fetch_dataset(test_data, pre_scan=True, read_only=True)
+        else:
+            with download_lock(self._target_dir):
+                # Need to acquire lock as files may be downloaded/written.
+                data_available = fetch_dataset(
+                    test_data, pre_scan=True, read_only=False
+                )
+        if data_available:
+            return {"result": self._target_dir.join(test_data)}
+        else:
+            return {"result": False}
diff --git a/docs/_templates/breadcrumbs.html b/docs/_templates/breadcrumbs.html
@@ -0,0 +1,43 @@
+{# Support for Sphinx 1.3+ page_source_suffix, but don't break old builds. #}
+
+{% if page_source_suffix %}
+{% set suffix = page_source_suffix %}
+{% else %}
+{% set suffix = source_suffix %}
+{% endif %}
+
+{% if meta is defined and meta is not none %}
+{% set check_meta = True %}
+{% else %}
+{% set check_meta = False %}
+{% endif %}
+
+<div role="navigation" aria-label="breadcrumbs navigation">
+
+  <ul class="wy-breadcrumbs">
+    {% block breadcrumbs %}
+      <li><a href="{{ pathto(master_doc) }}">{{ _('Docs') }}</a> &raquo;</li>
+        {% for doc in parents %}
+          <li><a href="{{ doc.link|e }}">{{ doc.title }}</a> &raquo;</li>
+        {% endfor %}
+      <li>{{ title }}</li>
+    {% endblock %}
+    {% block breadcrumbs_aside %}
+      <li class="wy-breadcrumbs-aside">
+        <a href="https://github.com/dials/data" class="fa fa-github"> View project on GitHub</a>
+      </li>
+    {% endblock %}
+  </ul>
+
+  {% if (theme_prev_next_buttons_location == 'top' or theme_prev_next_buttons_location == 'both') and (next or prev) %}
+  <div class="rst-breadcrumbs-buttons" role="navigation" aria-label="breadcrumb navigation">
+      {% if next %}
+        <a href="{{ next.link|e }}" class="btn btn-neutral float-right" title="{{ next.title|striptags|e }}" accesskey="n">Next <span class="fa fa-arrow-circle-right"></span></a>
+      {% endif %}
+      {% if prev %}
+        <a href="{{ prev.link|e }}" class="btn btn-neutral float-left" title="{{ prev.title|striptags|e }}" accesskey="p"><span class="fa fa-arrow-circle-left"></span> Previous</a>
+      {% endif %}
+  </div>
+  {% endif %}
+  <hr/>
+</div>
diff --git a/docs/conf.py b/docs/conf.py
@@ -47,15 +47,15 @@
 
 # General information about the project.
 project = u"DIALS data"
-copyright = u"2018, Markus Gerstel"
+copyright = u"2018-2019, Diamond Light Source"
 author = u"Markus Gerstel"
 
 # The version info for the project you're documenting, acts as replacement
 # for |version| and |release|, also used in various other places throughout
 # the built documents.
 #
 # The short X.Y version.
-version = "0.6.0"
+version = "1.0.0"
 # The full version, including alpha/beta/rc tags.
 release = version
 
@@ -83,7 +83,7 @@
 # The theme to use for HTML and HTML Help pages.  See the documentation for
 # a list of builtin themes.
 #
-html_theme = "alabaster"
+html_theme = "sphinx_rtd_theme"
 
 # Theme options are theme-specific and customize the look and feel of a
 # theme further.  For a list of options available for each theme, see the

diff --git a/docs/index.rst b/docs/index.rst
@@ -1,5 +1,14 @@
-Welcome to the DIALS data documentation!
-========================================
+DIALS data documentation
+========================
+
+.. raw:: html
+
+   <!-- http://tholman.com/github-corners/ --><a href="https://github.com/dials/data" class="github-corner" aria-label="View source on GitHub"><svg width="80" height="80" viewBox="0 0 250 250" style="fill:#70B7FD; color:#fff; position: absolute; top: 0; border: 0; right: 0;" aria-hidden="true"><path d="M0,0 L115,115 L130,115 L142,142 L250,250 L250,0 Z"></path><path d="M128.3,109.0 C113.8,99.7 119.0,89.6 119.0,89.6 C122.0,82.7 120.5,78.6 120.5,78.6 C119.2,72.0 123.4,76.3 123.4,76.3 C127.3,80.9 125.5,87.3 125.5,87.3 C122.9,97.6 130.6,101.9 134.4,103.2" fill="currentColor" style="transform-origin: 130px 106px;" class="octo-arm"></path><path d="M115.0,115.0 C114.9,115.1 118.7,116.5 119.8,115.4 L133.7,101.6 C136.9,99.2 139.9,98.4 142.2,98.6 C133.8,88.0 127.5,74.4 143.8,58.0 C148.5,53.4 154.0,51.2 159.7,51.0 C160.3,49.4 163.2,43.6 171.4,40.1 C171.4,40.1 176.1,42.5 178.8,56.2 C183.1,58.6 187.2,61.8 190.9,65.4 C194.5,69.0 197.7,73.2 200.1,77.6 C213.8,80.2 216.3,84.9 216.3,84.9 C212.7,93.1 206.9,96.0 205.4,96.6 C205.1,102.4 203.0,107.8 198.3,112.5 C181.9,128.9 168.3,122.5 157.7,114.1 C157.9,116.9 156.7,120.9 152.7,124.9 L141.0,136.5 C139.8,137.7 141.6,141.9 141.8,141.8 Z" fill="currentColor" class="octo-body"></path></svg></a><style>.github-corner:hover .octo-arm{animation:octocat-wave 560ms ease-in-out}@keyframes octocat-wave{0%,100%{transform:rotate(0)}20%,60%{transform:rotate(-25deg)}40%,80%{transform:rotate(10deg)}}@media (max-width:500px){.github-corner:hover .octo-arm{animation:none}.github-corner .octo-arm{animation:octocat-wave 560ms ease-in-out}}</style>
+
+.. image:: https://img.shields.io/github/issues-raw/dials/data.svg?style=flat
+.. image:: https://img.shields.io/github/issues-pr-raw/dials/data.svg?style=flat
+.. image:: https://img.shields.io/github/license/dials/data.svg?style=flat
+
 
 .. toctree::
    :maxdepth: 2
@@ -10,9 +19,3 @@ Welcome to the DIALS data documentation!
    installation
    contributing
    history
-
-Indices and tables
-==================
-* :ref:`genindex`
-* :ref:`modindex`
-* :ref:`search`
diff --git a/requirements.txt b/requirements.txt
@@ -1,3 +1,4 @@
+py
 pytest
 pyyaml
 setuptools