First commit

.github/workflows/lint.yml

@@ -0,0 +1,20 @@
+name: Lint
+on: [push, pull_request]
+env:
+  BASEDIR: https://raw.githubusercontent.com/open-contracting/standard-maintenance-scripts/master
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@v2
+    - uses: actions/setup-python@v1
+      with:
+        python-version: 3.6
+    - uses: actions/cache@v1
+      with:
+        path: ~/.cache/pip
+        key: ${{ runner.os }}-pip-${{ hashFiles('.github/workflows/lint.yml') }}
+        restore-keys: |
+          ${{ runner.os }}-pip-
+    - run: curl -s -S --retry 3 $BASEDIR/tests/install.sh | bash -
+    - run: curl -s -S --retry 3 $BASEDIR/tests/script.sh | bash -

.gitignore

@@ -0,0 +1 @@
+/docs/_build

LICENSE.md

@@ -0,0 +1,5 @@
+# License
+
+Copyright 2020 Open Contracting Partnership
+
+This work is licensed under a [Creative Commons Attribution 4.0 International License](https://creativecommons.org/licenses/by/4.0/).

README.md

@@ -0,0 +1,29 @@
+# OCP Software Development Handbook
+
+A guide for developers of OCP's tools.
+
+This repository was originally based on OpenDataServices' [sphinx-base](https://github.com/OpenDataServices/sphinx-base). See its [instructions for ReadTheDocs setup](https://github.com/OpenDataServices/sphinx-base#building-on-readthedocs).
+
+## Build and view the documentation
+
+Create and activate a virtual environment, then install requirements:
+
+```shell
+pip install -r requirements.txt
+```
+
+And build the documentation:
+
+```
+cd docs
+make html
+```
+
+The built documentation is in `_build/html` under `docs`. To view the documentation:
+
+```shell
+cd _build/html
+python -m http.server
+```
+
+And open <http://localhost:8000/> in a browser.

docs/Makefile

@@ -0,0 +1,20 @@
+# Minimal makefile for Sphinx documentation
+#
+
+# You can set these variables from the command line, and also
+# from the environment for the first two.
+SPHINXOPTS    ?=
+SPHINXBUILD   ?= sphinx-build
+SOURCEDIR     = .
+BUILDDIR      = _build
+
+# Put it first so that "make" without argument is like "make help".
+help:
+	@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
+
+.PHONY: help Makefile
+
+# Catch-all target: route all unknown targets to Sphinx using the new
+# "make mode" option.  $(O) is meant as a shortcut for $(SPHINXOPTS).
+%: Makefile
+	@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)

docs/conf.py

@@ -0,0 +1,78 @@
+# Configuration file for the Sphinx documentation builder.
+#
+# This file only contains a selection of the most common options. For a full
+# list see the documentation:
+# https://www.sphinx-doc.org/en/master/usage/configuration.html
+import os
+
+from recommonmark.transform import AutoStructify
+
+# -- Path setup --------------------------------------------------------------
+
+# If extensions (or modules to document with autodoc) are in another directory,
+# add these directories to sys.path here. If the directory is relative to the
+# documentation root, use os.path.abspath to make it absolute, like shown here.
+#
+# import os
+# import sys
+# sys.path.insert(0, os.path.abspath('.'))
+
+
+# -- Project information -----------------------------------------------------
+
+project = 'OCDS Development Handbook'
+copyright = '2017, Open Contracting Partnership'
+author = 'Open Contracting Partnership'
+
+
+# -- General configuration ---------------------------------------------------
+
+# Add any Sphinx extension module names here, as strings. They can be
+# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
+# ones.
+extensions = [
+    'recommonmark',
+]
+
+# Add any paths that contain templates here, relative to this directory.
+templates_path = ['_templates']
+
+# List of patterns, relative to source directory, that match files and
+# directories to ignore when looking for source files.
+# This pattern also affects html_static_path and html_extra_path.
+exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store']
+
+
+# -- Options for HTML output -------------------------------------------------
+
+# The theme to use for HTML and HTML Help pages.  See the documentation for
+# a list of builtin themes.
+#
+html_theme = 'default'
+
+# Add any paths that contain custom static files (such as style sheets) here,
+# relative to this directory. They are copied after the builtin static files,
+# so a file named "default.css" will overwrite the builtin "default.css".
+html_static_path = ['_static']
+
+
+# -- Extension configuration -------------------------------------------------
+
+# Needed for ReadTheDocs (Sphinx 1.8).
+master_doc = 'index'
+
+on_rtd = os.environ.get('READTHEDOCS', None) == 'True'
+
+if not on_rtd:  # only import and set the theme if we're building docs locally
+    import sphinx_rtd_theme
+    html_theme = 'sphinx_rtd_theme'
+    html_theme_path = [sphinx_rtd_theme.get_html_theme_path()]
+
+
+def setup(app):
+    app.add_config_value('recommonmark_config', {
+        'auto_toc_tree_section': 'Contents',
+        'enable_eval_rst': True,
+    }, True)
+
+    app.add_transform(AutoStructify)

docs/github/index.rst

@@ -0,0 +1,117 @@
+GitHub
+======
+
+Organizations
+-------------
+
+-  `open-contracting <https://github.com/open-contracting/>`__: all repositories supported by OCP
+-  `open-contracting-extensions <https://github.com/open-contracting-extensions/>`__: OCDS extensions and OCDS profiles
+-  `open-contracting-archive <https://github.com/open-contracting-archive/>`__: all unsupported repositories
+-  `open-contracting-partnership <https://github.com/open-contracting-partnership/>`__: `www.open-contracting.org <https://www.open-contracting.org/>`__-related repositories
+
+Teams
+-----
+
+In order to protect the private deploy repositories, the `base permissions <https://github.com/organizations/open-contracting/settings/member_privileges>`__ for ``open-contracting`` members is *None*. There are two teams:
+
+-  `General <https://github.com/orgs/open-contracting/teams/general>`__:
+
+   -  **Members**: All `people <https://github.com/orgs/open-contracting/people>`__, except Dogsbody Technology staff
+   -  **Repositories**: All repositories, except `deploy <https://github.com/open-contracting?q=deploy>`__ and `archived <https://github.com/open-contracting?type=archived>`__ repositories
+   -  **Permission levels**: *Triage* for issue-only repositories, *Write* for other repositories
+
+-  `Servers <https://github.com/orgs/open-contracting/teams/servers>`__:
+
+   -  **Members**: All people with `root access <https://ocdsdeploy.readthedocs.io/en/latest/reference/index.html#root-access>`__ to any server
+   -  **Repositories**: The `deploy <https://github.com/open-contracting?q=deploy>`__ repositories
+   -  **Permission levels**: *Write*
+
+Issues
+------
+
+Managing long issues
+~~~~~~~~~~~~~~~~~~~~
+
+Some issues produce long discussions, and the original intent of the issue may change over time; this can make it more difficult to catch up on the current state of the issue. To deal with this, the issue’s description should be edited to reflect the current state of the issue, summarize the discussion so far, and link to the most recent comment from which the discussion may continue.
+
+Restarting the discussion with a new issue causes the following problems:
+
+-  Old participants aren’t notified of activity on the new issue, and need to subscribe to it.
+-  New participants need to read all issues referencing the new issue to rebuild the context.
+-  If the old issue is closed in favor of a new issue, and the new issue is thereafter not resolved but is closed (for whatever reason, like insufficient demand), a reader of the old issue may assume that the new issue is either open or resolved. They would need to follow the chain to realize that it’s unresolved and closed, before adding a comment to say, “I need this.”
+
+Automatically closing issues with pull requests
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+When creating a pull request that fixes one or more issues, add the text “fixes #42” or “closes #42” in the pull request’s description so that GitHub `automatically closes them after merging <https://help.github.com/articles/closing-issues-using-keywords/>`__.
+
+Manually closing issues
+~~~~~~~~~~~~~~~~~~~~~~~
+
+All issues should be closed with a brief rationale. This makes it easy to understand what happened and affords participants an opportunity to engage with the rationale. For example:
+
+-  “Resolved in the above commit” if there’s a commit referencing the issue that appears nearby
+-  “Resolved in the [name] extension” with a link to the extension that was created
+-  “Closing, because [explanation]”
+
+Projects
+--------
+
+The ```open-contracting`` organization <https://github.com/orgs/open-contracting/projects>`__ uses `GitHub Projects <https://help.github.com/articles/about-project-boards/>`__ to organize work in a `Kanban <https://en.wikipedia.org/wiki/Kanban>`__ system.
+
+We also have an `agile board <https://crm.open-contracting.org/projects/ocds-team-tools-development-portfolio/agile/board>`__ in the CRM. Its uses and features are distinct (e.g. time tracking) and complementary (e.g. long-term project management) to what GitHub offers.
+
+People expect to have visibility of all of a repository’s issues within the *Issues* tab; therefore, a card that is not attached to an issue should never be added to a project.
+
+New projects should be created using the *Automated kanban* template.
+
+Branches
+--------
+
+In general, repositories should have only a default branch (usually ``master``) and pull request branches. If the repository is a fork, it may have a ``master`` branch for the source branch and an ``opencontracting`` (or ``open_contracting``) branch for the fork branch.
+
+If a repository needs multiple branches (like the standard and profiles), the needed branches should be protected. Otherwise, unprotected branches more than a month old should either be opened as pull requests, protected, or deleted.
+
+See the pages for the branch management of the `standard <../../standard/technical/repository>`__ and `profiles <../../profiles/technical/repository>`__ (including OC4IDS).
+
+Repository settings
+-------------------
+
+Metadata
+~~~~~~~~
+
+1. Add a description. The description should not describe the project’s status (‘draft’), because people frequently forget to update repository descriptions. Describe the status in the readme instead.
+2. Add a website to the repository, if relevant: for example, a link to a deployment of the tool or to its documentation.
+
+The ``fix:lint_repos`` and ``fix:protect_branches`` Rake tasks in `standard-maintenance-scripts <https://github.com/open-contracting/standard-maintenance-scripts>`__ otherwise normalize configurations.
+
+We don’t generally enable the following behaviors on `protected branches <https://help.github.com/articles/about-protected-branches/>`__ for the provided reasons:
+
+-  **Require branches to be up to date before merging**: While this may avoid introducing errors, it slows development in an environment in which there are many simultaneous pull requests, because each would require an extra step before merging. If the automated tests fail after merging, the error can be corrected, or the changes can be reverted.
+-  **Require pull request reviews before merging**: While this is a best practice, it slows development as the team is not sufficiently large to staff it. It is okay, for example, for an author to self-merge a simple change. Authors may, of course, request reviews for significant changes.
+
+Archival
+~~~~~~~~
+
+Repositories that are no longer supported should be archived.
+
+1. Agree whether to archive the repository. The archived repositories presently include:
+
+   -  Superseded repositories (e.g. `json-merge-patch <https://github.com/OpenDataServices/json-merge-patch>`__ supersedes `jsonmerge <https://github.com/open-contracting-archive/jsonmerge>`__)
+   -  Abandoned extensions (e.g. `ocds-equityTransferCaps-extension <https://github.com/open-contracting-archive/ocds-equityTransferCaps-extension>`__)
+   -  Merged changes to the core standard, expressed as extension repositories (``ocds_upgrade_###``)
+   -  Exploratory repositories from pre-1.0 and pre-2015
+
+2. Scan the repository’s open issues, milestones, pull requests and non-default branches in case any can be quickly closed, merged or deleted. Counter `GitHub’s recommendation <https://help.github.com/articles/about-archiving-repositories/>`__, open issues and pull requests indicate the development status of a repository, and should be left open.
+3. Change the repository’s description to describe the reason for archival. If the repository has been superseded, change it to “Superseded by [owner]/[repository]” and change the URL to the new repository’s URL.
+4. Run the ```fix:archive_repos REPOS=repo1,repo2`` <https://github.com/open-contracting/standard-maintenance-scripts#change-github-repository-configuration>`__ task on the repository.
+5. Move the archive to the ``open-contracting-archive`` organization.
+6. `Archive <https://help.github.com/articles/about-archiving-repositories/>`__ the repository through its settings.
+7. Run the ```local:badges`` <https://github.com/open-contracting/standard-maintenance-scripts#change-github-repository-configuration>`__ task.
+
+Integrations
+------------
+
+-  **Coveralls**, to measure test coverage
+-  **ReadTheDocs**, to build repository-specific documentation (`access all docs <https://github.com/open-contracting/standard-maintenance-scripts/blob/master/badges.md#readme>`__)
+-  **GitHub Actions**, to run tests, and to build documentation (`view all badges <https://github.com/open-contracting/standard-maintenance-scripts/blob/master/badges.md#readme>`__)

docs/index.rst

@@ -0,0 +1,10 @@
+OCP Software Development Handbook
+=================================
+
+A guide for developers of OCP's tools.
+
+.. toctree::
+
+   github
+   python
+   services

docs/python/applications.rst

@@ -0,0 +1,62 @@
+Applications
+============
+
+Python applications are different from :doc:`packages` in that:
+
+-  Applications are not declared as dependencies by other software, and therefore do not have a ``setup.py`` file
+-  Applications are deployed to servers, and therefore freeze requirements in ``requirements.txt`` to have consistent deploys
+
+The ``master`` branch of applications should always be deployable, which requires that:
+
+-  Tests pass on continuous integration
+-  Installation instructions are consistent with the ```deploy`` <https://github.com/open-contracting/deploy>`__ repository
+
+If installation instructions change (e.g. if a new service like Redis is required), then the ``deploy`` repository must be updated.
+
+Requirements
+------------
+
+Requirements are managed by four files at the root of a repository:
+
+-  ``requirements.in`` names all direct requirements needed in the production environment, i.e. all packages ``import``\ ’ed by the application.
+
+   -  If the application is incompatible with older or newer versions of a requirement, use the least specific `version specifier <https://www.python.org/dev/peps/pep-0440/#version-specifiers>`__ possible, for example:
+
+      -  requires newer versions: use ``foo>=1.2`` instead of ``foo>=1.2.3``
+      -  requires older versions: use ``foo<2``
+      -  requires versions range: use ``foo>=1.2,<2``
+
+-  ``requirements_dev.in`` names all direct requirements needed exclusively in the development environment, and not in the production environment, e.g. \ ``pytest``.
+
+   -  This file typically also includes the direct requirements needed in the production environment, by having a first line of ``-r requirements.txt``.
+
+-  ``requirements.txt`` names all direct and indirect requirements needed in the production environment, all locked to specific versions.
+-  ``requirements_dev.txt`` names all direct and indirect requirements needed in the development environment, all locked to specific versions.
+
+The above ensures that:
+
+-  Development and production environments use the same versions of production requirements, to avoid errors or surprises during or after deployment due to differences between versions (e.g. a new version of Django requires upgrading application code).
+-  Different developers and continuous integration use the same versions of development requirements, to avoid unexpected test failures due to differences between versions (e.g. a new version of pytest requires upgrading test code, or a new version of flake8 has stricter linting rules).
+
+The ``requirements*.txt`` files should be periodically updated, both for security updates and to better distribute the maintenance burden of upgrading versions over time. ``pip-tools`` is used to manage the ``requirements*.txt`` files (it is included in ``requirements_dev.*``).
+
+To upgrade all dependencies:
+
+.. code:: shell
+
+   pip-compile --upgrade
+   pip-compile --upgrade requirements_dev.in
+
+To upgrade one dependency, run, for example:
+
+.. code:: shell
+
+   pip-compile -P requests
+   pip-compile -P requests requirements_dev.in
+
+After adding a dependency, run:
+
+.. code:: shell
+
+   pip-compile
+   pip-compile requirements_dev.in