CoVE is an web application to Convert, Validate and Explore data following certain open data standards - currently 360Giving and the Open Contracting Data standard.
Python HTML CSS Gherkin Shell
Clone or download
Latest commit 67a7efe Jul 2, 2018
Permalink
Failed to load latest commit information.
.tx Split out cove_iati translations into their own file Jan 16, 2018
cove Add T&Cs for review of submitted data to assess website operation Jun 21, 2018
cove_360 Update translations from Transifex Jun 20, 2018
cove_iati [#1030] Update flatten-tool version Jun 28, 2018
cove_ocds Merge branch 'master' into 1022-human-dates Jul 2, 2018
docs Update IATI CoVE live server name Mar 14, 2018
fts [#601] Split out tests. Add coveragerc and change travis. Mar 22, 2017
.coveragerc [#601] Split out tests. Add coveragerc and change travis. Mar 22, 2017
.gitignore [#899] Add org-ids.json.lock to .gitignore Dec 14, 2017
.travis.yml Add script to fetch secret data Apr 25, 2018
AGPLv3.txt Adds AGPL and Licence documents. Fixes Readme typo Apr 28, 2015
CONTRIBUTING.md Add some more information about the branches in this repo Apr 28, 2016
LICENCE.rst Merge branch 'master' into update-license-info Jul 4, 2017
README.rst Merge branch 'master' into 948-cd-docs Mar 13, 2018
ROADMAP.rst [#443] Belatedly add the roadmap from August Nov 11, 2016
get_secret_data.sh Update secret data test archive Apr 25, 2018
iati-cli [#827] iati cli: add iati-cli shell command Sep 7, 2017
manage.py [#601] manage.py: add a single manage.py instead of manage360.py, etc… Mar 17, 2017
ocds-cli [#827] iati cli: add iati-cli shell command Sep 7, 2017
pytest.ini Rename to cove Apr 29, 2015
requirements.in Merge branch 'master' into 1022-human-dates Jul 2, 2018
requirements.txt Merge branch 'master' into 1022-human-dates Jul 2, 2018
requirements_dev.in [#951] Render markdown in schema descriptions Mar 8, 2018
requirements_dev.txt Merge branch 'master' into 1022-human-dates Jul 2, 2018
requirements_iati.txt [#1030] Update flatten-tool version Jun 28, 2018
run_tests.sh Pass arguments to run_tests.sh to py.test Jul 10, 2017
secret_data_test.py Run secret data test against OCDS web interface Apr 26, 2018
setup.cfg [#89] Refactor explore view function Sep 23, 2015
setup.py [#307] Add a list of all the functions in cove.lib Jun 6, 2016
update_requirements.sh Update requirements Aug 9, 2017

README.rst

CoVE - Convert Validate & Explore

https://travis-ci.org/OpenDataServices/cove.svg?branch=master Requirements Status https://coveralls.io/repos/OpenDataServices/cove/badge.png?branch=master

Development work on CoVE by Open Data Services is carried out in sprints. The issues for each sprint can be found at https://github.com/OpenDataServices/cove/projects . Other work is carried out from time to time, and contributions from the community are welcome. Outstanding issues for CoVE can be found at https://github.com/OpenDataServices/cove/issues . Please report any bugs!

Introduction

CoVE is an web application to Convert, Validate and Explore data following certain open data standards - currently 360Giving and the Open Contracting Data standard. http://cove.opendataservices.coop

Why convert data?

The W3C Data on the Web Best Practices recommend making open data available in a range of formats to meet the needs of different users. Developers may want JSON, researchers might prefer a spreadsheet format.

CoVE manages the process of converting between JSON, Excel and CSV formats for structured data.

Validate and Explore

CoVE presents key validation information during the process, and can be configured to display information about the contents of a data file, so that it can be easily inspected.

Supported formats

CoVE currently supports conversion from:

  • JSON to multi-tabbed Excel files
  • Excel to JSON (it uses the flatten-tool for conversion)

If a JSON schema is supplied, CoVE can use either field names or user-friendly column titles.

User Flows

Overviews of how users flow through the application are maintained at https://docs.google.com/drawings/d/1pVbEu6dJaVk8t3NctjYuE5irsqltc9Th0gVQ_zeJyFA/edit and https://docs.google.com/drawings/d/1wFH4lZlBZWso7Tj_g7CyTF3YaFfnly59sVufpztmEg8/edit

Release Cycle

CoVE is in constant development. There are public instances in use at: https://dataquality.threesixtygiving.org/ http://standard.open-contracting.org/validator/

We deploy the latest version of CoVE at the end of each calendar month (usually the last Thursday of the month). We make a development version ready for user testing (mainly internally) two weeks before deployment. Our cut off date for new features to be considered in that cycle is the week before that.

Feature requests, bugs, questions and answers etc are all handled via GitHub. We use release cycle milestones to organise those issues. We also use GitHub projects as a way to prioritise issues and indicate what is being worked on.

Serious Bug fixes and 'priority' features, that need to make it into a release at short notice can be included by negotiation.

Requirements

This application is built using Django and Python 3.5

Installation

Steps to installation:

  • Clone the repository
  • Change into the cloned repository
  • Create a virtual environment (note this application uses python3)
  • Activate the virtual environment
  • Install dependencies
  • Set up the database (sqlite3)
  • Compile the translations
  • Run the development server
git clone https://github.com/OpenDataServices/cove.git
cd cove
virtualenv .ve --python=/usr/bin/python3
source .ve/bin/activate
pip install -r requirements_dev.txt
python manage.py migrate
python manage.py compilemessages

Then, for 360Giving run:

DJANGO_SETTINGS_MODULE=cove_360.settings python manage.py runserver

Or for OCDS run:

DJANGO_SETTINGS_MODULE=cove_ocds.settings python manage.py runserver

Follow the instructions in your terminal to open the application in your browser.

Extra installation steps for IATI

The following steps are for Ubuntu but equivalent packages are available for other distros.

sudo apt-get install build-essential libxml2-dev libxslt1-dev python3-dev
pip install -r requirements_iati.txt

Then run the development server:

DJANGO_SETTINGS_MODULE=cove_iati.settings python manage.py runserver

Deployment

See https://cove.readthedocs.io/en/latest/deployment/

Run tests

./run_tests.sh

To run functional tests with a different browser:

BROWSER=Chrome ./run_tests.sh

See http://selenium-python.readthedocs.io/api.html for browser options.

To generate a coverage report (in the htmlcov directory):

py.test --cov cove --cov-report html

The tests include functional tests (actually interacting with the website in selenium). These can also be run against a deployed copy of the website:

CUSTOM_SERVER_URL=http://dev.cove.opendataservices.coop py.test fts

We also use flake8 to test code quality, see https://github.com/OpenDataServices/developer-docs/blob/master/tests.md#flake8

The development requirements include xdist to allow running tests in parallel:

py.test -n2

Translations

We use Django's translation framework to provide this application in different languages.
We have used Google Translate to perform initial translations from English, but expect those translations to be worked on by humans over time.

Translations for Translators

Translators can provide translations for this application by becomming a collaborator on Transifex https://www.transifex.com/OpenDataServices/cove

Translations for Developers

For more information about Django's translation framework, see https://docs.djangoproject.com/en/1.8/topics/i18n/translation/

If you add new text to the interface, ensure to wrap it in the relevant gettext blocks/functions.

In order to generate messages and post them on Transifex:

First check the Transifex lock, because only one branch can be translated on Transifex at a time.

Then:

python manage.py makemessages -l en
tx push -s

In order to fetch messages from transifex:

tx pull -a

In order to compile them:

python manage.py compilemessages

Keep the makemessages and pull messages steps in thier own commits seperate from the text changes.

To check that all new text is written so that it is able to be translated you could install and run django-template-i18n-lint

pip install django-template-i18n-lint
django-template-i18n-lint cove

Adding and updating requirements

Add a new requirements to requirements.in or requirements_dev.in depending on whether it is just a development requirement or not.

Then, run ./update_requirements --new-only this will populate requirements.txt and/or requirements_dev.txt with pinned versions of the new requirement and it's dependencies.

WARNING: The ./update_requirements script will delete and recreate your current .ve directory.

./update_requirements without any flags will update all pinned requirements to the latest version. Generally we don't want to do this at the same time as adding a new dependency, to make testing any problems easier.

Command Line Interface

CoVE for OCDS and IATI can be run from the command line. To get a JSON file with validation errors plus other key information, use the following command:

OCDS

./ocds-cli --options file-name

file-name can be a JSON or an Excel file.

Options:

--output-dir -o Directory where the output will be created, defaults to the name of the file.

--exclude-file -e Do not include the file in the output directory.

--delete -d Delete the output directory if it already exists.

--schema-version -s Version of the schema to validate the data.

--convert -c Convert data from nested (JSON) to flat format (Excel and CSV). This option is redundant for spreadsheets as they are always converted to JSON format.

IATI

./iati-cli --options file-name

file-name can be a XML or an Excel/CSV file.

Options:

--output-dir -o Directory where the output will be created, defaults to the name of the file.

--exclude-file -e Do not include the file in the output directory.

--delete -d Delete the output directory if it already exists.

--orgids -i Run org-ids rule check for IATI identifier prefixes.

--openag -a Run ruleset checks for IATI OpenAg data.

If the file is in spreadsheet format, the output directory will contain a unflattened.xml file converted from Excel or CSV to XML format

OpenaAg rulesets check that the data contains the XML elements <opeang:tag> and <location>, and that they include the right attributes expected for OpenAg data. Please read OpenAg ruleset feature files (written in Gerkhin style) for more information.

Org-ids rulesets check that all organisation identifiers are prefixed with a registered org-ids prefix. Please read Org-ids ruleset feature file for more information