Skip to content
ReVAL: Reusable Validation & Aggregation Library - A Django App for validating and aggregating data via API and web interface
Python HTML
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
.circleci Update dependencies (#35) May 31, 2019
data_ingest
docs Add token authentication for `validate` endpoint (#53) Sep 6, 2019
examples
json
.bandit Add token authentication for `validate` endpoint (#53) Sep 6, 2019
.cfignore
.coveragerc Add support for CI Jun 20, 2018
.flake8
.gitattributes
.gitignore Template swap fix (#29) May 15, 2019
CONTRIBUTING.md generated by 18f init Apr 11, 2018
LICENSE.md generated by 18f init Apr 11, 2018
Pipfile Change all reference from django-data-ingest to ReVAL (#47) Jun 14, 2019
Pipfile.lock Add token authentication for `validate` endpoint (#53) Sep 6, 2019
README.md Change all reference from django-data-ingest to ReVAL (#47) Jun 14, 2019
runtests.py Add token authentication for `validate` endpoint (#53) Sep 6, 2019
setup.py Update Django (#52) Aug 26, 2019

README.md

ReVAL

ReVAL (Reusable Validation & Aggregation Library) is a Django App for validating and aggregating data via API and web interface.

For the web interface, it can manage data submitted as file uploads to a central gathering point, and it can perform data validation, basic change tracking and duplicate file handling. Each file generally contains multiple data rows, and each user may submit multiple files.

For the API, it can perform data validation, and view uploads that were done via the web interface.


Features

  • Flexible input format
  • Validation with:
  • Row-by-row feedback on validation results
  • Manage and track status of data submissions
  • Re-submit previous submissions
  • Flexible ultimate destination for data
  • API for validation

Requirements

  • Python (3.5, 3.6)
  • Django (1.11)
  • Goodtables
  • pyyaml
  • djangorestframework
  • psycopg2
  • json_logic_qubit
  • dj-database-url
  • requests

Quick Installation

If you would like to use ReVAL in your Django project, install using pipenv in your project...

  • Replace <version> with the latest tag i.e. v0.2 or
  • Replace with master if you would like to work with the latest development version
pipenv install -e git+https://github.com/18F/ReVAL.git@<version>#egg=data-ingest

Add 'rest_framework', and 'data_ingest' to your INSTALLED_APPS setting.

INSTALLED_APPS = (
    ...
    'rest_framework',
    'data_ingest',
)

Please see default installation for more setup instructions.


Examples

Several examples are provided to demonstrate default and customized behavior of data_ingest. Follow the development instructions to close this repository and install the dependencies required for the examples.

default installation

p02_budgets

p03_budgets


API

To perform data validation with API, see API documentation.


Deployment on Cloud.gov

All of the examples provided will show you how to run them locally. If you are interested in using cloud.gov as your platform, here's a basic installation guide on cloud.gov deployment.

Contributing

See CONTRIBUTING for additional information.


Development

To start developing on Django Data Ingest, clone the repository:

git clone git@github.com:18f/ReVAL.git

Install development dependencies:

pipenv install --dev

If you run into any issues installing packages in the Pipfile, you can try to install the particular package individually again.

Test Suite

To execute the test suite, install the development dependencies and run:

python runtests.py

Public domain

This project is in the worldwide public domain. As stated in CONTRIBUTING:

This project is in the public domain within the United States, and copyright and related rights in the work worldwide are waived through the CC0 1.0 Universal public domain dedication.

All contributions to this project will be released under the CC0 dedication. By submitting a pull request, you are agreeing to comply with this waiver of copyright interest.

You can’t perform that action at this time.