This API is utilized by USAspending.gov to obtain all federal spending data which is open source and provided to the public as part of the DATA Act.
Creating a Development Environment
Ensure the following dependencies are installed and working prior to continuing:
Requirements
docker
which will handle the other application dependencies.docker-compose
bash
or another Unix Shell equivalent- Bash is available on Windows as Windows Subsystem for Linux
git
make
for running common build/test/run targets inMakefile
- TIP: Review handy short-hand
make
targets inMakefile
after getting familiar with the below setup
- TIP: Review handy short-hand
If not using Docker:
Using Docker is recommended since it provides a clean environment. Setting up your own local environment requires some technical abilities and experience with modern software tools.
- Command line package manager
- Windows' WSL bash uses
apt-get
- MacOS users will use
Homebrew
- Linux users already know their package manager (yum, apt, pacman, etc.)
- Windows' WSL bash uses
PostgreSQL
version 10.x (with a dedicateddata_store_api
database)Elasticsearch
version 7.1- Python 3.7 environment
- Highly recommended to use a virtual environment. There are various tools and associated instructions depending on preferences
- See Required Python Libraries for an example using
pyenv
Cloning the Repository
Now, navigate to the base file directory where you will store the USAspending repositories
$ mkdir -p usaspending && cd usaspending
$ git clone https://github.com/fedspendingtransparency/usaspending-api.git
$ cd usaspending-api
.env
File
Create Your Copy the template .env
file with comment local runtime environment variables defined. Change as needed for your environment. This file is git-ignored and will not be committed by git if changed.
$ cp .env.template .env
usaspending-backend
Docker Image
Build This image is used as the basis for running application components and running containerized setup services.
$ docker-compose --profile usaspending build
requirements/requirements-app.txt
), since they are baked into the docker image at build-time.
Database Setup
A postgres database is required to run the app. You can run it in a postgres
docker container (preferred), or run a PostgreSQL server on your local machine. In either case, it will be empty until data is loaded.
⚠️ If running your own PostgreSQL server be sure to:- Have a DB named
data_store_api
- A superuser role (user), e.g.
ALTER ROLE <<role/user you created>> WITH SUPERUSER;
- Cross-check your
.env
or.envrc
files if used to be sure it references your DBs user, password, host, and port where needed
- Have a DB named
Start the Postgres DB Container
If not using your own local install...
$ docker-compose --profile usaspending up usaspending-db
... will create and run a Postgres database.
Bring DB Schema Up-to-Date
-
docker-compose run --rm usaspending-manage python3 -u manage.py migrate
will run Django migrations: https://docs.djangoproject.com/en/2.2/topics/migrations/. -
docker-compose run --rm usaspending-manage python3 -u manage.py matview_runner --dependencies
will provision the materialized views which are required by certain API endpoints.
Seeding and Loading Database Data
To just get essential reference data, you can run:
docker-compose run --rm usaspending-manage python3 -u manage.py load_reference_data
will load essential reference data (agencies, program activity codes, CFDA program data, country codes, and others).
To download a full production snapshot of the database or a subset of the database and loading it into PostgreSQL, use the pg_restore
tool as described here: USAspending Database Download
- Recreate matviews with the command documented in the previous section if this is done
Executing individual data-loaders to load in data is also possible, but requires more familiarity with those ad-hoc scripts and commands, and also requires an external data source (DATA Broker DB, or external file, etc.) from which to load the data.
- For details on loading reference data, DATA Act Broker submissions, and current USAspending data into the API, see loading_data.md.
- For details on how our data loaders modify incoming data, see data_reformatting.md.
Elasticsearch Setup
Some of the API endpoints reach into Elasticsearch for data.
$ docker-compose --profile usaspending up usaspending-es`
... will create and start a single-node Elasticsearch cluster as a docker container with data persisted to a docker volume.
-
The cluster should be reachable via at http://localhost:9200 ("You Know, for Search").
-
Optionally, to see log output, use
docker-compose logs usaspending-es
(these logs are stored by docker even if you don't use this).
Generate Elasticsearch Indexes
The following will generate two base indexes, one for transactions and one for awards:
$ docker-compose run --rm usaspending-manage python3 -u manage.py elasticsearch_indexer --create-new-index --index-name 01-26-2022-transactions --load-type transaction
$ docker-compose run --rm usaspending-manage python3 -u manage.py elasticsearch_indexer --create-new-index --index-name 01-26-2022-awards --load-type award
Running the API
docker-compose --profile usaspending up usaspending-api
... will bring up the Django app for the RESTful API
- You can update environment variables in
settings.py
(buckets, elasticsearch, local paths) and they will be mounted and used when you run this.
The application will now be available at http://localhost:8000
.
Note: if the code was run outside of Docker then compiled Python files will potentially trip up the docker environment. A useful command to run for clearing out the files on your host is:
find . | grep -E "(__pycache__|\.pyc|\.pyo$)" | xargs rm -rf
Using the API
In your local development environment, available API endpoints may be found at http://localhost:8000/docs/endpoints
Deployed production API endpoints and docs are found by following links here: https://api.usaspending.gov
Running Tests
Test Setup
To run all USAspending tests in the docker services run
docker-compose run --rm -e DATA_BROKER_DATABASE_URL='' usaspending-test
NOTE: If an env var named DATA_BROKER_DATABASE_URL
is set, Broker Integration tests will attempt to be run as well. If doing so, Broker dependencies must be met (see below) or ALL tests will fail hard. Running the above command with -e DATA_BROKER_DATABASE_URL=''
is a precaution to keep them excluded, unless you really want them (see below if so).
To run tests locally and not in the docker services, you need:
- Postgres A running PostgreSQL database server (See Database Setup above)
- Elasticsearch A running Elasticsearch cluster (See Elasticsearch Setup above)
- Required Python Libraries Python package dependencies downloaded and discoverable (See below)
- Environment Variables Tell python where to connect to the various data stores (See below)
Once these are satisfied, run:
(usaspending-api) $ pytest
Required Python Libraries
Create and activate the virtual environment using venv
, and ensure the right version of Python 3.7.x is being used (the latest RHEL package available for python36u
: as of this writing)
$ pyenv install 3.7.2
$ pyenv local 3.7.2
$ python -m venv .venv/usaspending-api
$ source .venv/usaspending-api/bin/activate
Your prompt should then look as below to show you are in the virtual environment named usaspending-api
(to exit that virtual environment, simply type deactivate
at the prompt).
(usaspending-api) $
pip
install
application dependencies
(usaspending-api) $ pip install -r requirements/requirements.txt
Environment Variables
.envrc
File
direnv is a shell extension that automatically runs shell commands in a .envrc
file (commonly env var export
commands) when entering or exiting a folder with that file
Create a .envrc
file in the repo root, which will be ignored by git. Change credentials and ports as-needed for your local dev environment.
export DATABASE_URL=postgres://usaspending:usaspender@localhost:5432/data_store_api
export ES_HOSTNAME=http://localhost:9200
export DATA_BROKER_DATABASE_URL=postgres://admin:root@localhost:5435/data_broker
If direnv
does not pick this up after saving the file, type
$ direnv allow
Alternatively, you could skip using direnv
and just export these variables in your shell environment.
.env
File
Declaring NAME=VALUE
variables in a git-ignored .env
file is a common way to manage environment variables in a declarative file. Certain tools, like docker-compose
, will read and honor these variables.
If you copied .env.template
to .env
, then review any variables you want to change to be consistent with your local runtime environment.
Including Broker Integration Tests
Some automated integration tests run against a Broker database. If certain dependencies to run such integration tests are not satisfied, those tests will bail out and be marked as Skipped.
(You can see messages about those skipped tests by adding the -rs
flag to pytest, like: pytest -rs
)
To satisfy these dependencies and include execution of these tests, do the following:
-
Ensure the
Broker
source code is checked out alongside this repo at../data-act-broker-backend
-
Ensure you have
Docker
installed and running on your machine -
Ensure you have built the
Broker
backend Docker image by running:(usaspending-api) $ docker build -t dataact-broker-backend ../data-act-broker-backend
-
Ensure you have the
DATA_BROKER_DATABASE_URL
environment variable set, and it points to what will be a live PostgreSQL server (no database required) at the time tests are run.- WARNING: If this is set at all, then ALL above dependencies must be met or ALL tests will fail (Django will try this connection on ALL tests' run)
- This DB could be one you always have running in a local Postgres instance, or one you spin up in a Docker container just before tests are run
-
If invoking
pytest
within a docker container (e.g. using theusaspending-test
container), you must mount the host's docker socket. This is declared already in thedocker-compose.yml
file services, but would be done manually with:-v /var/run/docker.sock:/var/run/docker.sock
NOTE: Broker source code should be re-fetched and image rebuilt to ensure latest integration is tested
Re-running the test suite using pytest -rs
with these dependencies satisfied should yield no more skips of the broker integration tests.
Example Test Invocations of Just a Few Broker Integration Tests: (i.e. using -k
)
From within a container
(NOTE: DATA_BROKER_DATABASE_URL
is set in the docker-compose.yml
file (and could pick up .env
values, if set)
(usaspending-api) $ docker-compose run --rm usaspending-test pytest --capture=no --verbose --tb=auto --no-cov --log-cli-level=INFO -k test_broker_integration
From Developer Desktop
(NOTE: DATA_BROKER_DATABASE_URL
is set in the .envrc
file and available in the shell)
(usaspending-api) $ pytest --capture=no --verbose --tb=auto --no-cov --log-cli-level=INFO -k test_broker_integration
Contributing
To submit fixes or enhancements, or to suggest changes, see CONTRIBUTING.md