Discovery: OASIS Market Research
Welcome to the wonderful world of procurement, a mechanism by which the government buys goods and services from the private sector. The procurement process is made up of several stages, one of the earliest being market research. Market Research is an attempt to predict what qualified bidders your solicitation for goods or services will attract. It helps acquisition personnel make decisions regarding various aspects of their particular procurement. Ideally, a solicitation should be structured so that it attracts a enough experienced bidders to be truly competitive.
After researching the marketplace, procurement officers may decide to break their solicitation up into smaller chunks, or choose one vehicle or schedule over another. Procurement vehicles and schedules are large umbrella contracts, with a set of vendors that have "pre-competed" for work in a general category. Discovery was developed to serve the OASIS family of vehicles, which has many vendors capable of providing integrated professional services. OASIS is actually two vehicles, OASIS SB, which includes small businesses only, and OASIS unrestricted, which includes a wider array of vendors.
Discovery is live at (https://discovery.gsa.gov). You can file issues on this repo, or you can check out our Trello board, which contains our product backlog to see what we have planned in the future.
Currently, Discovery makes use of vendor information from the System for Award Management (SAM) and historical contract information the Federal Procurement Data System (FPDS). There are a few management commands that load the initial data, but we also provide a SQL dump since the loading of data from FPDS for the past decade can take several hours.
This project uses Django, a Python framework. You can get started by:
Installing PostgreSQL (installation guides here)
Installing virtualenv and creating a virtual environment (hitchhiker's guide to virtualenv)
Installing the python requirements with
pip install -r requirements.txt
Creating a postgresql database and storing the settings in a
local_settings.pyfile, a sibling of
manage.py migrateto set up the database and
manage.py createcachetableto set up caching
Loading a SQL dump to get some data into the project
manage.py runserverto start the server. Optionally specify a host and port to serve on. For example
This project supports Python 2.7+. However for one package, there are different versions for Python 2 vs Python 3 package support, so there is a base
requirements.txt file for all of the common dependencies, and
requirements_py2.txt for python 2.7 and
requirements_py3.txt for python 3. There is also a
requirements-test.txt file for the testing packages (see below under Testing).
Getting the Data
The project comes with everything you need to get started, including fixtures for some static information, and management commands to load the rest of the data. The OASIS vehicles are broken out into categories or pools, which are enumerated in a fixture. The pools are defined by groups of NAICS codes. Vendors in the OASIS SB vehicle may also have certain setasides, such as women owned, service disabled veteran owned, etc. These setaside codes are also included in their own fixture. All of these fixtures can be found in
You have to options for getting the data. You can load the SQL dump provided above and get everything at once, or you can load the data manually.
To load the data manually, load each of the fixtures inside the
/vendor/fixtures/ directory. You can load these using the
loaddata manage command like so:
manage.py loaddata vendor/fixtures/naics.json
Make sure to load naics.json, pools.json, and setasides.json (in that order).
Now you can run the
load_vendors manage command. This loads in the vendors on OASIS (included in the repo on several CSVs that are grouped by pool), and fetches extra information about the vendors from SAM.
Note that this manage command requires you to specify a
SAM_API_KEY variable in your local settings file as shown in local_settings.example.py. This value should be a valid Data.gov API key. Left to its own devices, the loader runs slightly faster than the rate limiting on api.data.gov, so there is a sleep statement in the loader to keep it in check.
Once the server is started you can query the api at
Providing no query parameters will return all vendors. However you can also filter by NAICS shortcode or by setaside code.
will return vendors that have the setaside codes A5 and QF and also do business under the NAICS code 541330.
Once you've loaded the basic data, you'll need to load the contract history records for the Vendor detail pages. This is accomplished using the following manage command:
load_fpds command is configured to load ten years of contract history for each vendor so it takes some time (potentiall several hours). However, upon each successful run, a load object with the date and time is stored. Upon subsequent runs, the loader will only load contracts that have been created or modified since the last load date. You can also pass in parameters to modify the loader.
The --load_all flag forces a load of all contracts, regardless of last load date
The --id parameter only loads contractors for vendors with an id greater than or equal to ID, where ID is the vendor's id in the vendor table. Contracts are loaded in order of vendor id.
Deploying to cloud.gov
If the app does not yet exist in the target space, first create it:
cf push -f manifest.yml --no-start
Next, create and bind a database to the application:
cf create-service aws-rds shared-psql discovery-db cf bind-service acquisitions-staging discovery-db
Set some necessary environment variables:
cf set-env discovery-prod SAM_API_KEY [an api.data.gov key] cf set-env discovery-prod SECRET_KEY [a secret key] cf set-env discovery-prod API_HOST ""
Now, get the app up and running:
cf push -f manifest.yml
To load some data, SSH into the application using
cf ssh discovery-prod and follow the instructions above. See the cloud.gov documentation for additional information about working with databases.
Assets can be loaded locally, or from an S3 bucket. The default settings in settings.py and local_settings.example.py are set to force local assets to be loaded. To use S3, update the AWS settings in the local_settings.py file, and uncomment the
The tests can be run using Django's built in testing infrastructure:
You can test specific apps by passing in the app name to the testing command. Valid values are
selenium_tests (requires additional setup).
Locally with PhantomJS
- Install python testing requirements with
pip install -r requirements-test.txt
- Install Phantom JS
- From the project directory, run
manage.py test selenium_tests
Remotely using Sauce Labs
SAUCE = Falseto
SAUCE = True
- From the project directory, run
manage.py test selenium_tests
This project is in the public domain within the United States, and copyright and related rights in the work worldwide are waived through the CC0 1.0 Universal public domain dedication.
All contributions to this project will be released under the CC0 dedication. By submitting a pull request, you are agreeing to comply with this waiver of copyright interest.