JSON Validator and CKAN Search
POD Schema is used for JSON validation http://project-open-data.github.io/schema/
Package search API v.3 of data.gov catalog is used for search http://catalog.data.gov/api/3/action/package_search
Full agencies and their data.json urls are listed on the POD Dashboard http://data.civicagency.org/offices
-
Download the
composer.pharexecutable or use the installer.$ curl -sS https://getcomposer.org/installer | php -
Run Composer:
$ php composer.phar install
-
Copy
inc/config.sample.phptoinc/config.phpand check its values
-
Check and update
resources/agency_json_urls.csv. The format is simple:"AGENCY_TITLE", json_url"Department of Agriculture",http://www.usda.gov/data.json "Department of Education",http://www.ed.gov/data.json "Department of Energy",http://www.energy.gov/data.json -
To download latest JSONs run
$ php cli/download.php
The data/agency_json_download.log will contain overall statistics about latest json update, ex.:
Importing Department of Defense json . . . . . . . . . . . NETWORK ERROR 404
Importing Department of Education json . . . . . . . . . . .SUCCESS (FIXED UTF8)
Importing Department of Labor json . . . . . . . . . . . . SUCCESS
Importing Department of the Interior json . . . . . . . . . .SUCCESS (REPLACED "][" WITH "," )
Importing Department of the Treasury json . . . . . . . . . .SUCCESS
Importing Department of Transportation json . . . . . . . . . INVALID_JSON - Syntax error, malformed JSON
Importing Department of Veterans Affairs json . . . . . . . . SUCCESS
Importing General Services Administration json . . . . . . . . SUCCESS (REMOVED BOM)
To get latest federal and non-federal schemas run
$ php cli/update-schemas.phpBefore running the script, think of updating schema.json and updating dependencies:
$ php composer.phar update- Put all your JSON datasets to
/data/folder OR download them using download.php
Files must be in JSON, named by *.json pattern
example1.jsondepartment_treasury.jsonlast_department.json
-
Run script
$ php cli/validate.php
-
Grab the results from
/results/{date}_VALIDATION folderThe results will be called using data files name, with _results postfix.
example1_results.jsonexample1_results.csvdepartment_treasury_results.jsondepartment_treasury_results.csvlast_department_results.jsonlast_department_results.csv
Json file contains json validation information, and csv file has CKAN search results
The processing.log in same folder will give you some overall statistics information.
- JSON online editor (http://www.jsoneditoronline.org)
- POD online json validator (http://project-open-data.github.io/json-validator/)
- The CKAN API Documentation (http://docs.ckan.org/en/latest/api.html#ckan.logic.action.get.package_search)