Skip to content
Codebase for Dept of Ed Open Data Project
Branch: master
Clone or download
JGulic Merge pull request #185 from CivicActions/dataset-validation
Add not_empty validator to fields that were not validated
Latest commit 2e73dd4 Jun 21, 2019
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
bin Usdwds2.0 package item (#108) May 13, 2019
ckanext Merge branch 'master' into dataset-validation Jun 21, 2019
tests Setup testing Nov 15, 2018
.coveragerc Initial commit Aug 27, 2018
.gitignore Updated .gitignore Nov 15, 2018
.npmrc
.travis.yml Logged user show datasets (#82) May 8, 2019
LICENSE.md Renamed to ckanext-ed Nov 14, 2018
MANIFEST.in Fixed outstanding issues Nov 14, 2018
Makefile
README.md
gulpfile.js Search page filters and sidebar (#137) May 19, 2019
nightwatch.conf.js Setup testing Nov 15, 2018
package-lock.json Search page filters and sidebar (#137) May 19, 2019
package.json Stylesheets restrucutre (#132) May 17, 2019
setup.cfg Renamed to ckanext-ed Nov 14, 2018
setup.py Fixed outstanding issues Nov 14, 2018
test.ini Usdwds2.0 package item (#108) May 13, 2019

README.md

ckanext-ed

Build Status

This is the main repo for the US Department of Education ckan-based project. This documentation covers all of the development aspects.

Table of contents

Getting started

Requirements

Please follow installation instructions of the software below if needed. The following steps require:

  • docker
  • docker-compose
  • nvm/Node.js (optional)
  • /etc/hosts contains the 127.0.0.1 ckan-dev line

Setup environment

Clone the docker-ckan-ed repository (assuming that we're inside our projects directory):

$ git clone git@github.com:CivicActions/docker-ckan-ed.git
$ cd docker-ckan-ed

This is a docker compose setup for local development. It's designed to support live development of extensions which are stored inside the src directory.

Clone the ckanext-ed repository to the src folder (we also can create a symbolic link back to the projects directory):

$ cd src
$ git clone git@github.com:CivicActions/ckanext-ed.git
$ cd ckanext-ed

Now we have cloned all the required repositories and we're located in our main working directory docker-ckan-ed/src/ckanext-ed

Start development server

Take a look inside package.json to understand what's going on under the hood. Or if you don't have Node.js installed.

To start a development server, we have to build docker images:

$ make build

Let's start the development server:

For live development the first option is recommended to be launched in another terminal window

$ make start

You can work on the ckanext-ed codebase having it running. On every change to the codebase, the server will be reloaded automatically. It's important to mention that the ckan configuration and other things like cron/patches/etc are managed inside the docker-ckan-ed repo. If you want to update it you have to restart the server.

Now we can visit our local ckan instance at:

http://ckan-dev:5000/

To log in as an admin:

  • user: ckan_admin
  • pass: test1234

Development

Running unit tests

We write and store unit tests inside the ckanext/ed/tests directory. See CKAN documentation for more information regarding how to write tests. To run unit tests you need to be running the ckan-dev service.

In another terminal window run the test command:

$ make test

You can pass additional nosetests options inthe following way (note the two -- characters):

$ make ARGS='--ipdb --ipdb-failure' test

Right now all tests under ckanext/ed/tests will be run, it is not possible to choose a specific file or test.

Running E2E tests

For building static files, we can use npm. Let's enable a Node.js environment:

You have to have nvm installed or you can use any other way to get Node.js prepared - system setup etc

$ nvm install 10
$ nvm use 10
$ npm install

We write and store acceptance E2E tests inside the top-level tests directory. For running it we also have to switch to the test server:

$ make start

In another terminal window run the test command:

$ npm install nightwatch --no-save # Install test runner
$ npm run test:e2e

See the how to write E2E tests guide:

Working with static files

Put your scripts/fonts/etc inside the ckanext/ed/fanstatic folder and images inside the ckanext/ed/public folder. It can be used as usual ckan fanstatic and public contents.

At the same time, we use CSS preprocessor (LESS) to build our styles. Put your styles inside the ckanext/ed/less and build it:

$ npm run static:build # Option 1: one-time build
$ npm run static:watch # Option 2: build on every change

Processed styles will be put to the ckanext/ed/fanstatic/css folder.

Working with i18n

To extract i18n messages and compile the catalog we have to have our development server running.

In another terminal window run these commands:

$ make i18n_extract
$ make i18n_compile

See CKAN documentation for more on i18n management.

Testing email notifications

We use a fake SMTP server to test email notifications:

  • log into https://mailtrap.io/inboxes
  • select Demo Inbox
  • copy SMTP credentials
  • past to docker-ckan-ed:.env (mail service connection section)
  • restart the development server

Now all email sent by from ckan.lib.mailer import mail_user should be sent to the Demo Inbox at Mailtrap.

Generating data.json

See the "Open Data" reference: https://project-open-data.cio.gov/v1.1/schema/

See the metadata analysis regarding the project: https://docs.google.com/spreadsheets/d/1ZPRXxKCMST-z5Exvvuf0MxDVjE2l-4jC1oKWOnpDz9M/edit#gid=44728761

We generate data.json using our fork of ckanext-datajson at https://github.com/okfn/ckanext-datajson/tree/ed (the ed branch).

To update the translation map (package -> data.json) edit export_map/export.map.json. It uses a self-explanatory structure. Our focus is mostly on field and extra fields. We use ckanext-scheming so extra should be false for all relevant fields.

Log into the container

To issue commands inside a running container (after $ make start):

$ make bash

Now you can tweak the running ckan-dev docker container from inside. Please take into account that all changes will be lost after the next container restart.

Updating docker images

Sometimes we need to update the base docker images ckan/ckan-dev. We can do it using:

$ make pull
$ make build

Stop service containers

Stop all service containers

$ make stop

Stop specific service container

$ make SERVICE=ckan-dev stop

removing stopped service containers

Remove all service containers

$ make remove

Remove specific service container

$ make SERVICE=ckan-dev remove

Reseting docker

If you want to start everything from scratch there is a way to prune your docker environment:

It will destroy all your projects inside docker!!!

$ docker system prune -a --volumes

Generating TOC

To update this readme table of contents run:

$ make readme

Troubleshooting

The admin credentials don't work

There had been a bug in ckan-dev that was fixed. Run the following commands to update your ckan image:

$ make pull
$ make build

References

You can’t perform that action at this time.