Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP
moving data from DabbleDB to Democracy in Action CRM
PHP JavaScript Ruby
branch: master

Fetching latest commit…

Cannot retrieve the latest commit at this time

Failed to load latest commit information.
application
config
lib
public
scraps
test
.gitignore
Main.csv
README.md
clear_dia.php5
load_dia.php5
load_orm.php5
proof_of_concept.php5
start.php5

README.md

Center for Progressive Leadership

Dabble to DIA synchronization code

This project is licensed to CPL and RadicalDesigns under GPLv2.

All code for this project is under version control with git. A public repository exists at: http://github.com/radicaldesigns/cpl-data

The git executable can be found on the production server at: ~/lib/git-1.6.3/git

Database interactions are managed via the Kohana ORM framework, documentation and source available here: http://www.kohanaphp.com/

This project includes a testing suite based on Simpletest. To run the testing suite, run: php5 test/all_unit_tests.php

Overview

DIA to Dabble synchronization

Data is gathered from DIA and stored on the CPL server via a cron job, set to run hourly. The cron runs the following script:

public/nominations.php5

You can edit the cron using crontab -e

The nominations script stores data in the members and memberships tables in a mysql database.

Once a member record is pulled from DIA, it will not be updated. To reset a member record in the sync, it is necessary to remove the record from the members table.

Only DIA supporter records for targetted groups are pulled across into the local system. To change which groups are supported, you can add new DIA group_KEY values to the ‘targetted_groups’ array.

To Dabble

Dabble does not support push integration, instead it offers “Scheduled Imports”. Each Dabble database has 2 imports scheduled to run hourly, one for nominators, one for nominees.

To review the information being sent to Dabble, you’ll need a DIA group_KEY. Then check the following url:
http://[user:pass@domain-name]/dabble/nominators.php5?group_id=[group key here]

This page should be protected via HTTP Basic Auth.
This script can be edited in public/dabble/nominators.php5

Creating a new Dabble Import

The scheduled imports in dabble have helpful names like “OH PLF Nominators” and “OH PLF Nominees”. It is not possible to change the column mapping of an existing import, but creating a new import from scratch shouldn’t be difficult.

  • Choose “Import Data from a Website”

  • Enter the url below with the DIA group you’d like to import, and click Continue
    http://[user:pass@domain-name]/dabble/nominators.php5?group_id=[group key here]

    • You’ll want to be able to see the column headers coming in, so click “Don’t use as Field Names”
    • Do not import ID or supporter_key
    • PLF_Channel and PLF_Detail correspond to “What medium did you hear about PLF...” and “Specific Details on How....”, and the CPL_Channel and CPL_Detail columns are similar
    • the “Nominated_By” field is the name of the nominator, and that should come in for nominee imports only. The “Online Nominator” field is the email, and it should not be imported
    • nominated_for_program, groups_key, dabble_id, last_modified_dia, and last_modified_dabble are all not used and should not be imported
    • The Nominee and Prospect ( or sometimes Nominator ) fields correspond to the checkbox fields for those attributes

    • Once the mapping is complete, be sure to click “Don’t use as field names” at the left side of the field map to hide the column headers.

    • You’ll want to select “Update existing entries when key data matches”, to match incoming data with existing data. The KEY column should be “Email”. “Ignore Blank Values” should be checked.

Dabble to DIA Synchronization

Data is gathered from Dabble and pushed to DIA on an hourly basis. A cron job is set to run the following script:
public/dabble_data.php5

Each of the state Dabble databases has a view called DIA Outbound. This view has a unique url which exports it to JSON format. The Dabble data is converted to DIA format, then saved in a table called diasupporters on the CPL server.

Only Dabble records with an identifying column checked are synced to DIA. As part of the sync, the supporter is linked to one or more DIA groups, representing different phases of the application process. These group ids are saved into a column in the diasupporters table called groups. If the checked columns change, the groups values change as well, and the sync system can remove and add group memberships in DIA as necessary.

The DIA sync process uses a library called democracyinaction-php, which is available online here: http://github.com/radicaldesigns/democracyinaction-php

Technical Overview

Configuration Files

There is a configuration file for the mysql database connection named application/config/database.php. The options within the file should be clear to a database-savvy individual.

The main configuration file is config/application.php5.

Configuration “Environments”

This file is set up to support up multiple run environments on a single server. The standard environments are:

production:
Connects to CPLs main Dabble databases and DIA instance

development:
Connects to sample files pulled from Dabble and included in the repository, along with a demonstration DIA instance. For use during code testing

test:
For use in conjunction with the automated testing suite.

common:
Settings in this environment will be included in all other environments, unless they are overridden by environment-specific settings.

The current environment is specified at the top of config/application.php5, in this form:

define( 'APP_ENV', 'development');

The APP_ENV value is then used to pull a group of settings from the $config_setup array.

The current environment settings are stored in a global variable named $config for the duration of script execution.

Configuration Settings

*dabble_sources: *
an array with the form ‘state_name’ => ‘resource_url’.
These are the JSON formatted dabble views to be synced across to DIA.
This setting accepts local filenames or web urls.

democracy_in_action:
an array with the keys ‘node’, ‘login’, and ‘password’. ‘node’ generally indicates the domain on which your DIA account is accessed, minus the prefix ‘hq-’.

*dabble_to_dia: *
a group of settings governing dabble to DIA sync

active_states:
an array of state names, matching the keys defined in dabble_sources. Only states included in active_states will be pulled from Dabble.

group_mappings:
an array of the form ‘Dabble column name’ => DIA group id OR custom function

Each Dabble record is checked for each of the column names defined in the group mappings.
If none of the defined column names exist and are checked, the record is not saved into the diasupporters table.
If the column exists and is checked, the record is marked to sync to the indicated DIA group id.
If a function name is provided instead of a numeric group id, the record is passed to that function for further evaluation of which DIA group it should sync to. The custom function is expected to return a DIA group id.

application_status_mappings
an array of the form

‘state_name’ => array( ‘incomplete’ => DIA group id, ‘complete’ => DIA group id )

this setting is used by the function assess_application_status to check whether an applicant has completed their application or not, and return the appropriate DIA group for each of these cases.

*dia_to_dabble: *
a group of settings governing DIA to dabble sync

targetted_groups:
An array of the form

‘state_name’ => array( ‘Nominator’ => DIA group id, ‘Nominee’ => DIA group id )

The defined group ids are pulled across from DIA. Nominator and Nominee groups contain different columns when they are presented for output.

Something went wrong with that request. Please try again.