master branch has changed to a neutral, stripped version. In order to use it, you would have to implement your configuration/handlers in a separate extension. For an example, have a look at this extension.
The original master has now moved to the aivl_master branch.
Native CiviCRM extension for Amnesty International Flanders (AIVL) to import street recruitment and welcome call csvfiles into CiviCRM. The extension was created initially by Erik Hommel (CiviCooP) and Björn Endres (Systopia) for AIVL.
[1 BASIC FUNCTIONALITY] (#basic-functionality)
[2 INSTALLATION] (#installation)
[3 IMPORT SETTINGS] (#import-settings)
[3.1 Employee handling errors] (#employee-handling-errors)
[3.2 Employee doing folluw up calls] (#employee-doing-follow-up-calls)
[3.3 Folder to get CSV files from] (#folder-to-get-csv-files-from)
[3.4 Folder to move processed CSV files to] (#folder-to-move-processed-csv-files-to)
[3.5 Folder to move failed CSV files to] (#folder-to-move-failed-csv-files-to)
[3.6 Group for newsletter] (#newsletter-group)
[3.7 Membership type] (#membership-type)
[3.8 Offset days for SDD mandate] (#offset-days-for-sdd)
[3.9 Offset days for Follow Up activity] (#offset-days-for-follow-up-activity)
[3.10 Phone type for landlines] (#phone-type-for-landlines)
[3.11 Phone type for mobiles] (#phone-type-for-mobiles)
[3.12 Location type for address, email and first phones] (#location-type-for-address-and-1st-phone)
[3.13 Location type for additional phones] (#location-type-for-additional-phones)
[3.14 Default country] (#default-country)
[3.15 Default financial type for SDD] (#default-financial-type-for-sdd)
[3.16 Prefix for individual or household] (#prefix-for-individualhousehold)
[3.17 Gender for female contacts imported] (#gender-for-female-import)
[3.18 Gender for male contacts imported] (#gender-for-male-imports)
[3.19 Gender for unknown import] (#gender-for-unknown-import)
[3.20 Relationship types for employees or other] (#relationship-types-for-otheremployee)
[3.21 Date format used in CSV] (#date-format-of-csv)
[3.22 Group for deduping contacts] (#group-for-deduping-new-contacts)
AIVL use street recruitment to get new donors and SEPA Direct Debits (SDD). The actual street recruitment is done by a supplier, who follows up the recruitment with a welcoming call to the new donor within a week of recruitment. Daily they will put a csv file with either street recruitment data or welcome call data on the server, which will be imported by this extension. The records will contain:
- data identifying the recruiting organization and the recruiter
- donor data (name, address, phones, email, birth date, gender, newsletter Y/N, become member, follow up call)
- SEPA mandate data (bank account, frequency, amount, mandate reference, start date, end date)
Depending on the configuration the street recruitment record import will automatically do the following:
- a new contact for the recruiter if it does not exist yet with the contact sub type 'recruiter' and an active relationship 'recruiter for' with the recruiting organization
- a new contact created for the donor, also storing the donorID at the recruitment organization in a custom group. The recrutier contact id will be found with the API Contact Identify, which is part of the extension de.systopia.identitytracker, so you will need that extension to be able to run the streetimport extension. This dependency will be checked when you install the streetimport extension.
- the contact will be added to the newsletter group if appropriate
- a membership will be created for the contact if appropriate
- an activity of the type 'Street Recruitment' will be added, with the recruiter as the activity source and the donor as the target contact with status scheduled. In a custom group linked to this activity type all the imported data will be recorded as a snapshot of the street recruitment.
- an SDD is generated (a recurring contribution wtih SEPA mandate and a specific campaign, the mandate reference being generated by the recruiting organization)
- if appropriate, an activity of the type 'Follow Up Call' will be generated with the status scheduled, with the recruiter as the source contact, the donor as the target and the fundraiser in the settings (see section Import Settings) as the assignee contact
- if any error occurred in the process, the data will still be imported if possible but an activity of the type 'Import Error' will be created with the flagged problem, assigned to the error handling employee specified in the settings (see section Import Settings)
Depending on the configuration the welcome call record import will automatically do the following:
- a new contact for the recruiter if it does not exist yet with the contact sub type 'recruiter' and an active relationship 'recruiter for' with the recruiting organziation
- it will lookup the donor with the donorID of the recruiting organization, using the custom group where the donorID was stored at Street Recruitment
- it will add phone number(s) if there is one for the donor that is not in CiviCRM yet
- it will add email(s) if there is one for the donor that is not in CiviCRM yet
- it will add an address for the donor if there is one that CiviCRM does not know yet, and set that new address as the primary one
- it will update birth date and gender if it has to
- it will add a bank account if not there yet
- the contact will be added to the newsletter group if appropriate if it not there yet
- a membership will be created for the contact if appropriate and not existing yet
- an activity of the type 'Welcome Call' will be added, with the recruiter as the activity source and the donor as the target contact with status scheduled. In a custom group linked to this activity type all the imported data will be recorded as a snapshot of the welcome call.
- the SDD will be updated or even deleted if required. If the SDD can not be removed because stuff has been sent to the bank, the SDD will be ended and a new one will be added with a generated mandate reference
- if appropriate, an activity of the type 'Follow Up Call' will be generated with the status scheduled, with the recruiter as the source contact, the donor as the target and the fundraiser in the settings (see section Import Settings) as the assignee contact
- if any error occurred in the process, the data will still be imported if possible but an activity of the type 'Import Error' will be created with the flagged problem, assigned to the error handling employee specified in the settings (see section Import Settings)
Once the complete file is processed, the file will be moved to a folder specified in the settings. If the import of the file fails, the file will be moved to a folder specified in the settings.
You can install the extension by downloading a zip file from GitHub or by pulling, fetching or cloning the repository. You can then use the CiviCRM manage extensions menu option to install the extension.
In the resources folder are a couple of CiviCRM entities that will be created upon installation of the extension (or whenever the extension is referenced in CiviCRM). This is done in the CRM_Streetimport_Config class constructor. The Config object is instantiated in the install hook of the streetimport.php file. The entities are in JSON-files:
- activity_types.json
- contact_sub_types.json
- custom_data.json
- groups.json
- option_Groups.json
- relationship_types.json
You can adapt these files to suit your needs, but please check the CRM_Streetimport_Config to understand what the impact is. This extension is created specifically for AIVL in their context. You are quite welcome to use and change this extension for your own needs but you will have to ensure you understand the structure before you do :-)
The settings used in the import process are stored in a JSON file import_settings.json in the resources folder of the extension. You can manipulate the JSON file to update the settings, but there is also an option in the CiviCRM menu Administer/CiviContribute with the name AIVL Import Settings. If you click on this option you will see all the import settings. When you hit the save button, the results will be stored in the JSON file import_settings.json in the resources folder. All settings will be discussed below (you might get Dutch headings if you have a Dutch CiviCRM installation but we think you will understand anyway).
During the import whenever an error is logged, an activity of the type Import Error will be created. In this setting you select the contact that will be set as one the activity will be assigned to. In the select list you will get a list of all contacts in your database that are considered as an employee (based on the setting Relationship types for other/employee as AIVL has more than one relationship type that can signify an employee in the sense for this setting. By default it will show those contacts that have the 'employee' relationship. It is possible that you have to change the Relationship types for other/employee first, save the settings and then select the contacts.
If a donor during street recruitment or welcome call answered that they would like a follow up call, an activity of the type FollowUp Call will be created with the status scheduled. In this setting you select the contact that will be set as one the activity will be assigned to. In the select list you will get a list of all contacts in your database that are considered as an employee (based on the setting Relationship types for other/employee as AIVL has more than one relationship type that can signify an employee in the sense for this setting. By default it will show those contacts that have the 'employee' relationship. It is possible that you have to change the Relationship types for other/employee first, save the settings and then select the contacts.
The scheduled job that calls the street recruitment process api will by default try to read all CSV files in this folder in alphabetical order.
Once the CSV file has been processed completely, it will be moved to this folder to make sure the files are kept but not processed more than one time. A log file (showing the name of the file and ending on .log) will detail what has been done.
Once the CSV file fails, it will be moved to this folder. A log file (showing the name of the file and ending on .log) will details what the error was.
If the CSV record for the donor has a value that is seen as 'yes' in the newsletter column, the contact will be added to this group.
If the CSV record for the donor has a value that is seen as 'yes' in the membership column, a membership of this type will be added (if it does not exist yet)
When the mandate is created, this is the number of days that will be added to the start date before the first transaction will be done.
When the follow up activity is created, the activity will be scheduled for the run date + the amount of days specified in this setting.
The phone type that will be used for the first and second land ('normal') phone numbers in the donor CSV record.
The phone type that will be used for the first and second mobile phone numbers in the donor CSV record.
The location type that will be used for the address, email address and the first phones (poth landline and mobile) for the donor CSV record.
The location type that will be used for the second phone number (landline and/or mobile) for the donor CSV record.
The country that will be used as a default when no country is entered in the donor CSV record.
The financial type that will be used when generating the SDD.
In the CSV record for the donor there will be column 'organization yes/no'. If this is set to yes, a contact of the type 'Organization' will be generated for the donor. If this column is set to no, the prefix of the CSV record will determine if a contact of the type 'Individual' or of the type 'Household' will be generated.
In the CSV record the prefix will determine the gender. The value 'mevrouw' signifies a male, and this is the gender id that will be used when creating the contact in CiviCRM.
In the CSV record the prefix will determine the gender. The value 'meneer' signifies a female, and this is the gender id that will be used when creating the contact in CiviCRM.
In the CSV record the prefix will determine the gender. If the value is not 'meneer' or 'mevrouw', this is the gender id that will be used when creating the contact in CiviCRM.
When specifying the employee dealing with the Import Errors or Folluw Up Calls, you do not get a select list of all contacts in your database but only 'your' employees. By default the 'Employee of' relationship type id will be used, but here you can specify more relationsships to be seen as employees.
Details the date format that is used in the CSV import files. You can select an option from the list, other types are not allowed.
All imported contacts will be added to this group with the purpose of deduping the newly imported contacts regularly.