SOAP Synchronization

How It Works

The ETL process follows the following steps:

• Request data from the SOAP Webservices
• Load any raw data fields by copying values over
• After all models are updated with new data, execute update_relationships which updates appropriate foreign keys. For example, linking a position to its post

Management Commands and Scheduling

The API provides the following management commands to allow for scheduling and manually executing synchronizations (For further help with these commands, you can call python manage.py COMMAND help):

schedule_synchronization_job

This command allows for the scheduling of synchronization jobs on a per-model basis

Examples:

• python manage.py schedule_synchronization_job --list - Lists all current synchronization jobs
• python manage.py schedule_synchronization_job --set-defaults - Creates a default set of synchronization jobs
• python manage.py schedule_synchronization_job --reset-all - Resets the last synchronization date for all models
• python manage.py schedule_synchronization_job --reset position.Position - Resets the last synchronization date for the position.Position model job
• python manage.py schedule_synchronization_job MODEL DELTA PRIORITY - Schedules or updates a synchronization job for MODEL, using DELTA time in seconds between synchronizations. Priority indicates the order it will be executed in relative to other jobs, lower is sooner.
• python manage.py schedule_synchronization_job --remove position.Position - Removes the specified model's synchronization job

synchronize_data

This command synchronizes the data from the SOAP webservices by executing all jobs created with the schedule_synchronization_job command whose time delta has passed since their last synchronization.

Examples:

• python manage.py synchronize_data --list - List all synchronization jobs
• python manage.py synchronize_data - Synchronizes all jobs whose time delta has passed since last synchronization
• python manage.py synchronize_data --model position.Position - Synchronizes only the position.Position model job

Environment Variables

The following environment variables must be specified in setup_environment.sh to use SOAP synchronization services.

Variable Name	Description
WSDL_LOCATION	The location of the WSDL file
WSDL_SSL_CERT	The certificate used to authenticate the SOAP connection - if not specified, will generate a warning; but does not prevent use
DJANGO_SYNCHRONIZATION_HEADER_ANY_STRING	Used to specify HTTP headers for use with the SOAP synchronization client; you may specify any number of these variables if you require more than one header. For example: DJANGO_SYNCHRONIZATION_HEADER_IP="X-thumbprint-client=ipip" and DJANGO_SYNCHRONIZATION_HEADER_CLIENTNAME="X-client-name=talentmap" would set the X-thumbprint-client and X-client-name headers
DJANGO_SOAP_NS_OVERRIDE_ANY_STRING	Used to override namespace definitions in XML requests to the SOAP service; like DJANGO_SYNCHRONIZATION_HEADER_ANY_STRING, you can specify multiples

Recurring Synchronizations

To automate synchronization, use the following script with a cron job:

#!/bin/bash
source /path/to/virtualenvironment/bin/activate/
source /path/to/api/code/setup_environment.sh
cd /path/to/api/code/
python manage.py synchronize_data

Troubleshooting

If data is not synchronizing properly, you should attempt the following:

• Ensure there are synchronization jobs (python manage.py synchronize_data --list) - if not, create the default set (python manage.py schedule_synchronization_job --set-defaults)
• Verify all environment variables are set to appropriate values
• Validate any SSL certificates are in place
• Validate that the environment is being properly sourced by your shell and/or script
• Validate that the database migrations are up-to-date
• Check with the SOAP service provider that the services are turned on
• Check with the SOAP service provider that the IP of the machine is whitelisted
• Check with the SOAP service provider that your requests are valid

Transformations

Standard data transformations:

• Incoming boolean values are converted to python booleans
• Dates are converted to standard ISO format and UTC timezone
• Extra spaces are stripped from strings

Data Mapping

Please find below a list of mappings from SOAP XML responses to TalentMAP data models. You can find these mappings, and any extra ETL processes, by checking the source. The mappings are broken up by RequestName, which delineates the data provided by the webservice.

SOAP RequestName 'skill'

SOAP XML Tag	TalentMAP Model	Model Field
code	position.Skill	code
description	position.Skill	description

SOAP RequestName 'grade'

SOAP XML Tag	TalentMAP Model	Model Field
code	position.Grade	code

SOAP RequestName 'tods'

SOAP XML Tag	TalentMAP Model	Model Field
code	organization.TourOfDuty	code
short_description	organization.TourOfDuty	short_description
long_description	organization.TourOfDuty	long_description
months	organization.TourOfDuty	months
status	organization.TourOfDuty	_status
is_active	organization.TourOfDuty	is_active

SOAP RequestName 'organization'

SOAP XML Tag	TalentMAP Model	Model Field
code	organization.Organization	code
short_description	organization.Organization	short_description
long_description	organization.Organization	long_description
parent_organization	organization.Organization	_parent_organization_code
bureau_organization	organization.Organization	_parent_bureau_code
city_code	organization.Organization	_location_code
is_bureau	organization.Organization	is_bureau
is_regional	organization.Organization	is_regional

SOAP RequestName 'language'

SOAP XML Tag	TalentMAP Model	Model Field
code	language.Language	code
long_description	language.Language	long_description
short_description	language.Language	short_description

SOAP RequestName 'country'

SOAP XML Tag	TalentMAP Model	Model Field
code	organization.Country	code
name	organization.Country	name
short_name	organization.Country	short_name
short_code	organization.Country	short_code
location_prefix	organization.Country	location_prefix

SOAP RequestName 'location'

SOAP XML Tag	TalentMAP Model	Model Field
code	organization.Location	code
city	organization.Location	city
state	organization.Location	state
country	organization.Location	_country

SOAP RequestName 'orgpost'

SOAP XML Tag	TalentMAP Model	Model Field
location_code	organization.Post	_location_code
tod_code	organization.Post	tour_of_duty (Foreign keyed by code)
cost_of_living_adjustment	organization.Post	cost_of_living_adjustment
differential_rate	organization.Post	differential_rate
rest_relaxation_point	organization.Post	rest_relaxation_point
danger_pay	organization.Post	danger_pay
has_consumable_allowance	organization.Post	has_consumable_allowance
has_service_needs_differential	organization.Post	has_service_needs_differential

SOAP RequestName 'positioncapsule'

SOAP XML Tag	TalentMAP Model	Model Field
POSITION_ID	position.CapsuleDescription	_pos_seq_num
CONTENT	position.CapsuleDescription	content
DATE_CREATED	position.CapsuleDescription	date_created
DATE_UPDATED	position.CapsuleDescription	date_updated

SOAP RequestName 'position'

SOAP XML Tag	TalentMAP Model	Model Field
pos_id	position.Position	_seq_num
position_number	position.Position	position_number
title	position.Position	title
org_code	position.Position	_org_code
bureau_code	position.Position	_bureau_code
skill_code	position.Position	_skill_code
is_overseas	position.Position	is_overseas
grade	position.Position	_grade_code
tod_code	position.Position	tour_of_duty (Foreign keyed by code)
language_code_1	position.Position	_language_1_code
language_code_2	position.Position	_language_2_code
location_code	position.Position	_location_code
spoken_proficiency_1	position.Position	_language_1_spoken_proficiency_code
reading_proficiency_1	position.Position	_language_1_reading_proficiency_code
spoken_proficiency_2	position.Position	_language_2_spoken_proficiency_code
reading_proficiency_2	position.Position	_language_2_reading_proficiency_code
create_date	position.Position	create_date
update_date	position.Position	update_date
effective_date	position.Position	effective_date

SOAP RequestName 'jobcategoryskill'

SOAP XML Tag	TalentMAP Model	Model Field
JC_ID	position.SkillCone	_id
JC_NM_TXT	position.SkillCone	name
skills	position.SkillCone	_skill_codes (From nested tag code)

SOAP RequestName 'cycles'

SOAP XML Tag	TalentMAP Model	Model Field
id	bidding.BidCycle	_id
name	bidding.BidCycle	name
category_code	bidding.BidCycle	_category_code
BEGIN_DATE // CYCLE	bidding.BidCycle	cycle_start_date
END_DATE // CYCLE	bidding.BidCycle	cycle_end_date
BEING_DATE // BIDDUE	bidding.BidCycle	cycle_deadline_date

SOAP RequestName 'availableposition'

SOAP XML Tag	TalentMAP Model	Model Field
STATUS_CODE	position.Position	status_code
STATUS	position.Position	status
DATE_UPDATED	position.Position	effective_date
CP_POST_DT	position.Position	posted_date
TED	position.Assignment	estimated_end_date