Embulk Input Plugin for Marketo
Clone or download
luongnhattruong Merge pull request #90 from treasure-data/PLT-10964
Support Custom Objects for Marketo
Latest commit 9c34f8d Nov 19, 2018

README.md

Build Status Code Climate Test Coverage Gem Version

Marketo input plugin for Embulk

embulk-input-marketo is the gem preparing Embulk input plugins for Marketo.

  • Lead(lead)
  • Activity log(activity)
  • Lead by list(all_lead_with_list_id)
  • Lead by program(all_lead_with_program_id)
  • Campaign(campaign)
  • Assets Programs (program)

This plugin uses Marketo REST API.

Overview

Required Embulk version >= 0.8.33 (since 0.6.0).

  • Plugin type: input
  • Resume supported: no
  • Cleanup supported: no
  • Guess supported: no

Install

$ embulk gem install embulk-input-marketo

Configuration

API

Below parameters are shown in "Admin" > "Web Services" page in Marketo.

Base configuration parameter

All target have this configuration parameters

name required default value description
target true Marketo targets
account_id true Marketo Muchkin id
client_id true Marketo REST client id
client_secret true Marketo REST client secret
marketo_limit_interval_milis false 20 Marketo have limitation of 100 calls per 20 second. If REST API calls are failed they will wait this amount of time before retry
batch_size false 300 Token paging batch size. Some REST API support batch
max_return false 200 Max return for Endpoint that use offset paging

Bulk extract target configuration parameter (Lead and Activity)

All bulk extract target use this configuration parameter

name required default value description
from_date true Import data since this date. Example: 2017-10-11T06:43:24+00:00
fetch_days false 1 Amount of days to fetch since from_date
polling_interval_second false 60 Amount of time to wait between pooling job status in second
bulk_job_timeout_second false 3600 Amount of time to wait for bulk job to complete in second
incremental false true If incremental is set to true, next run will have from_date set to the previous to_date(calculated by from_date + fetch_days)
incremental_column false createdAt Column use to filter from_date and to_date

Lead

Lead target extract all Marketo leads, it use Marketo bulk extract feature. Configuration include bulk extract configuration.

target: lead

Configuration:

name required default value description
use_updated_at false false Support filter with updateAt column, but not all Marketo Account have the feature to filter by updatedAt, updatedAt don't support incremental ingestion
included_fields false null List of lead fields to included in export request sent to Marketo, can be used to reduce the size of BulkExtract file

Schema type: Dynamic via describe lead endpoint.

Incremental support: yes

Range ingestion: yes

Activity

Activity target extract all Marketo activity log. Configuration include all bulk extract configuration

target: activity

Schema type: Static schema

Incremental support: yes

Range ingestion: yes

Campaign

Campaign extract all campaign data from Marketo

target: campaign

Schema type: Static schema

Incremental support: no

Range ingestion: no

Lead by list

Extract all Lead data including lead's list id

target: all_lead_with_list_id

Configuration:

name required default value description
included_fields false null List of lead fields to included in export request sent to Marketo, can be used to reduce request, response size

Schema type: Dynamic via describe leads. Schema will have 1 addition column name listId that contain the id of the list the lead belong to

Incremental support: no

Range ingestion: no

Lead by program

Extract all Lead data including lead's program id

target: all_lead_with_program_id

Configuration:

name required default value description
included_fields false null List of lead fields to included in export request sent to Marketo, can be used to reduce request, response size

Schema type: Dynamic via describe leads. Schema will have 1 addition column name listId that contain the id of the list the lead belong to

Incremental support: no

Range ingestion: no

Assets programs

Get Assets Programs by Query Tag type, Date range or all if no query by specified.

target: program

Configuration:

name required default value description
query_by false null Get assets programs by query, supported values date_range, tag_type leave unset to fetch all programs
earliest_updated_at false null Required if query by date_range is selected. Exclude programs prior to this date. Must be valid ISO-8601 string
latest_updated_at false null Required if query by date_range is selected. Exclude programs after this date. Must be valid ISO-8601 string
filter_type false null Optional value send with query by date_range is selected to filter out the result from Marketo. Supported values id, programId, folderId, workspace
filter_values false null Set the values associated with filter_type
tag_type false null Required if query by tag_type is selected. Type of program tag
tag_value false null Required if query by tag_type is selected. Value of the tag
report_duration false null Amount of milliseconds to fetch from earliest_updated_at. If incremental = true this value will automatically calculated for the first run by latest_updated_at - earliest_updated_at
incremental false true If incremental is set to true, next run will have earliest_updated_at set to the previous latest_updated_at + report_duration. Incremental import only support by query date_range

Schema type: Static schema

Incremental support: yes (Query by date_range only)

Range ingestion: yes

Example

For lead, you have partial-config.yml like below:

in:
  type: marketo
  target: lead
  account_id: ACCOUNT_ID
  client_id: CLIENT_ID
  client_secret: CLIENT_SECRET
  from_date: 2017-09-01
  fetch_days: 1
out:
  type: stdout

You can run embulk guess partial-config.yml -o lead-config.yml and got lead-config.yml. lead-config.yml includes a schema for Lead.

Next, you can run embulk preview lead-config.yml for preview and embulk run lead-config.yml for run.

Example of Assets Programs config

in:
  account_id: ACCOUNT_ID
  client_id: CLIENT_ID
  client_secret: CLIENT_SECRET
  target: program
  type: marketo
  query_by: date_range
  filter_type: folderId
  filter_values: 
   - 2598
   - 1001
  earliest_updated_at: 2018-08-20T00:00:00.000Z
  latest_updated_at: 2018-08-31T00:00:00.000Z
  incremental: true