This is a Singer tap that produces JSON-formatted data following the Singer spec.
This tap:
- Pulls raw data from the Twitter Ads API, version 7 using the Twitter Ads Python SDK.
Twitter Ads Tap: Sync Review: Summary of tap sync process and looping
Twitter Ads Hierarchy and Terminology
-
Extracts the following normal GET endpoints:
-
cards:
-
line_items (aka ad_groups)
- targeting_criteria (for each line_item)
-
tweets (scheduled and published, not draft)
-
Extracts the following targeting GET endpoints:
- targeting_conversations
- targeting_devices
- targeting_events (for countries)
- targeting_interests
- targeting_languages
- targeting_locations (for countries)
- targeting_network_operators (for countries)
- targeting_platform_versions
- targeting_platforms
- targeting_tv_markets
- targeting_tv_shows (for each)
-
Extracts Asyncronous Reports using:
- Supports many reports, each with Report Config Settings:
- Name: Name of report
- Entity: 7 Entity Types
- Segment: 20 Segmentation Types
- Granularity: DAY, HOUR, or TOTAL
- Twitter Ads Entity Segmentation Rules (Google Sheet)
- Metrics and Segmentation Rules
- active_entities
- async_queued_jobs
- async_job_status
- async_results (download URL)
- Supports many reports, each with Report Config Settings:
-
Outputs the schema for each resource
-
Incrementally pulls data based on the input state
Twitter Ads requires authentication headers using OAuth 1.0a with an access token obtained via 3-legged OAuth flow. The access token, once generated, is permanent, but request tokens are short-lived without a documented expiry. The process is described in Obtaining Ads Account Credential.
-
Install
Clone this repository, and then install using setup.py. We recommend using a virtualenv:
> virtualenv -p python3 venv > source venv/bin/activate > python setup.py install OR > cd .../tap-twitter-ads > pip install .
-
Dependent libraries The following dependent libraries were installed.
> pip install singer-python > pip install singer-tools > pip install target-stitch > pip install twitter-ads (v7.0.0)
-
Create your tap's
config.jsonwith the following parameters:start_date: Absolute beginning date for bookmarked endpoints.user_agent: Tap name and email address for API logging.- OAuth 1.0a credentials:
consumer_keyconsumer_secretaccess_tokenaccess_token_secret
account_ids: Comma-delimited list of Twitter Ad Account IDs.attribution_window: Number of days for latency look-back period to allow analytical reporting numbers to stabilize.with_deleted: true or false; specifies whether to include logically deleted records in the results.country_codes: Comma-delimited list of ISO 2-letter country codes for targeting and segmenttation.page_size: An optional parameter to configure custom page_size.reports: Object array of specified reports with name, entity, segment, and granularity.request_timeout: To configure the read and connect timeout for twitter-ads client. Default is 300 seconds.
{ "start_date": "2019-01-01T00:00:00Z", "user_agent": "tap-twitter-ads <api_user_email@your_company.com>", "consumer_key": "YOUR_TWITTER_ADS_CONSUMER_KEY", "consumer_secret": "YOUR_TWITTER_ADS_CONSUMER_SECRET", "access_token": "YOUR_TWITTER_ADS_ACCESS_TOKEN", "access_token_secret": "YOUR_TWITTER_ADS_ACCESS_TOKEN_SECRET", "account_ids": "id1, id2, id3", "attribution_window": "14", "with_deleted": "true", "country_codes": "US, CA, MX, DE", "page_size": 1000, "reports": [ { "name": "campaigns_genders_hourly_report", "enitity": "CAMPAIGN", "segment": "GENDER", "granularity": "HOUR" }, { "name": "line_items_regions_daily_report", "enitity": "LINE_ITEM", "segment": "REGIONS", "granularity": "DAY" } ], "request_timeout": 300 }Optionally, also create a
state.jsonfile.currently_syncingis an optional attribute used for identifying the last object to be synced in case the job is interrupted mid-stream. The next run would begin where the last job left off. Each bookmarked endpoint that supports INCREMENTAL syncs will be listed with its max last processed record based onupdated_at,created_at, orend_time(depending on the endpoint).{ "currently_syncing": "creatives", "bookmarks": { "accounts": { "account_id_1": "2019-06-11T13:37:55Z", "account_id_2": "2019-06-11T18:37:55Z", }, "account_media": { "account_id_1": "2019-06-19T19:48:42Z", "account_id_2": "2019-05-19T19:48:42Z" } "..." } } -
Run the Tap in Discovery Mode This creates a catalog.json for selecting objects/fields to integrate:
> tap-twitter-ads --config config.json --discover > catalog.json
See the Singer docs on discovery mode here.
-
Run the Tap in Sync Mode (with catalog) and write out to state file
For Sync mode:
> tap-twitter-ads --config tap_config.json --catalog catalog.json > state.json > tail -1 state.json > state.json.tmp && mv state.json.tmp state.json
To load to json files to verify outputs:
> tap-twitter-ads --config tap_config.json --catalog catalog.json | target-json > state.json > tail -1 state.json > state.json.tmp && mv state.json.tmp state.json
To pseudo-load to Stitch Import API with dry run:
> tap-twitter-ads --config tap_config.json --catalog catalog.json | target-stitch --config target_config.json --dry-run > state.json > tail -1 state.json > state.json.tmp && mv state.json.tmp state.json
-
Test the Tap
While developing the twitter Ads tap, the following utilities were run in accordance with Singer.io best practices: Pylint to improve code quality:
> pylint tap_twitter_ads -d missing-docstring -d logging-format-interpolation -d too-many-locals -d too-many-argumentsPylint test resulted in the following score:
Your code has been rated at 9.72/10.
To check the tap and verify working:
> tap-twitter-ads --config tap_config.json --catalog catalog.json | singer-check-tap > state.json > tail -1 state.json > state.json.tmp && mv state.json.tmp state.json
Check tap resulted in the following:
The output is valid. It contained 872667 messages for 62 streams. 63 schema messages 872432 record messages 172 state messages Details by stream: +--------------------------------------------+---------+---------+ | stream | records | schemas | +--------------------------------------------+---------+---------+ | account_media | 0 | 1 | | accounts | 1 | 1 | | accounts_conversion_tags_hourly_report | 0 | 1 | | accounts_daily_report | 87 | 1 | | accounts_metros_hourly_report | 226240 | 1 | | advertiser_business_categories | 16 | 1 | | tracking_tags | 64 | 1 | | campaigns | 2 | 1 | | campaigns_conversion_tags_daily_report | 0 | 1 | | campaigns_daily_report | 10 | 1 | | campaigns_genders_hourly_report | 504 | 1 | | cards_image_conversation | 0 | 1 | | cards_poll | 0 | 1 | | cards_video_conversation | 0 | 1 | | content_categories | 392 | 1 | | funding_instruments | 1 | 1 | | funding_instruments_age_daily_report | 175 | 1 | | funding_instruments_devices_hourly_report | 15288 | 1 | | funding_instruments_hourly_report | 84 | 1 | | iab_categories | 15 | 1 | | line_item_apps | 0 | 1 | | line_items | 2 | 1 | | line_items_conversion_tags_hourly_report | 0 | 1 | | line_items_daily_report | 10 | 1 | | line_items_platform_versions_hourly_report | 3108 | 1 | | line_items_platforms_hourly_report | 588 | 1 | | line_items_regions_daily_report | 630 | 1 | | media_creatives | 0 | 1 | | media_creatives_daily_report | 0 | 1 | | organic_tweets_daily_report | 82 | 2 | | organic_tweets_hourly_report | 1344 | 1 | | preroll_call_to_actions | 0 | 1 | | promotable_users | 0 | 1 | | promoted_accounts | 0 | 1 | | promoted_accounts_daily_report | 0 | 1 | | promoted_tweets | 2 | 1 | | promoted_tweets_daily_report | 10 | 1 | | promoted_tweets_interests_daily_report | 3210 | 1 | | promoted_tweets_keywords_hourly_report | 420 | 1 | | promoted_tweets_languages_daily_report | 205 | 1 | | scheduled_promoted_tweets | 0 | 1 | | tailored_audiences | 0 | 1 | | targeting_app_store_categories | 84 | 1 | | targeting_conversations | 41136 | 1 | | targeting_criteria | 10 | 1 | | targeting_devices | 355 | 1 | | targeting_events | 355 | 1 | | targeting_interests | 361 | 1 | | targeting_languages | 21 | 1 | | targeting_locations | 56682 | 1 | | targeting_network_operators | 161 | 1 | | targeting_platform_versions | 38 | 1 | | targeting_platforms | 4 | 1 | | targeting_tv_markets | 39 | 1 | | targeting_tv_shows | 520694 | 1 | | tweets | 2 | 1 | +--------------------------------------------+---------+---------+
Copyright © 2019 Stitch