Skip to content
A Singer tap for extracting data from Oracle Responsys' SFTP Exports.
Python
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
tap_responsys
tests
.gitignore
CHANGELOG.md
LICENSE
README.md
setup.py

README.md

tap-responsys

This is a Singer tap that produces JSON-formatted data following the Singer spec.

This tap pulls data from CSV files exported to an SFTP server through Responsys' Connect export jobs. Features of the extraction are:

  • Automatic Stream Name discovery with the pattern [optional_date_prefix]stream_name[optional_date_suffix].[csv|txt]
  • Automatic Schema discovery by sampling 1000 records from the first 5 files found per stream
  • Bookmarking on the file's last_modified timestamp, and only requesting new files greater than that value on future runs with --state specified

Requirements of Exports

In order to handle the wealth of options available for exports in Responsys Connect, some things are assumed about the files being discovered and extracted using this tap:

  • Ready files are required to ensure all data has been written before reading, the extension ready should be used.
  • UTF-8 is required
  • CSV exports must be comma-delimited and quoted with a double-quote character (")
  • Column headers are strictly required
  • Encryption and compression are not supported at this time

Quick start

  1. Install

    > pip install tap-responsys
  2. Upload your public key to the SFTP server used by Responsys

    The tap will need to use an SSH public/private keypair to connect to the SFTP server. In order for this to work, the server needs to trust the public key.

    If using the hosted Responsys SFTP Server: You can work with Responsys to get the public key uploaded and trusted on the server.

    If using a custom SFTP server: Consult with the administrator of that server to upload and trust the SSH public key file.

  3. Create the config file

    Create a JSON file called config.json containing the local path to the private key file, and server information.

    {
      "private_key_file": "~/.ssh/responsys_rsa",
      "username": "test_scp",
      "start_date": "2017-09-21T00:00:00Z",
      "path": "exports",
      "host": "files.responsys.net",
      "port": "22"
    }
  4. Run the application

    Discovery mode

    tap-responsys --config config.json --discover > catalog.json

    Sync Mode

    You only need to add "selected": true metadata to the stream level in the catalog, since fields are selected automatically. Once that is done, you can run sync mode using this command, with optional state from a previous run:

    tap-responsys --config config.json --catalog catalog.json [--state state.json]

Copyright © 2019 Stitch

You can’t perform that action at this time.