Skip to content
No description, website, or topics provided.
Python
Branch: master
Clone or download
jacobrobertbaca and nick-mccoy Readme (#9)
* added readme file

* made small edit to file

* changed brackets

* Update README.md
Latest commit 5321324 Aug 19, 2019
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
bin add script to populate test dbs Jul 25, 2019
spikes add support for full table sync Jul 30, 2019
tap_mongodb changed exception message Aug 16, 2019
.gitignore Fix replication_method -> replication-method catalog config typo Nov 1, 2018
LICENSE add readme and license Oct 15, 2018
README.md Readme (#9) Aug 19, 2019
setup.py bumped version to 0.0.2 Aug 16, 2019

README.md

tap-mongodb

This is a Singer tap that produces JSON-formatted data following the Singer spec from a MongoDB source.

Set up Virtual Environment

python3 -m venv ~/.virtualenvs/tap-mongodb
source ~/.virtualenvs/tap-mongodb/bin/activate

Install tap

pip install -U pip setuptools
pip install tap-mongodb

Set up Config file

Create json file called config.json, with the following contents:

{
  "password": "<password>",
  "user": "<username>",
  "host": "<host ip address>",
  "port": "<port>",
  "database": "<database name>",
}

All of the above attributes are required by the tap to connect to your mongo instance.

Run in discovery mode

Run the following command and redirect the output into the catalog file

tap-mongodb --config ~/config.json --discover > ~/catalog.json

Your catalog file should now look like this:

{
  "streams": [
    {
      "table_name": "<table name>",
      "tap_stream_id": "<tap_stream_id>",
      "metadata": [
        {
          "breadcrumb": [],
          "metadata": {
            "row-count":<int>,
            "is-view": <bool>,
            "database-name": "<database name>",
            "table-key-properties": [
              "_id"
            ],
            "valid-replication-keys": [
              "_id"
            ]
          }
        }
      ],
      "stream": "<stream name>",
      "schema": {
        "type": "object"
      }
    }
  ]
}

Edit Catalog file

Using valid json, edit the config.json file

To select a stream, enter the following to the stream's metadata:

"selected": true,
"replication-method": <replication method>,

<replication-method> must be either FULL_TABLE or LOG_BASED

To add a projection to a stream, add the following to the stream's metadata field:

"tap-mongodb.projection": <projection>

For example, if you were to edit the example stream to select the stream as well as add a projection, config.json should look this:

{
  "streams": [
    {
      "table_name": "<table name>",
      "tap_stream_id": "<tap_stream_id>",
      "metadata": [
        {
          "breadcrumb": [],
          "metadata": {
            "row-count": <int>,
            "is-view": <bool>,
            "database-name": "<database name>",
            "table-key-properties": [
              "_id"
            ],
            "valid-replication-keys": [
              "_id"
            ],
            "selected": true,
            "replication-method": "<replication method>",
            "tap-mongodb.projection": "<projection>"
          }
        }
      ],
      "stream": "<stream name>",
      "schema": {
        "type": "object"
      }
    }
  ]
}

Run in sync mode:

tap-mongodb --config ~/config.json --catalog ~/catalog.json

The tap will write bookmarks to stdout which can be captured and passed as an optional --state state.json parameter to the tap for the next sync.

Supplemental MongoDB Info

Local MongoDB Setup

If you haven't yet set up a local mongodb client, follow these instructions

Copyright © 2019 Stitch

You can’t perform that action at this time.