Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Incompatible State Message Format Handling in tap-airbyte-wrapper for Incremental Extraction #17

Open
JichaoS opened this issue May 22, 2024 · 0 comments

Comments

@JichaoS
Copy link
Contributor

JichaoS commented May 22, 2024

Description:
Certain airbyte taps are incompatible with incremental extraction with tap-airbyte-wrapper

For incremental to work, airbyte is expecting a state message in this form, a list of AirbyteStateMessage

  [
    {
      "type": "STREAM",
      "stream": {
        "stream_descriptor": {
          "name": "events",
          "namespace": null
        },
        "stream_state": {
          "server_upload_time": "2024-05-13 19:35:13.833000"
        }
      }
    }
  ]

The internal dict above is actually what gets sent by airbyte in a state message. But the tap-airbyte-wrapper unpack this and saves the unpacked version in the state table, which then gets feds back without modifications to airbyte binary as state.json. This prevents certain airbyte extractors from not working. (For example intercom’s one works totally fine, while amplitude does not)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: No status
Development

No branches or pull requests

1 participant