Allow for lookback window on incremental load - Stripe/All Sources

[JCWahoo]

Tell us about the problem you're trying to solve

I'm attempting to replace my existing Stitch(Singer) Stripe integration w/ Airbyte. I've run into a few scenarios where updates are arriving late to existing objects (charges, invoices, subscriptions). Due to this, I must use a full refresh rather than incremental load for those streams

Describe the solution you’d like

In reviewing the Singer spec, their connection is using a 30 day lookback window for incremental load. I'd like the ability to set this lookback window as part of any incremental source, but specifically Stripe for this use case. Rather than requesting records where >= last_cursor, I'd like to retrieve where >= last_cursor - {my prescribed interval in days})

Describe the alternative you’ve considered or used

I'm currently using Stitch :-)

Additional context

By either adding this default lookback or allowing the user to choose, we're meeting/exceeding the capabilities of our competitors

Are you willing to submit a PR?

This is one where I feel like it sounds easy enough to add the lookback, though I suspect this cursor implementation has further ramifications depending on source

Acceptance Criteria

1. Add an optional integer input lookback_window_days to the stripe connector which causes the connector . This option should default to 0. The option should also be annotated with advanced=true for when there is a separate advanced options section in the UI.

[sherifnada]

thanks for the suggestion @JCWahoo ! Will make this available soon. I've updated the description to make it clearer what airbyte should do when supporting this.

[eliziario]

/publish-cdk dry-run=false

❌ https://github.com/airbytehq/airbyte/actions/runs/1234013232
❌ https://github.com/airbytehq/airbyte/actions/runs/1234013232
❌ https://github.com/airbytehq/airbyte/actions/runs/1234013232
❌ https://github.com/airbytehq/airbyte/actions/runs/1234013232

[sherifnada]

@eliziario moving this out of the sprint for now

[JCWahoo]

Do we know when it might be worked on?

…

On Fri, Sep 24, 2021, 1:33 AM Sherif A. Nada ***@***.***> wrote: @eliziario <https://github.com/eliziario> moving this out of the sprint for now — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#5765 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AUAW7TBP4D6J6Y3ZJHOO33TUDQEUDANCNFSM5DEYSIYQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[avida]

@JCWahoo Added lookback_window_days parameter please update connector to 0.1.20 version

[JCWahoo]

@dmytro It doesnt appear I can save this setting, both save and retest are not available [image: image.png]

…

On Tue, Oct 5, 2021 at 5:55 AM Dmytro ***@***.***> wrote: @JCWahoo <https://github.com/JCWahoo> Added lookback_window_days parameter please update connector to 0.1.20 version — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#5765 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AUAW7THGVGPFVC4UKJUBEXTUFLDRZANCNFSM5DEYSIYQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.