FB Marketing: lookback window logic not functioning correctly

[sherifnada]

Expected Behavior

Insights data synced from facebook marketing could be updated up to 28 days after its generation. Therefore I expect the Airbyte connector to pull data from the past 28 days even with incremental sync.

Current Behavior

A user @MaxKrog has reported that the connector is only inserting records for the new days, not the past 28 days.

Logs

logs-9-0.txt


Slack conversation: https://airbytehq.slack.com/archives/C01MFR03D5W/p1617701356298100

Steps to Reproduce

1. Setup FB marketing connector to replicate incrementally
2. on the second sync, the connector does not replicate data from the past 28 days

Severity of the bug for you

High -- missing data

Version Information

see logs

[keu]

Insights streams have loopback logic when it requests data from the API at

airbyte/airbyte-integrations/connectors/source-facebook-marketing/source_facebook_marketing/client/api.py

Line 379 in 99f1448

buffered_start_date = self._state.subtract(days=self.buffer_days)

, but then it performs additional filtration as part of general read logic for all streams with states (the state doesn't take into account 28 days from the past):

airbyte/airbyte-integrations/connectors/source-facebook-marketing/source_facebook_marketing/client/api.py

Line 159 in 99f1448

if self._state and self._state >= cursor:

Should be easy fix - implement custom read for InsightStream.
2h - implementation
2h - testing

Total: 4hours

[MaxKrog]

Great stuff! Im still manually reseting & rerunning this connection every morning. Can’t wait for a fix to be released.

[sherifnada]

@MaxKrog we should have this released very soon -- it's in the current sprint!

[vitaliizazmic]

@MaxKrog

We just merged this bug into master and released a new version of the connector.

Upgrade your connector to version 0.2.4 and get started. To upgrade your connector version, go to the admin panel in the left hand side of the UI, find this connector in the list, and input the latest connector version.

Please let us know if you have any further questions.

Enjoy!

[MaxKrog]

This is great news! Thanks for all the hard work.