DatetimeBasedCursor should support incremental syncs without steps

[maxi297]

What area the feature impact?

Connectors

Revelant Information

Tell us about the problem you're trying to solve

Today when configuring an incremental sync, the concept of step and granularity is very confusing for a couple of reasons:

• It is implementation details and the user doesn't really understand why
• The possible values are complicated

It is possible to create a sync that is incremental without providing those. Therefore, those values should be optional.

Describe the solution you’d like

Bundle step and granularity together and make them optional. This means that this YAML component is valid:

    incremental_sync:
      type: "DatetimeBasedCursor"
      start_datetime:
        datetime: "{{ config['start_date'] }}"
        datetime_format: "%Y-%m"
      datetime_format: "%Y-%m-%dT%H:%M:%S%z"
      cursor_field: "pub_date"

which would lead to only one slice.

However, we can define it like this:

    incremental_sync:
      type: "DatetimeBasedCursor"
      start_datetime:
        datetime: "{{ config['start_date'] }}"
        datetime_format: "%Y-%m"
      step: "P1M"
      cursor_granularity: "PT1S"
      datetime_format: "%Y-%m-%dT%H:%M:%S%z"
      cursor_field: "pub_date"

... which will create multiple slices. We can set the step totimedelta.max within the DatetimeBasedCursor or something when the step is not provided. It should stop iterating at end_datetime so it should be fine.

Acceptance Criteria

• Implement the proposed solution in the CDK
  • Throw and error when only one between step and cursor_granularity is defined
• Deprecated step and cursor_granularity
• Update Connector Builder to reflect that. Right now, step is marked as “advanced” but not the cursor granularity and therefore those concepts are far away even though they are linked. It should be clear that both go together. We should as well set them both behind a checkbox under “Advanced”
• Update the documentation (docs folder and declarative_component_schema.yaml)

[lmossman]

A connector builder user has run into an issue that I believe is related to Step. Pasting my email response here for context:

I have a theory as to what is going on here, though it is hard for me to test it without access to your account, so I'd like you to try out a change to see if it fixes the problem.

What I think may be happening here is that because your connector's Incremental Sync section is only configured to pass a "start time" to the API and not any "end time", our connector framework is not able to break up the entire set of results into separate timespan chunks, because it can only provide a start time to the API.

The problem with this is that the "Step" configuration (in the "Advanced" part of the Incremental Sync section) defaults to P1M (i.e. 1 month), which means the connector is trying to break up the requests into 1-month timeframe chunks to have better checkpointing in the case of failed syncs. But since it cannot provide an end time to the API, it ends up re-requesting the entire set of records, but with start times that are increased by 1 month at a time. You can read more about Step in our docs here if you are interested.

TL;DR: Because end times are not being passed to the API, the Step configuration is causing this to re-request data that has already been requested.

To fix this, please try setting the Step value to a very large timeframe, such as P1000Y (i.e. 1000 years), which will mean that the low-code framework will not try to break up the timeframe into chunks and it should work as expected. Requests will still be paginated of course.

Another way to fix this would be to configure your connector to also pass an end date to the API, but this is likely more complicated to do.

[maxi297]

Grooming notes:

• Make them optional and once we have a way to migrate these, migrate them to the new format. In the meanwhile, UI would need to support both

At first, the proposition was to have a steps subobject like this:

    incremental_sync:
      type: "DatetimeBasedCursor"
      start_datetime:
        datetime: "{{ config['start_date'] }}"
        datetime_format: "%Y-%m"
      steps:
        duration: "P1M"
        cursor_granularity: "PT1S"
      datetime_format: "%Y-%m-%dT%H:%M:%S%z"
      cursor_field: "pub_date"

However, this would implies a migration. Therefore, we will keep the same schema for now