connector builder: Set state on stream slices #37109
Conversation
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎ 1 Ignored Deployment
|
| elif message.type == MessageType.STATE: | ||
| latest_state_message = message.state |
There was a problem hiding this comment.
@girarda my understanding of this feature was that every slice would output its own state message, so we would expect every item in the slices response of the read_stream request to contain a state object.
But when I tested this, only the last slice had a state in the response of read_stream.
Is this expected that only the last slice will ever output a state message? If so, why does this happen instead of each slice outputting a state?
There was a problem hiding this comment.
how did you test this? My initial hypothesis is this might be because we don't necessarily produce a stream slice due to record and page limits but I can dig more into the issue
There was a problem hiding this comment.
I used the exchange rates connector that you build in the tutorial:
exchange_rates.yaml
And I pointed my local connector builder server at the local CDK on this branch
There was a problem hiding this comment.
The issue is that the builder server submits a request for a full refresh read instead of incremental. Fix is here https://github.com/airbytehq/airbyte-platform-internal/pull/12179
There was a problem hiding this comment.
With the fix for the sync mode in https://github.com/airbytehq/airbyte-platform-internal/pull/12179, this is now working as expected for me when used with the Builder
What
Update the builder's message grouper to set the state on the output stream slices.
The BE server does not need to be updated, but we'll need to update the webapp to display them.
Solves a part of https://github.com/airbytehq/airbyte-internal-issues/issues/1561
How
Example response from the server:
