Source Instagram: fix bug with transient failures #2666
Conversation
…ures, added clear pagination for media resources to be sure that the backoff will always take place.
yevhenii-ldv
requested review from
keu and
vitaliizazmic
and removed request for
sherifnada and
davinchia
| return getattr(ig_user, attribute)(params=params, fields=fields) | ||
|
|
||
| def pagination(self, instance: Cursor) -> Iterator[Any]: | ||
| yield from instance._queue |
There was a problem hiding this comment.
It looks like we can't decorate pagination with backoff without accessing private attributes of the Cursor.
Let's document this approach via comment and insert TODO warning about possible breaking changes in Facebook API implementation.
We can consider creating an issue for facebook_business client.
| return instagram_user.get_media(params=params, fields=fields) | ||
| media = self.get_instance_cursor(instagram_user, "get_media", params=params, fields=fields) | ||
| for media_obj in self.pagination(media): | ||
| yield media_obj |
There was a problem hiding this comment.
_get_media and _get_stories looks identical except "get_*" attribute name string.
We can consider moving get_instance_cursor into pagination and extend pagination parameters, e. g.:
for media_obj in self.pagination("get_media"):
yield media_obj
Or we can consider having wrapper function for the 3 lines above.
| return instagram_user.get_stories(params=params, fields=fields) | ||
| stories = self.get_instance_cursor(instagram_user, "get_stories", params=params, fields=fields) | ||
| for story_obj in self.pagination(stories): | ||
| yield story_obj |
There was a problem hiding this comment.
Please, see comment regarding _get_media.
| ): | ||
| return True | ||
| return False | ||
| return exc.api_transient_error() |
There was a problem hiding this comment.
are you sure that this also applies to rate limiting? I don't see it in docs anywhere. Why not just add this as an OR clause?
There was a problem hiding this comment.
I found information that this value also applies to Rate Limiting, but now I could not find it.
To be more safely, I added this check to the general list of errors to be handled using backoff.
…/source-instagram-retry-transient-server-failures
|
/test connector=source-instagram
|
|
/publish connector=connectors/source-instagram
|
|
/test connector=source-instagram
|
yevhenii-ldv
deleted the
ykurochkin/source-instagram-retry-transient-server-failures
branch
What
Pre-merge Checklist
Recommended reading order
test.javacomponent.ts