🎉 New Source: Monday #7168
🎉 New Source: Monday #7168
Conversation
|
Hey @tuanchris thanks for the contribution. We will review it soon. Can you share the credentials over slack (DM) for me test it out. |
|
Please share unit test and integration tests screen shoot showing they are passing locally. |
|
@marcosmarxm Here are the tests: Integration test: I tried looking at the two output file: |
|
The issue was due to an uncaught retry problem. Fixed that and the local integration test is passing now.
|
…marcos/test-pr-7168
| def should_retry(self, response: requests.Response) -> bool: | ||
| # Monday API return code 200 with and errors key if complexity is too high. | ||
| # https://api.developer.monday.com/docs/complexity-queries | ||
| is_complex_query = response.json().get("errors") | ||
| return response.status_code == 429 or 500 <= response.status_code < 600 or is_complex_query |
There was a problem hiding this comment.
This will request the same query? If its a complex query this means will keep in a loop?
There was a problem hiding this comment.
I ran integration tests and got backoff error, I need to add a logging to understand the error:
2021-10-26 01:10:13 INFO () DefaultAirbyteStreamFactory(lambda$create$0):53 - [{'message': "Field 'users' doesn't accept argument 'page'", 'locations': [{'line': 1, 'column': 16}], 'fields': ['query', 'users', 'page']}]
2021-10-26 01:10:13 INFO () DefaultAirbyteStreamFactory(internalLog):86 - Backing off _send(...) for 15.0s (airbyte_cdk.sources.streams.http.exceptions.DefaultBackoffException)
2021-10-26 01:10:29 INFO () DefaultAirbyteStreamFactory(internalLog):86 - Caught retryable error '' after 1 tries. Waiting 15 seconds then retrying...
There was a problem hiding this comment.
According to this document, the free trial account (which we are on) can only call queries with 1 mil or less per minute. In case of integration test, I think we reached the limit and this is the response.txt from the request:
{"errors":[{"message":"Complexity budget exhausted, query cost 202535 budget remaining 189860 out of 1000000 reset in 43 seconds"}],"account_id":10204492}
The code above is meant to check if status_code == 200 and there is an errors key in the response.json()
There was a problem hiding this comment.
Got it, the question is: if a query can stay in Complex Query and always keep in backoff.
There was a problem hiding this comment.
I don't think this is possible. From the same link: Complexity levels are calculated based on the query.. If a query return more than the complexity limit, we couldn't do the initial sync. The limit was hit because, with the integration test, we are running many successive queries in a short period of time. The retry and backoff will handle that.
…urce.py Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>
* add items stream * add boards stream * add remaining streams * check point * lint * fix unit test & linting * Delete launch.json * fix schema * add retry logic * Update airbyte-integrations/connectors/source-monday/source_monday/source.py Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com> * small fix + add creds * bump version and config file Co-authored-by: Marcos Marx <marcosmarxm@gmail.com> Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>
What
How
Describe the solution
Recommended reading order
x.javay.pythonPre-merge Checklist
Expand the relevant checklist and delete the others.
New Connector
Community member or Airbyter
airbyte_secret./gradlew :airbyte-integrations:connectors:<name>:integrationTest.README.mdbootstrap.md. See description and examplesdocs/SUMMARY.mddocs/integrations/<source or destination>/<name>.mdincluding changelog. See changelog exampledocs/integrations/README.mdairbyte-integrations/builds.mdAirbyter
If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.
/test connector=connectors/<name>command is passing./publishcommand described hereUpdating a connector
Community member or Airbyter
airbyte_secret./gradlew :airbyte-integrations:connectors:<name>:integrationTest.README.mdbootstrap.md. See description and examplesdocs/integrations/<source or destination>/<name>.mdincluding changelog. See changelog exampleAirbyter
If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.
/test connector=connectors/<name>command is passing./publishcommand described hereConnector Generator
-scaffoldin their name) have been updated with the latest scaffold by running./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplatesthen checking in your changes