-
Notifications
You must be signed in to change notification settings - Fork 3.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Source Zuora: fix decimals in stringified datetimes #24460
Conversation
/test connector=connectors/source-zuora
Build FailedTest summary info:
|
The tests came up with this error:
@frans-k before I remove |
airbyte-integrations/connectors/source-zuora/unit_tests/unit_test.py
Outdated
Show resolved
Hide resolved
/test connector=connectors/source-zuora
Build FailedTest summary info:
|
It looks like the Zuora integration test credentials need to be refreshed. I've submitted a request to get it investigated. |
Just let me know if there's anything I can do, @sh4sh. |
The "TypeError: 'NoneType' object does not support item assignment" error is caused by a bug in zuora_auth.py not an authentication failure (code is trying to edit a null object to remove a field, instead of building the body and removing the items). The following version of zuora_auth.py should fix the error:
|
Good call @jetvp, fixed the OAuth issue. Now I'm getting a new issue though. Zuora gets spammed with hundreds of Posting a short excerpt from the logs, I can't see anything that stands out however.
Those last messages are repeated, unclear if they're related to the issue. |
I came over this issue the other day. The ZuoraObjectsBase > def get_json_schema(self) should have a cache above it:
You'll also need to import:
That means it will only call describe once and use that result for the other rows (instead of firing of an API call for every row!). I'm currently fixing it to make sure it does not return more than the 50k rows, and itinerates from the last date (but can have that in a separate patch). |
@frans-k , let me know if that works for you. |
@jetvp Yes, that part seems to work now, which of course means I'm running into new issues. Unrelated to this PR, but just to share the first new issue:
Fails on Even a table that don't have this issue is failing on something as well:
Don't have the time today to try and debug this any further. |
Sorry the long delay to get this reviewed and merged. Our team is finishing the August Hackathon review contributions in next two weeks and after we're going to return to the community backlog. |
@frans-k circling back to this PR and hoping to get it merged soon. Since it's been a while, I wanted to confirm - are you still interested in continuing this work? Thanks in advance! |
Closed because user didn't return last comment. @frans-k let us know if you want to continue the work here. |
What
It outputs timestamps in a format that Zuora can cast to TIMESTAMP.
Today Zuora accepts 6 decimals for fractions of a second when casting to TIMESTAMP, but if that's in a WHERE filter on for example
updateddate
, that filter is ignored for input rows.So if the table has > 10m rows, it hits the Data Query 10m input row limit. Haven't tested if it affects output rows.
How
Changed from 6 decimals to 3 for second fractions. Instead of
strftime
it uses Pendulum'sformat
.Recommended reading order
x.java
y.python
🚨 User Impact 🚨
Are there any breaking changes? What is the end result perceived by the user? If yes, please merge this PR with the 🚨🚨 emoji so changelog authors can further highlight this if needed.
Pre-merge Checklist
Expand the relevant checklist and delete the others.
New Connector
Community member or Airbyter
airbyte_secret
./gradlew :airbyte-integrations:connectors:<name>:integrationTest
.README.md
bootstrap.md
. See description and examplesdocs/integrations/<source or destination>/<name>.md
including changelog. See changelog exampledocs/integrations/README.md
airbyte-integrations/builds.md
Airbyter
If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.
/test connector=connectors/<name>
command is passing/publish
command described hereUpdating a connector
Community member or Airbyter
airbyte_secret
./gradlew :airbyte-integrations:connectors:<name>:integrationTest
.README.md
bootstrap.md
. See description and examplesdocs/integrations/<source or destination>/<name>.md
including changelog. See changelog exampleAirbyter
If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.
/test connector=connectors/<name>
command is passing/publish
command described hereConnector Generator
-scaffold
in their name) have been updated with the latest scaffold by running./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates
then checking in your changes