Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature/improve schema logging #27

Conversation

JohannesRudolph
Copy link

I was running into some issues where taps did not provide good schemas, which unfortunately lead to exceptions in target-bigquery. These were hard to debug at times, because it was difficult to pinpoint what exactly went wrong. Adding the following log statements helped my debugging of these problems a lot.

Note for reviewers:

  • Python is not my home turf, I used f-string for formatting log statements, not sure if that's the right way to go
  • I chose info log level for both statements. 1) The JSON schema is logged at info level as well, so logging the BQ schema at info felt right. 2) failing to format the record crashes the target always, so logging it at info level felt right, so that users would not have to re-run it with debug level

@RuslanBergenov RuslanBergenov changed the base branch from master to feature/incremental_sync_and_schema_logging January 11, 2022 23:20
@RuslanBergenov RuslanBergenov merged commit f2a6d0b into adswerve:feature/incremental_sync_and_schema_logging Jan 11, 2022
@JohannesRudolph JohannesRudolph deleted the feature/improve-schema-logging branch January 12, 2022 11:22
@RuslanBergenov
Copy link
Collaborator

@JohannesRudolph , do you mind sharing a schema which caused the format_record_to_schema function to break? We may use it in our testing.

@JohannesRudolph
Copy link
Author

@RuslanBergenov sure you can find one in a log output at AutoIDM/tap-clickup#100 (comment) which links back to this issue here :-)

@RuslanBergenov
Copy link
Collaborator

@JohannesRudolph, sorry for the delay, we released your feature, thank you for the contribution.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants