docs: adds bigtable sink documentation #58
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* feat: add bigtable sink with stringified odpf messages as values * feat: add BIGTABLE_CREDENTIAL_PATH to access google cloud * feat: add wrapper class on bigtable client * refactor: fix checkstyle * feat: add bigtable parser tests * feat: add bigtable sink tests * feat: add bigtable client tests * chore: revert version bump * chore: revert version change in build.gradle Co-authored-by: mayur.gubrele <2310-mayur.gubrele@users.noreply.source.golabs.io> Co-authored-by: lavkesh <lavkesh51@gmail.com>
* feat: Bigtable Sink (#33) * feat: add bigtable sink with stringified odpf messages as values * feat: add BIGTABLE_CREDENTIAL_PATH to access google cloud * feat: add wrapper class on bigtable client * refactor: fix checkstyle * feat: add bigtable parser tests * feat: add bigtable sink tests * feat: add bigtable client tests * chore: revert version bump * chore: revert version change in build.gradle Co-authored-by: mayur.gubrele <2310-mayur.gubrele@users.noreply.source.golabs.io> Co-authored-by: lavkesh <lavkesh51@gmail.com> * feat: create bigtable records using InputOutputFieldMapping provided as configuration * refactor: fix checkstyle and add unit tests * review: minor refactor * refactor: add BigTableSchemaTest and fix BigTableRecordParserTest * refactor: fix checkstyle * tests: add few more tests Co-authored-by: mayur.gubrele <2310-mayur.gubrele@users.noreply.source.golabs.io> Co-authored-by: lavkesh <lavkesh51@gmail.com>
MayurGubrele
force-pushed
the
bigtable-documentation
branch
from
f0000b9 to
3ddd809
Compare
lavkesh
added a commit
that referenced
this pull request
Nov 29, 2022
* feat: Bigtable Sink (#33) * feat: add bigtable sink with stringified odpf messages as values * feat: add BIGTABLE_CREDENTIAL_PATH to access google cloud * feat: add wrapper class on bigtable client * refactor: fix checkstyle * feat: add bigtable parser tests * feat: add bigtable sink tests * feat: add bigtable client tests * chore: revert version bump * chore: revert version change in build.gradle Co-authored-by: mayur.gubrele <2310-mayur.gubrele@users.noreply.source.golabs.io> Co-authored-by: lavkesh <lavkesh51@gmail.com> * chore: qa and review fixes * chore: change default SCHEMA_REGISTRY_STENCIL_CACHE_AUTO_REFRESH to true * feat: Bigtable record parser (#39) * feat: Bigtable Sink (#33) * feat: add bigtable sink with stringified odpf messages as values * feat: add BIGTABLE_CREDENTIAL_PATH to access google cloud * feat: add wrapper class on bigtable client * refactor: fix checkstyle * feat: add bigtable parser tests * feat: add bigtable sink tests * feat: add bigtable client tests * chore: revert version bump * chore: revert version change in build.gradle Co-authored-by: mayur.gubrele <2310-mayur.gubrele@users.noreply.source.golabs.io> Co-authored-by: lavkesh <lavkesh51@gmail.com> * feat: create bigtable records using InputOutputFieldMapping provided as configuration * refactor: fix checkstyle and add unit tests * review: minor refactor * refactor: add BigTableSchemaTest and fix BigTableRecordParserTest * refactor: fix checkstyle * tests: add few more tests Co-authored-by: mayur.gubrele <2310-mayur.gubrele@users.noreply.source.golabs.io> Co-authored-by: lavkesh <lavkesh51@gmail.com> * chore: schema refactoring * chore: naming conventions * feat: add functionality to create rowkey from configured template (#44) * chore: refactor Template validation * chore: change exception message in Template * feat: add bigtable sink metrics and logging (#51) * feat: Bigtable Sink (#33) * feat: add bigtable sink with stringified odpf messages as values * feat: add BIGTABLE_CREDENTIAL_PATH to access google cloud * feat: add wrapper class on bigtable client * refactor: fix checkstyle * feat: add bigtable parser tests * feat: add bigtable sink tests * feat: add bigtable client tests * chore: revert version bump * chore: revert version change in build.gradle Co-authored-by: mayur.gubrele <2310-mayur.gubrele@users.noreply.source.golabs.io> Co-authored-by: lavkesh <lavkesh51@gmail.com> * chore: qa and review fixes * chore: change default SCHEMA_REGISTRY_STENCIL_CACHE_AUTO_REFRESH to true * feat: Bigtable record parser (#39) * feat: Bigtable Sink (#33) * feat: add bigtable sink with stringified odpf messages as values * feat: add BIGTABLE_CREDENTIAL_PATH to access google cloud * feat: add wrapper class on bigtable client * refactor: fix checkstyle * feat: add bigtable parser tests * feat: add bigtable sink tests * feat: add bigtable client tests * chore: revert version bump * chore: revert version change in build.gradle Co-authored-by: mayur.gubrele <2310-mayur.gubrele@users.noreply.source.golabs.io> Co-authored-by: lavkesh <lavkesh51@gmail.com> * feat: create bigtable records using InputOutputFieldMapping provided as configuration * refactor: fix checkstyle and add unit tests * review: minor refactor * refactor: add BigTableSchemaTest and fix BigTableRecordParserTest * refactor: fix checkstyle * tests: add few more tests Co-authored-by: mayur.gubrele <2310-mayur.gubrele@users.noreply.source.golabs.io> Co-authored-by: lavkesh <lavkesh51@gmail.com> * chore: schema refactoring * chore: naming conventions * feat: add functionality to create rowkey from configured template (#44) * chore: refactor Template validation * chore: change exception message in Template * feat: add bigtable sink metrics and logging (#51) * feat: Bigtable error parsing (#55) * feat: Bigtable Sink (#33) * feat: add bigtable sink with stringified odpf messages as values * feat: add BIGTABLE_CREDENTIAL_PATH to access google cloud * feat: add wrapper class on bigtable client * refactor: fix checkstyle * feat: add bigtable parser tests * feat: add bigtable sink tests * feat: add bigtable client tests * chore: revert version bump * chore: revert version change in build.gradle Co-authored-by: mayur.gubrele <2310-mayur.gubrele@users.noreply.source.golabs.io> Co-authored-by: lavkesh <lavkesh51@gmail.com> * chore: qa and review fixes * chore: change default SCHEMA_REGISTRY_STENCIL_CACHE_AUTO_REFRESH to true * feat: Bigtable record parser (#39) * feat: Bigtable Sink (#33) * feat: add bigtable sink with stringified odpf messages as values * feat: add BIGTABLE_CREDENTIAL_PATH to access google cloud * feat: add wrapper class on bigtable client * refactor: fix checkstyle * feat: add bigtable parser tests * feat: add bigtable sink tests * feat: add bigtable client tests * chore: revert version bump * chore: revert version change in build.gradle Co-authored-by: mayur.gubrele <2310-mayur.gubrele@users.noreply.source.golabs.io> Co-authored-by: lavkesh <lavkesh51@gmail.com> * feat: create bigtable records using InputOutputFieldMapping provided as configuration * refactor: fix checkstyle and add unit tests * review: minor refactor * refactor: add BigTableSchemaTest and fix BigTableRecordParserTest * refactor: fix checkstyle * tests: add few more tests Co-authored-by: mayur.gubrele <2310-mayur.gubrele@users.noreply.source.golabs.io> Co-authored-by: lavkesh <lavkesh51@gmail.com> * chore: schema refactoring * chore: naming conventions * feat: add functionality to create rowkey from configured template (#44) * chore: refactor Template validation * chore: change exception message in Template * feat: add bigtable sink metrics and logging (#51) * feat: parse bigtable errors and create odpf response * feat: capture error metrics with predefined tags * chore: add tests for bigtable response parser * chore: remove deprecated jcenter repo from build.gradle * chore: comment out classpath dependencies * chore: remove unused classpath dependencies * chore: minor refactor * chore: remove unused classpath dependencies * refactor: change BigtableRecord,Response contracts, improved logging, added tests Co-authored-by: mayur.gubrele <2310-mayur.gubrele@users.noreply.source.golabs.io> Co-authored-by: lavkesh <lavkesh51@gmail.com> * chore: minor fixes (#59) * feat: Bigtable Sink (#33) * feat: add bigtable sink with stringified odpf messages as values * feat: add BIGTABLE_CREDENTIAL_PATH to access google cloud * feat: add wrapper class on bigtable client * refactor: fix checkstyle * feat: add bigtable parser tests * feat: add bigtable sink tests * feat: add bigtable client tests * chore: revert version bump * chore: revert version change in build.gradle Co-authored-by: mayur.gubrele <2310-mayur.gubrele@users.noreply.source.golabs.io> Co-authored-by: lavkesh <lavkesh51@gmail.com> * chore: qa and review fixes * chore: change default SCHEMA_REGISTRY_STENCIL_CACHE_AUTO_REFRESH to true * feat: Bigtable record parser (#39) * feat: Bigtable Sink (#33) * feat: add bigtable sink with stringified odpf messages as values * feat: add BIGTABLE_CREDENTIAL_PATH to access google cloud * feat: add wrapper class on bigtable client * refactor: fix checkstyle * feat: add bigtable parser tests * feat: add bigtable sink tests * feat: add bigtable client tests * chore: revert version bump * chore: revert version change in build.gradle Co-authored-by: mayur.gubrele <2310-mayur.gubrele@users.noreply.source.golabs.io> Co-authored-by: lavkesh <lavkesh51@gmail.com> * feat: create bigtable records using InputOutputFieldMapping provided as configuration * refactor: fix checkstyle and add unit tests * review: minor refactor * refactor: add BigTableSchemaTest and fix BigTableRecordParserTest * refactor: fix checkstyle * tests: add few more tests Co-authored-by: mayur.gubrele <2310-mayur.gubrele@users.noreply.source.golabs.io> Co-authored-by: lavkesh <lavkesh51@gmail.com> * chore: schema refactoring * chore: naming conventions * feat: add functionality to create rowkey from configured template (#44) * chore: refactor Template validation * chore: change exception message in Template * feat: add bigtable sink metrics and logging (#51) * feat: parse bigtable errors and create odpf response * feat: capture error metrics with predefined tags * chore: add tests for bigtable response parser * chore: remove deprecated jcenter repo from build.gradle * chore: add bq error logs (#57) * chore: comment out classpath dependencies * chore: remove unused classpath dependencies * chore: minor refactor * chore: remove unused classpath dependencies * refactor: change BigtableRecord,Response contracts, improved logging, added tests * chore: small fixes Co-authored-by: Mayur Gubrele <mayur.gubrele@gmail.com> Co-authored-by: mayur.gubrele <2310-mayur.gubrele@users.noreply.source.golabs.io> * docs: adds bigtable sink documentation (#58) * feat: Bigtable Sink (#33) * feat: add bigtable sink with stringified odpf messages as values * feat: add BIGTABLE_CREDENTIAL_PATH to access google cloud * feat: add wrapper class on bigtable client * refactor: fix checkstyle * feat: add bigtable parser tests * feat: add bigtable sink tests * feat: add bigtable client tests * chore: revert version bump * chore: revert version change in build.gradle Co-authored-by: mayur.gubrele <2310-mayur.gubrele@users.noreply.source.golabs.io> Co-authored-by: lavkesh <lavkesh51@gmail.com> * chore: qa and review fixes * chore: change default SCHEMA_REGISTRY_STENCIL_CACHE_AUTO_REFRESH to true * feat: Bigtable record parser (#39) * feat: Bigtable Sink (#33) * feat: add bigtable sink with stringified odpf messages as values * feat: add BIGTABLE_CREDENTIAL_PATH to access google cloud * feat: add wrapper class on bigtable client * refactor: fix checkstyle * feat: add bigtable parser tests * feat: add bigtable sink tests * feat: add bigtable client tests * chore: revert version bump * chore: revert version change in build.gradle Co-authored-by: mayur.gubrele <2310-mayur.gubrele@users.noreply.source.golabs.io> Co-authored-by: lavkesh <lavkesh51@gmail.com> * feat: create bigtable records using InputOutputFieldMapping provided as configuration * refactor: fix checkstyle and add unit tests * review: minor refactor * refactor: add BigTableSchemaTest and fix BigTableRecordParserTest * refactor: fix checkstyle * tests: add few more tests Co-authored-by: mayur.gubrele <2310-mayur.gubrele@users.noreply.source.golabs.io> Co-authored-by: lavkesh <lavkesh51@gmail.com> * chore: schema refactoring * chore: naming conventions * feat: add functionality to create rowkey from configured template (#44) * chore: refactor Template validation * chore: change exception message in Template * feat: add bigtable sink metrics and logging (#51) * docs: add bigtable sink documentation Co-authored-by: mayur.gubrele <2310-mayur.gubrele@users.noreply.source.golabs.io> Co-authored-by: lavkesh <lavkesh51@gmail.com> * chore: rc version * feat: exclude io.grpc from bigtable library and bump the version to 0.3.4-beta.4 * chore: bump up depot version to 0.3.4 * refactor: fix template test and remove whitespace Co-authored-by: mayur.gubrele <2310-mayur.gubrele@users.noreply.source.golabs.io> Co-authored-by: lavkesh <lavkesh51@gmail.com>
lavkesh
added a commit
that referenced
this pull request
Dec 5, 2022
* feat: Bigtable Sink (#33) * feat: add bigtable sink with stringified odpf messages as values * feat: add BIGTABLE_CREDENTIAL_PATH to access google cloud * feat: add wrapper class on bigtable client * refactor: fix checkstyle * feat: add bigtable parser tests * feat: add bigtable sink tests * feat: add bigtable client tests * chore: revert version bump * chore: revert version change in build.gradle Co-authored-by: mayur.gubrele <2310-mayur.gubrele@users.noreply.source.golabs.io> Co-authored-by: lavkesh <lavkesh51@gmail.com> * chore: qa and review fixes * chore: change default SCHEMA_REGISTRY_STENCIL_CACHE_AUTO_REFRESH to true * feat: Bigtable record parser (#39) * feat: Bigtable Sink (#33) * feat: add bigtable sink with stringified odpf messages as values * feat: add BIGTABLE_CREDENTIAL_PATH to access google cloud * feat: add wrapper class on bigtable client * refactor: fix checkstyle * feat: add bigtable parser tests * feat: add bigtable sink tests * feat: add bigtable client tests * chore: revert version bump * chore: revert version change in build.gradle Co-authored-by: mayur.gubrele <2310-mayur.gubrele@users.noreply.source.golabs.io> Co-authored-by: lavkesh <lavkesh51@gmail.com> * feat: create bigtable records using InputOutputFieldMapping provided as configuration * refactor: fix checkstyle and add unit tests * review: minor refactor * refactor: add BigTableSchemaTest and fix BigTableRecordParserTest * refactor: fix checkstyle * tests: add few more tests Co-authored-by: mayur.gubrele <2310-mayur.gubrele@users.noreply.source.golabs.io> Co-authored-by: lavkesh <lavkesh51@gmail.com> * chore: schema refactoring * chore: naming conventions * feat: add functionality to create rowkey from configured template (#44) * chore: refactor Template validation * chore: change exception message in Template * feat: add bigtable sink metrics and logging (#51) * feat: Bigtable Sink (#33) * feat: add bigtable sink with stringified odpf messages as values * feat: add BIGTABLE_CREDENTIAL_PATH to access google cloud * feat: add wrapper class on bigtable client * refactor: fix checkstyle * feat: add bigtable parser tests * feat: add bigtable sink tests * feat: add bigtable client tests * chore: revert version bump * chore: revert version change in build.gradle Co-authored-by: mayur.gubrele <2310-mayur.gubrele@users.noreply.source.golabs.io> Co-authored-by: lavkesh <lavkesh51@gmail.com> * chore: qa and review fixes * chore: change default SCHEMA_REGISTRY_STENCIL_CACHE_AUTO_REFRESH to true * feat: Bigtable record parser (#39) * feat: Bigtable Sink (#33) * feat: add bigtable sink with stringified odpf messages as values * feat: add BIGTABLE_CREDENTIAL_PATH to access google cloud * feat: add wrapper class on bigtable client * refactor: fix checkstyle * feat: add bigtable parser tests * feat: add bigtable sink tests * feat: add bigtable client tests * chore: revert version bump * chore: revert version change in build.gradle Co-authored-by: mayur.gubrele <2310-mayur.gubrele@users.noreply.source.golabs.io> Co-authored-by: lavkesh <lavkesh51@gmail.com> * feat: create bigtable records using InputOutputFieldMapping provided as configuration * refactor: fix checkstyle and add unit tests * review: minor refactor * refactor: add BigTableSchemaTest and fix BigTableRecordParserTest * refactor: fix checkstyle * tests: add few more tests Co-authored-by: mayur.gubrele <2310-mayur.gubrele@users.noreply.source.golabs.io> Co-authored-by: lavkesh <lavkesh51@gmail.com> * chore: schema refactoring * chore: naming conventions * feat: add functionality to create rowkey from configured template (#44) * chore: refactor Template validation * chore: change exception message in Template * feat: add bigtable sink metrics and logging (#51) * feat: Bigtable error parsing (#55) * feat: Bigtable Sink (#33) * feat: add bigtable sink with stringified odpf messages as values * feat: add BIGTABLE_CREDENTIAL_PATH to access google cloud * feat: add wrapper class on bigtable client * refactor: fix checkstyle * feat: add bigtable parser tests * feat: add bigtable sink tests * feat: add bigtable client tests * chore: revert version bump * chore: revert version change in build.gradle Co-authored-by: mayur.gubrele <2310-mayur.gubrele@users.noreply.source.golabs.io> Co-authored-by: lavkesh <lavkesh51@gmail.com> * chore: qa and review fixes * chore: change default SCHEMA_REGISTRY_STENCIL_CACHE_AUTO_REFRESH to true * feat: Bigtable record parser (#39) * feat: Bigtable Sink (#33) * feat: add bigtable sink with stringified odpf messages as values * feat: add BIGTABLE_CREDENTIAL_PATH to access google cloud * feat: add wrapper class on bigtable client * refactor: fix checkstyle * feat: add bigtable parser tests * feat: add bigtable sink tests * feat: add bigtable client tests * chore: revert version bump * chore: revert version change in build.gradle Co-authored-by: mayur.gubrele <2310-mayur.gubrele@users.noreply.source.golabs.io> Co-authored-by: lavkesh <lavkesh51@gmail.com> * feat: create bigtable records using InputOutputFieldMapping provided as configuration * refactor: fix checkstyle and add unit tests * review: minor refactor * refactor: add BigTableSchemaTest and fix BigTableRecordParserTest * refactor: fix checkstyle * tests: add few more tests Co-authored-by: mayur.gubrele <2310-mayur.gubrele@users.noreply.source.golabs.io> Co-authored-by: lavkesh <lavkesh51@gmail.com> * chore: schema refactoring * chore: naming conventions * feat: add functionality to create rowkey from configured template (#44) * chore: refactor Template validation * chore: change exception message in Template * feat: add bigtable sink metrics and logging (#51) * feat: parse bigtable errors and create odpf response * feat: capture error metrics with predefined tags * chore: add tests for bigtable response parser * chore: remove deprecated jcenter repo from build.gradle * chore: comment out classpath dependencies * chore: remove unused classpath dependencies * chore: minor refactor * chore: remove unused classpath dependencies * refactor: change BigtableRecord,Response contracts, improved logging, added tests Co-authored-by: mayur.gubrele <2310-mayur.gubrele@users.noreply.source.golabs.io> Co-authored-by: lavkesh <lavkesh51@gmail.com> * chore: minor fixes (#59) * feat: Bigtable Sink (#33) * feat: add bigtable sink with stringified odpf messages as values * feat: add BIGTABLE_CREDENTIAL_PATH to access google cloud * feat: add wrapper class on bigtable client * refactor: fix checkstyle * feat: add bigtable parser tests * feat: add bigtable sink tests * feat: add bigtable client tests * chore: revert version bump * chore: revert version change in build.gradle Co-authored-by: mayur.gubrele <2310-mayur.gubrele@users.noreply.source.golabs.io> Co-authored-by: lavkesh <lavkesh51@gmail.com> * chore: qa and review fixes * chore: change default SCHEMA_REGISTRY_STENCIL_CACHE_AUTO_REFRESH to true * feat: Bigtable record parser (#39) * feat: Bigtable Sink (#33) * feat: add bigtable sink with stringified odpf messages as values * feat: add BIGTABLE_CREDENTIAL_PATH to access google cloud * feat: add wrapper class on bigtable client * refactor: fix checkstyle * feat: add bigtable parser tests * feat: add bigtable sink tests * feat: add bigtable client tests * chore: revert version bump * chore: revert version change in build.gradle Co-authored-by: mayur.gubrele <2310-mayur.gubrele@users.noreply.source.golabs.io> Co-authored-by: lavkesh <lavkesh51@gmail.com> * feat: create bigtable records using InputOutputFieldMapping provided as configuration * refactor: fix checkstyle and add unit tests * review: minor refactor * refactor: add BigTableSchemaTest and fix BigTableRecordParserTest * refactor: fix checkstyle * tests: add few more tests Co-authored-by: mayur.gubrele <2310-mayur.gubrele@users.noreply.source.golabs.io> Co-authored-by: lavkesh <lavkesh51@gmail.com> * chore: schema refactoring * chore: naming conventions * feat: add functionality to create rowkey from configured template (#44) * chore: refactor Template validation * chore: change exception message in Template * feat: add bigtable sink metrics and logging (#51) * feat: parse bigtable errors and create odpf response * feat: capture error metrics with predefined tags * chore: add tests for bigtable response parser * chore: remove deprecated jcenter repo from build.gradle * chore: add bq error logs (#57) * chore: comment out classpath dependencies * chore: remove unused classpath dependencies * chore: minor refactor * chore: remove unused classpath dependencies * refactor: change BigtableRecord,Response contracts, improved logging, added tests * chore: small fixes Co-authored-by: Mayur Gubrele <mayur.gubrele@gmail.com> Co-authored-by: mayur.gubrele <2310-mayur.gubrele@users.noreply.source.golabs.io> * docs: adds bigtable sink documentation (#58) * feat: Bigtable Sink (#33) * feat: add bigtable sink with stringified odpf messages as values * feat: add BIGTABLE_CREDENTIAL_PATH to access google cloud * feat: add wrapper class on bigtable client * refactor: fix checkstyle * feat: add bigtable parser tests * feat: add bigtable sink tests * feat: add bigtable client tests * chore: revert version bump * chore: revert version change in build.gradle Co-authored-by: mayur.gubrele <2310-mayur.gubrele@users.noreply.source.golabs.io> Co-authored-by: lavkesh <lavkesh51@gmail.com> * chore: qa and review fixes * chore: change default SCHEMA_REGISTRY_STENCIL_CACHE_AUTO_REFRESH to true * feat: Bigtable record parser (#39) * feat: Bigtable Sink (#33) * feat: add bigtable sink with stringified odpf messages as values * feat: add BIGTABLE_CREDENTIAL_PATH to access google cloud * feat: add wrapper class on bigtable client * refactor: fix checkstyle * feat: add bigtable parser tests * feat: add bigtable sink tests * feat: add bigtable client tests * chore: revert version bump * chore: revert version change in build.gradle Co-authored-by: mayur.gubrele <2310-mayur.gubrele@users.noreply.source.golabs.io> Co-authored-by: lavkesh <lavkesh51@gmail.com> * feat: create bigtable records using InputOutputFieldMapping provided as configuration * refactor: fix checkstyle and add unit tests * review: minor refactor * refactor: add BigTableSchemaTest and fix BigTableRecordParserTest * refactor: fix checkstyle * tests: add few more tests Co-authored-by: mayur.gubrele <2310-mayur.gubrele@users.noreply.source.golabs.io> Co-authored-by: lavkesh <lavkesh51@gmail.com> * chore: schema refactoring * chore: naming conventions * feat: add functionality to create rowkey from configured template (#44) * chore: refactor Template validation * chore: change exception message in Template * feat: add bigtable sink metrics and logging (#51) * docs: add bigtable sink documentation Co-authored-by: mayur.gubrele <2310-mayur.gubrele@users.noreply.source.golabs.io> Co-authored-by: lavkesh <lavkesh51@gmail.com> * chore: rc version * fix: fix uri and exception handler * feat: exclude io.grpc from bigtable library and bump the version to 0.3.4-beta.4 * chore: add iterator in http record and toString in Template * chore: add more exception handler * Revert "chore: add more exception handler" This reverts commit 0edb3b7. * Revert "chore: add iterator in http record and toString in Template" This reverts commit a18bad6. * fix: fix template, uri and http exception handling * fix: fix exception handling on response and request body * fix: fix template exception * Revert "fix: fix template exception" This reverts commit 746754a. * chore: move IOException handling for request and response body Co-authored-by: Mayur Gubrele <mayur.gubrele@gmail.com> Co-authored-by: mayur.gubrele <2310-mayur.gubrele@users.noreply.source.golabs.io> Co-authored-by: lavkesh <lavkesh51@gmail.com>
sumitaich1998
referenced
this pull request
in goto/depot
Apr 24, 2024
* feat: create common http client (#53) * feat: create bare minimum http sink (#56) * feat: create bare minimum http sink * feat: add unit test * test: fix test * refactor: refactor request creation into factory class * refactor: remove comments * chore: minor fixes * chore: disable parameters in configs * chore: minor fixes * fix: fix metadata to also support generic value * chore: abstract metadata util * chore: http sink config extend to http client config * chore: refactor config class definition * chore: query param as map * chore: remove null check * chore: refactor * chore: minor refactoring * chore: create converter for header config Co-authored-by: lavkesh <lavkesh51@gmail.com> Co-authored-by: jesry.pandawa <jesry.pandawa@gojek.com> * fix: fix single request (#64) * feat: Bigtable Sink (#33) * feat: add bigtable sink with stringified odpf messages as values * feat: add BIGTABLE_CREDENTIAL_PATH to access google cloud * feat: add wrapper class on bigtable client * refactor: fix checkstyle * feat: add bigtable parser tests * feat: add bigtable sink tests * feat: add bigtable client tests * chore: revert version bump * chore: revert version change in build.gradle Co-authored-by: mayur.gubrele <2310-mayur.gubrele@users.noreply.source.golabs.io> Co-authored-by: lavkesh <lavkesh51@gmail.com> * chore: qa and review fixes * chore: change default SCHEMA_REGISTRY_STENCIL_CACHE_AUTO_REFRESH to true * feat: Bigtable record parser (#39) * feat: Bigtable Sink (#33) * feat: add bigtable sink with stringified odpf messages as values * feat: add BIGTABLE_CREDENTIAL_PATH to access google cloud * feat: add wrapper class on bigtable client * refactor: fix checkstyle * feat: add bigtable parser tests * feat: add bigtable sink tests * feat: add bigtable client tests * chore: revert version bump * chore: revert version change in build.gradle Co-authored-by: mayur.gubrele <2310-mayur.gubrele@users.noreply.source.golabs.io> Co-authored-by: lavkesh <lavkesh51@gmail.com> * feat: create bigtable records using InputOutputFieldMapping provided as configuration * refactor: fix checkstyle and add unit tests * review: minor refactor * refactor: add BigTableSchemaTest and fix BigTableRecordParserTest * refactor: fix checkstyle * tests: add few more tests Co-authored-by: mayur.gubrele <2310-mayur.gubrele@users.noreply.source.golabs.io> Co-authored-by: lavkesh <lavkesh51@gmail.com> * chore: schema refactoring * chore: naming conventions * feat: add functionality to create rowkey from configured template (#44) * chore: refactor Template validation * chore: change exception message in Template * feat: add bigtable sink metrics and logging (#51) * feat: Bigtable Sink (#33) * feat: add bigtable sink with stringified odpf messages as values * feat: add BIGTABLE_CREDENTIAL_PATH to access google cloud * feat: add wrapper class on bigtable client * refactor: fix checkstyle * feat: add bigtable parser tests * feat: add bigtable sink tests * feat: add bigtable client tests * chore: revert version bump * chore: revert version change in build.gradle Co-authored-by: mayur.gubrele <2310-mayur.gubrele@users.noreply.source.golabs.io> Co-authored-by: lavkesh <lavkesh51@gmail.com> * chore: qa and review fixes * chore: change default SCHEMA_REGISTRY_STENCIL_CACHE_AUTO_REFRESH to true * feat: Bigtable record parser (#39) * feat: Bigtable Sink (#33) * feat: add bigtable sink with stringified odpf messages as values * feat: add BIGTABLE_CREDENTIAL_PATH to access google cloud * feat: add wrapper class on bigtable client * refactor: fix checkstyle * feat: add bigtable parser tests * feat: add bigtable sink tests * feat: add bigtable client tests * chore: revert version bump * chore: revert version change in build.gradle Co-authored-by: mayur.gubrele <2310-mayur.gubrele@users.noreply.source.golabs.io> Co-authored-by: lavkesh <lavkesh51@gmail.com> * feat: create bigtable records using InputOutputFieldMapping provided as configuration * refactor: fix checkstyle and add unit tests * review: minor refactor * refactor: add BigTableSchemaTest and fix BigTableRecordParserTest * refactor: fix checkstyle * tests: add few more tests Co-authored-by: mayur.gubrele <2310-mayur.gubrele@users.noreply.source.golabs.io> Co-authored-by: lavkesh <lavkesh51@gmail.com> * chore: schema refactoring * chore: naming conventions * feat: add functionality to create rowkey from configured template (#44) * chore: refactor Template validation * chore: change exception message in Template * feat: add bigtable sink metrics and logging (#51) * feat: Bigtable error parsing (#55) * feat: Bigtable Sink (#33) * feat: add bigtable sink with stringified odpf messages as values * feat: add BIGTABLE_CREDENTIAL_PATH to access google cloud * feat: add wrapper class on bigtable client * refactor: fix checkstyle * feat: add bigtable parser tests * feat: add bigtable sink tests * feat: add bigtable client tests * chore: revert version bump * chore: revert version change in build.gradle Co-authored-by: mayur.gubrele <2310-mayur.gubrele@users.noreply.source.golabs.io> Co-authored-by: lavkesh <lavkesh51@gmail.com> * chore: qa and review fixes * chore: change default SCHEMA_REGISTRY_STENCIL_CACHE_AUTO_REFRESH to true * feat: Bigtable record parser (#39) * feat: Bigtable Sink (#33) * feat: add bigtable sink with stringified odpf messages as values * feat: add BIGTABLE_CREDENTIAL_PATH to access google cloud * feat: add wrapper class on bigtable client * refactor: fix checkstyle * feat: add bigtable parser tests * feat: add bigtable sink tests * feat: add bigtable client tests * chore: revert version bump * chore: revert version change in build.gradle Co-authored-by: mayur.gubrele <2310-mayur.gubrele@users.noreply.source.golabs.io> Co-authored-by: lavkesh <lavkesh51@gmail.com> * feat: create bigtable records using InputOutputFieldMapping provided as configuration * refactor: fix checkstyle and add unit tests * review: minor refactor * refactor: add BigTableSchemaTest and fix BigTableRecordParserTest * refactor: fix checkstyle * tests: add few more tests Co-authored-by: mayur.gubrele <2310-mayur.gubrele@users.noreply.source.golabs.io> Co-authored-by: lavkesh <lavkesh51@gmail.com> * chore: schema refactoring * chore: naming conventions * feat: add functionality to create rowkey from configured template (#44) * chore: refactor Template validation * chore: change exception message in Template * feat: add bigtable sink metrics and logging (#51) * feat: parse bigtable errors and create odpf response * feat: capture error metrics with predefined tags * chore: add tests for bigtable response parser * chore: remove deprecated jcenter repo from build.gradle * chore: comment out classpath dependencies * chore: remove unused classpath dependencies * chore: minor refactor * chore: remove unused classpath dependencies * refactor: change BigtableRecord,Response contracts, improved logging, added tests Co-authored-by: mayur.gubrele <2310-mayur.gubrele@users.noreply.source.golabs.io> Co-authored-by: lavkesh <lavkesh51@gmail.com> * chore: minor fixes (#59) * feat: Bigtable Sink (#33) * feat: add bigtable sink with stringified odpf messages as values * feat: add BIGTABLE_CREDENTIAL_PATH to access google cloud * feat: add wrapper class on bigtable client * refactor: fix checkstyle * feat: add bigtable parser tests * feat: add bigtable sink tests * feat: add bigtable client tests * chore: revert version bump * chore: revert version change in build.gradle Co-authored-by: mayur.gubrele <2310-mayur.gubrele@users.noreply.source.golabs.io> Co-authored-by: lavkesh <lavkesh51@gmail.com> * chore: qa and review fixes * chore: change default SCHEMA_REGISTRY_STENCIL_CACHE_AUTO_REFRESH to true * feat: Bigtable record parser (#39) * feat: Bigtable Sink (#33) * feat: add bigtable sink with stringified odpf messages as values * feat: add BIGTABLE_CREDENTIAL_PATH to access google cloud * feat: add wrapper class on bigtable client * refactor: fix checkstyle * feat: add bigtable parser tests * feat: add bigtable sink tests * feat: add bigtable client tests * chore: revert version bump * chore: revert version change in build.gradle Co-authored-by: mayur.gubrele <2310-mayur.gubrele@users.noreply.source.golabs.io> Co-authored-by: lavkesh <lavkesh51@gmail.com> * feat: create bigtable records using InputOutputFieldMapping provided as configuration * refactor: fix checkstyle and add unit tests * review: minor refactor * refactor: add BigTableSchemaTest and fix BigTableRecordParserTest * refactor: fix checkstyle * tests: add few more tests Co-authored-by: mayur.gubrele <2310-mayur.gubrele@users.noreply.source.golabs.io> Co-authored-by: lavkesh <lavkesh51@gmail.com> * chore: schema refactoring * chore: naming conventions * feat: add functionality to create rowkey from configured template (#44) * chore: refactor Template validation * chore: change exception message in Template * feat: add bigtable sink metrics and logging (#51) * feat: parse bigtable errors and create odpf response * feat: capture error metrics with predefined tags * chore: add tests for bigtable response parser * chore: remove deprecated jcenter repo from build.gradle * chore: add bq error logs (#57) * chore: comment out classpath dependencies * chore: remove unused classpath dependencies * chore: minor refactor * chore: remove unused classpath dependencies * refactor: change BigtableRecord,Response contracts, improved logging, added tests * chore: small fixes Co-authored-by: Mayur Gubrele <mayur.gubrele@gmail.com> Co-authored-by: mayur.gubrele <2310-mayur.gubrele@users.noreply.source.golabs.io> * docs: adds bigtable sink documentation (#58) * feat: Bigtable Sink (#33) * feat: add bigtable sink with stringified odpf messages as values * feat: add BIGTABLE_CREDENTIAL_PATH to access google cloud * feat: add wrapper class on bigtable client * refactor: fix checkstyle * feat: add bigtable parser tests * feat: add bigtable sink tests * feat: add bigtable client tests * chore: revert version bump * chore: revert version change in build.gradle Co-authored-by: mayur.gubrele <2310-mayur.gubrele@users.noreply.source.golabs.io> Co-authored-by: lavkesh <lavkesh51@gmail.com> * chore: qa and review fixes * chore: change default SCHEMA_REGISTRY_STENCIL_CACHE_AUTO_REFRESH to true * feat: Bigtable record parser (#39) * feat: Bigtable Sink (#33) * feat: add bigtable sink with stringified odpf messages as values * feat: add BIGTABLE_CREDENTIAL_PATH to access google cloud * feat: add wrapper class on bigtable client * refactor: fix checkstyle * feat: add bigtable parser tests * feat: add bigtable sink tests * feat: add bigtable client tests * chore: revert version bump * chore: revert version change in build.gradle Co-authored-by: mayur.gubrele <2310-mayur.gubrele@users.noreply.source.golabs.io> Co-authored-by: lavkesh <lavkesh51@gmail.com> * feat: create bigtable records using InputOutputFieldMapping provided as configuration * refactor: fix checkstyle and add unit tests * review: minor refactor * refactor: add BigTableSchemaTest and fix BigTableRecordParserTest * refactor: fix checkstyle * tests: add few more tests Co-authored-by: mayur.gubrele <2310-mayur.gubrele@users.noreply.source.golabs.io> Co-authored-by: lavkesh <lavkesh51@gmail.com> * chore: schema refactoring * chore: naming conventions * feat: add functionality to create rowkey from configured template (#44) * chore: refactor Template validation * chore: change exception message in Template * feat: add bigtable sink metrics and logging (#51) * docs: add bigtable sink documentation Co-authored-by: mayur.gubrele <2310-mayur.gubrele@users.noreply.source.golabs.io> Co-authored-by: lavkesh <lavkesh51@gmail.com> * chore: rc version * fix: fix uri and exception handler * feat: exclude io.grpc from bigtable library and bump the version to 0.3.4-beta.4 * chore: add iterator in http record and toString in Template * chore: add more exception handler * Revert "chore: add more exception handler" This reverts commit 0edb3b7. * Revert "chore: add iterator in http record and toString in Template" This reverts commit a18bad6. * fix: fix template, uri and http exception handling * fix: fix exception handling on response and request body * fix: fix template exception * Revert "fix: fix template exception" This reverts commit 746754a. * chore: move IOException handling for request and response body Co-authored-by: Mayur Gubrele <mayur.gubrele@gmail.com> Co-authored-by: mayur.gubrele <2310-mayur.gubrele@users.noreply.source.golabs.io> Co-authored-by: lavkesh <lavkesh51@gmail.com> * feat: add batch request (#68) Co-authored-by: lavkesh <lavkesh51@gmail.com> * Http batch request (#70) * feat: add batch request * chore: minor refactor Co-authored-by: jesrypandawa <jesry.p.pandawa@gmail.com> * feat: header builder generates headers from template (#63) * feat: header builder generates headers from template * refactor: fix for the bug where baseheaders were getting updated with template ones * add a maven repo in build.gradle * feat: add messageContainer class and handle exceptions for headerBuilder flow * refactor: add tests for headers in singleRequestTest * refactor: resolve review comments * refactor: minor refactoring as per review comments * feat: add http query parameter builder (#71) * feat: add query param builder * chore: add minor changes * refactor: refactor builder class to cache schema, header,queryparam,and url * feat: get sink_retryable_error for configured retry http response code (#75) * feat: get sink_retryable_error for configured retry http response code ranges * refactor: changes based on review comments * refactor: getMapping on ParsedMessage * feat: add ProtoJsonProvider to work with JSONPath * feat: use ProtoJsonProvider for getFieldByName on ProtoParsedMessage * chore: remove unused imports * feat: add config to log http request based on response code (#74) * feat: add config to log http request based on response code * refactor: remove unused import * refactor: fix error after merge * refactor: refactor log request method name * refactor: refactor http request for parsedMessage design changes * refactor: refactor request body (#77) * feat: add http templatized json body (#79) * refactor: refactor request body * feat: add templatized json body * chore: explicit the exception in http sink factory, add more test for templatized json * feat: moving to gotocompany * chore: cleanup schema class references * feat: implement getMapping using ParsedMessage interface * wip: add proto implementation * feat: add json implementation * chore: add tests for schema implementation * chore: add tests for generic json schema implementation * chore: remoeve getMapping from parsedMessage and clean up test * chore: add tests for byte and enum types in RecordConverter * chore: clean up unused protoField and jsonfield class * chore: add tests for getFields ProtoParsedMessage * feat: include default fields in Proto message * feat: add float point check for Double * refactor: inline proto logical type parse * refactor: ProtoLogicalValue constructor --------- Co-authored-by: jesry.pandawa <jesry.pandawa@gojek.com> * feat: add toJson in parsedMessage and implementation (raystack#6) * feat: add toJson in parsedMessage and implementation * chore: fix config and unit test * chore: fix checkstyle * chore: fix test * chore: remove preserve proto field names config * feat: add http sink message payload type (#8) * feat: add http sink json payload type (#7) * feat: add http sink json payload type * fix: http sink payload key cases --------- Co-authored-by: Harikrishna <harikrishnakanchi@gmail.com> * feat: add http delete method (#9) * feat: add http delete method * refactor: remove comments --------- Co-authored-by: Harikrishna <harikrishnakanchi@gmail.com> * chore: requestutils refactor * refactor: http to httpv2 prefix (#20) * refactor: http to httpv2 prefix * refactor: http to httpv2 prefix * refactor: http to httpv2 prefix * refactor: http to httpv2 prefix * refactor: http to httpv2 prefix * refactor: http to httpv2 prefix * refactor: http to httpv2 prefix * fix: add SINK prefix in HttpClientConfigs (#17) * fix: add SINK prefix in HttpClientConfigs * fix add SINK prefix in HttpClientConfigs * fix: add httpv2 prefix in http configs * tests: fix tests * tests: fix tests * tests: fix tests * fix: fix configs * fix: fix checkstyle * feat: http sink metrics (#14) * feat: http sink metrics * test: add http metrics tests * chore: beta release version * refactor: update exception message * chore: version bump * chore: version bump * fix: remove simple date format (#23) * fix: throw exception when template is used in batch mode for url , query and header * fix: log request payload body in debug mode * chore: fix build * test: log request payload body in debug mode * test: log request payload body in debug mode * fix: add config for default value for empty fields (#25) * fix: add config for default value for empty fields * fix: add config for default value for empty fields * refactor: rename variables which are not final anymore * fix: default value behaviour for empty fields * test: fix tests * test: fix tests * test: fix tests * test: fix tests * test: fix tests * feat: add support for JSON array and primitives in Json body template (#40) * feat: add support for JSON array and primitives in Json body template * feat: add support for JSON array and primitives in Json body template * feat: add support for JSON array and primitives in Json body template * feat: add support for JSON array and primitives in Json body template * feat: add support for JSON array and primitives in Json body template * feat: add support for JSON array and primitives in Json body template * feat: add support for JSON array and primitives in Json body template * feat: add support for JSON array and primitives in Json body template * feat: add support for JSON array and primitives in Json body template * tests: add unit tests for json parser * refactor: removed extra checks * fix: fix check style * fix: fix check style * fix: fix check style * fix: fix check style * fix: fix check style * tests: fix unit tests * tests: fix unit tests * refactor: change implementation to use jackson library instead of gson * refactor: change implementation to use jackson library instead of gso * refactor: move json parsing logic to JsonParserUtils * tests: add unit tests for JsonParserUtil * fix: fix check style * fix: fix check style * fix: fix bugs and unit tests * fix: fix check style * fix: fix check style * fix: fix check style * fix: fix check style * fix: fix check style * fix: fix PR review comments * docs: add java doc for utility methods * test: add unit tests for JsonParserUtils * test: add unit tests for JsonParserUtils * test: add unit tests for JsonParserUtils * test: add unit tests for JsonParserUtils --------- Co-authored-by: lavkesh <lavkesh51@gmail.com> * docs: add docs for Httpv2 sink (#41) * docs: add docs for Httpv2 sink * docs: add docs for Httpv2 sink * docs: add docs for Httpv2 sink * docs: add docs for Httpv2 sink * docs: add docs for Httpv2 sink * docs: add docs for Httpv2 sink * docs: add docs for Httpv2 sink * docs: add docs for Httpv2 sink * docs: add docs for Httpv2 sink * docs: add docs for Httpv2 sink * docs: add docs for Httpv2 sink * docs: add docs for Httpv2 sink * docs: add docs for Httpv2 sink * docs: add docs for Httpv2 sink * docs: add docs for Httpv2 sink * docs: add docs for Httpv2 sink * docs: add docs for Httpv2 sink * docs: add docs for Httpv2 sink * docs: add docs for Httpv2 sink * fix: remove messageschema get schema method * fix: remove default values of oauth configs * refactor: revert bq storage sink changes (#42) * fix: fix http connection pool error (#43) * fix: fix http connection pool error * fix: fix build error * fix: fix response code logic and consuming response body --------- Co-authored-by: gagandhand <gagan.deep@gojek.com> Co-authored-by: gdgagan696 <33114447+gdgagan696@users.noreply.github.com> * docs: add docs for httpv2 sink metrics * docs: add docs for httpv2 sink metrics * fix: change default field value enable to true --------- Co-authored-by: jesrypandawa <78069094+jesrypandawa@users.noreply.github.com> Co-authored-by: lavkesh <lavkesh51@gmail.com> Co-authored-by: jesry.pandawa <jesry.pandawa@gojek.com> Co-authored-by: Mayur Gubrele <mayur.gubrele@gmail.com> Co-authored-by: mayur.gubrele <2310-mayur.gubrele@users.noreply.source.golabs.io> Co-authored-by: jesrypandawa <jesry.p.pandawa@gmail.com> Co-authored-by: Harikrishna <harikrishnakanchi@gmail.com> Co-authored-by: gagandhand <gagan.deep@gojek.com> Co-authored-by: gdgagan696 <33114447+gdgagan696@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.