GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
No description provided.
Initial schema conversion for flat tables
Implemented test checks
arrays of primitive types
Finished schema generation
Added _USED_HCAT_FILTER column
Changed partitioning descriptor
Added consideration of table postfixes
Generalizing HCat Record Traversal
1st refactoring of HCatSchema traversal, not yet generic enough
Made transformation more functional, and support transformation along…
… HCatSchema on the record level as well.
First Implementation of Record Mapping
Reshaped Constructor Interface to not use functions but present hooks…
… as normal interface methods
Implemented record conversion and tested insertion of converted recor…
…ds into tables
Added retry function to bigqueryutils
Implemented OutputFormat and RecordWriter for BigQuery export job
Corrected passing of configuration parameters to BigQueryOutputFormat
Changing streaming approach to avoid temporal tables
Implemented Job Rollback in case of Error
Implemented OutputFormat Tests, failed though at streaming into older…
… partitions. Refactoring to use Cloud Storage required.
Finally managed to get BigQuery output format running and loading dat…
…a into historical partitioned tables
Fixed logging properties in tests
Added java doc, support for proxies, and table postfixes
Created MR Job skeleton
Finished MapReduce job for BigQuery export and tested it
Outlined Schedoscope export syntax for BigQueryk
Implemented first version of Schedoscope export syntax for BigQuery e…
Made export use an identity reducer to control parallelism
Implemented BigQuery export integration test with schedoscope export DSL