New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
FixNPEIncremental #9
Commits on Nov 14, 2021
Commits on Nov 15, 2021
-
[HUDI-2758] remove redundant code in the hoodieRealtimeInputFormatUit…
…ls.getRealtimeSplits (apache#3994)
-
[MINOR] Fix typo in IntervalTreeBasedGlobalIndexFileFilter (apache#3993)
Co-authored-by: 闫杜峰 <yandufeng@sinochem.com>
-
[HUDI-2744] Fix parsing of metadadata table compaction timestamp when…
… metrics are enabled (apache#3976)
-
[HUDI-2683] Parallelize deleting archived hoodie commits (apache#3920)
Co-authored-by: yuezhang <yuezhang@freewheel.tv>
Commits on Nov 16, 2021
Commits on Nov 17, 2021
-
[MINOR] Fixed checkstyle config to be based off Maven root-dir (requi…
…res Maven >=3.3.1 to work properly); (apache#4009) Updated README
Alexey Kudinkin committedNov 17, 2021 -
[HUDI-2151] Part3 Enabling marker based rollback as default rollback …
…strategy (apache#3950) * Enabling timeline server based markers * Enabling timeline server based markers and marker based rollback * Removing constraint that timeline server can be enabled only for hdfs * Fixing tests
-
Check --source-avro-schema-path parameter (apache#3987)
Co-authored-by: 0x3E6 <dragon1996>
-
[MINOR] Add the Schema for GooseFS to StorageSchemes (apache#3982)
Co-authored-by: lubo <bollu@tencent.com>
-
[HUDI-2314] Add support for DynamoDb based lock provider (apache#3486)
- Co-authored-by: Wenning Ding <wenningd@amazon.com> - Co-authored-by: Sivabalan Narayanan <n.siva.b@gmail.com>
-
Commits on Nov 18, 2021
-
[HUDI-2362] Add external config file support (apache#3416)
Co-authored-by: Wenning Ding <wenningd@amazon.com>
-
[HUDI-2641] Avoid deleting all inflight commits heartbeats while roll…
…ing back failed writes (apache#3956)
Commits on Nov 19, 2021
-
-
[HUDI-2593] Virtual keys support for metadata table (apache#3968)
- Metadata table today has virtual keys disabled, thereby populating the metafields for each record written out and increasing the overall storage space used. Hereby adding virtual keys support for metadata table so that metafields are disabled for metadata table records. - Adding a custom KeyGenerator for Metadata table so as to not rely on the default Base/SimpleKeyGenerators which currently look for record key and partition field set in the table config. - AbstractHoodieLogRecordReader's version of processing next data block and createHoodieRecord() will be a generic version and making the derived class HoodieMetadataMergedLogRecordReader take care of the special creation of records from explictly passed in partition names.
Commits on Nov 20, 2021
-
[HUDI-2472] Enabling metadata table for TestHoodieMergeOnReadTable an…
…d TestHoodieCompactor (apache#4023)
-
[HUDI-2796] Metadata table support for Restore action to first commit (…
…apache#4039) - Adding support for the metadata table to restore to first commit and take proper action for the bootstrap on subequent commits.
-
[HUDI-2242] Add configuration inference logic for few options (apache…
…#3359) Co-authored-by: Wenning Ding <wenningd@amazon.com>
-
[HUDI-2742] Added S3 object filter to support multiple S3EventsHoodie…
…IncrSources single S3 meta table (apache#4025)
-
[HUDI-2795] Add mechanism to safely update,delete and recover table p…
…roperties (apache#4038) * [HUDI-2795] Add mechanism to safely update,delete and recover table properties - Fail safe mechanism, that lets queries succeed off a backup file - Readers who are not upgraded to this version of code will just fail until recovery is done. - Added unit tests that exercises all these scenarios. - Adding CLI for recovery, updation to table command. - [Pending] Add some hash based verfication to ensure any rare partial writes for HDFS * Fixing upgrade/downgrade infrastructure to use new updation method
Commits on Nov 21, 2021
-
[MINOR] optimize in constructor of inputbatch class (apache#4040)
Co-authored-by: 闫杜峰 <yandufeng@sinochem.com>
-
[HUDI-1932] Update Hive sync timestamp when change detected (apache#3053
) * Update Hive sync timestamp when change detected Only update the last commit timestamp on the Hive table when the table schema has changed or a partition is created/updated. When using AWS Glue Data Catalog as the metastore for Hive this will ensure that table versions are substantive (including schema and/or partition changes). Prior to this change when a Hive sync is performed without schema or partition changes the table in the Glue Data Catalog would have a new version published with the only change being the timestamp property. https://issues.apache.org/jira/browse/HUDI-1932 * add conditional sync flag * fix testSyncWithoutDiffs * fix HiveSyncConfig Co-authored-by: Raymond Xu <2701446+xushiyan@users.noreply.github.com>
-
Commits on Nov 22, 2021
-
[HUDI-1870] Add more Spark CI build tasks (apache#4022)
* [HUDI-1870] Add more Spark CI build tasks - build for spark3.0.x - build for spark-shade-unbundle-avro - fix build failures - delete unnecessary assertion for spark 3.0.x - use AvroConversionUtils#convertAvroSchemaToStructType instead of calling SchemaConverters#toSqlType directly to solve the compilation failures with spark-shade-unbundle-avro (apache#5) Co-authored-by: Yann <biyan900116@gmail.com>
-
[HUDI-2533] New option for hoodieClusteringJob to check, rollback and…
… re-execute the last failed clustering job (apache#3765) * coding finished and need to do uts * add uts * code review * code review Co-authored-by: yuezhang <yuezhang@freewheel.tv>
-
[HUDI-2472] Enabling metadata table for TestHoodieIndex test case (ap…
…ache#4045) - Enablng the metadata table for testSimpleGlobalIndexTagLocationWhenShouldUpdatePartitionPath. This is more of a test issue.
-
[HUDI-2559] Converting commit timestamp format to millisecs (apache#4024
Commits on Nov 23, 2021
-
[HUDI-2550] Expand File-Group candidates list for appending for MOR t…
…ables (apache#3986)
Alexey Kudinkin committedNov 23, 2021 -
[HUDI-2737] Use earliest instant by default for async compaction and …
…clustering jobs (apache#3991) Address review comments Fix test failures Co-authored-by: Sagar Sumit <sagarsumit09@gmail.com>
-
[HUDI-1937] Rollback unfinished replace commit to allow updates (apac…
…he#3869) * [HUDI-1937] Rollback unfinished replace commit to allow updates while clustering * Revert and delete requested replacecommit too * Rollback pending clustering instants transactionally * No double locking and add a config to enable rollback * Update config to be clear about rollback only on conflict
-
[MINOR] Add more configuration to Kafka setup script (apache#3992)
* [MINOR] Add more configuration to Kafka setup script * Add option to reuse Kafka topic * Minor fixes to README
-
[HUDI-2778] Optimize statistics collection related codes and add some…
… docs for z-order add fix some bugs (apache#4013) * [HUDI-2778] Optimize statistics collection related codes and add more docs for z-order. * add test code for multi-thread parquet footer read
-
[HUDI-2409] Using HBase shaded jars in Hudi presto bundle (apache#3623)
* using hbase-shaded-jars-in-hudi-presto-hundle * test * add hudi-common-bundle * code review * code review * code review * code review * test * test Co-authored-by: yuezhang <yuezhang@freewheel.tv>
-
[HUDI-2332] Add clustering and compaction in Kafka Connect Sink (apac…
…he#3857) * [HUDI-2332] Add clustering and compaction in Kafka Connect Sink * Disable validation check on instant time for compaction and adjust configs * Add javadocs * Add clustering and compaction config * Fix transaction causing missing records in the target table * Add debugging logs * Fix kafka offset sync in participant * Adjust how clustering and compaction are configured in kafka-connect * Fix clustering strategy * Remove irrelevant changes from other published PRs * Update clustering logic and others * Update README * Fix test failures * Fix indentation * Fix clustering config * Add JavaCustomColumnsSortPartitioner and make async compaction enabled by default * Add test for JavaCustomColumnsSortPartitioner * Add more changes after IDE sync * Update README with clarification * Fix clustering logic after rebasing * Remove unrelated changes
-
[HUDI-2325] Add hive sync support to kafka connect (apache#3660)
Co-authored-by: Rajesh Mahindra <rmahindra@Rajeshs-MacBook-Pro.local>
Commits on Nov 24, 2021
-
[HUDI-2831] Securing usages of
SimpleDateFormat
to be thread-safe (a…Alexey Kudinkin committedNov 24, 2021 -
Revert "[HUDI-2799] Fix the classloader of flink write task (apache#4042
)" (apache#4069) This reverts commit 8281cbf.
-
[HUDI-2759] extract HoodieCatalogTable to coordinate spark catalog ta…
…ble and hoodie table (apache#3998)
-
[HUDI-2671] Fix kafka offset handling in Kafka Connect protocol (apac…
…he#4021) Co-authored-by: Rajesh Mahindra <rmahindra@Rajeshs-MacBook-Pro.local>
-
[HUDI-2443] Hudi KVComparator for all HFile writer usages (apache#3889)
* [HUDI-2443] Hudi KVComparator for all HFile writer usages - Hudi relies on custom class shading for Hbase's KeyValue.KVComparator to avoid versioning and class loading issues. There are few places which are still using the Hbase's comparator class directly and version upgrades would make them obsolete. Refactoring the HoodieKVComparator and making all HFile writer creation using the same shaded class. * [HUDI-2443] Hudi KVComparator for all HFile writer usages - Moving HoodieKVComparator from common.bootstrap.index to common.util * [HUDI-2443] Hudi KVComparator for all HFile writer usages - Retaining the old HoodieKVComparatorV2 for boostrap case. Adding the new comparator as HoodieKVComparatorV2 to differentiate from the old one. * [HUDI-2443] Hudi KVComparator for all HFile writer usages - Renamed HoodieKVComparatorV2 to HoodieMetadataKVComparator and moved it under the package org.apache.hudi.metadata. * Make comparator classname configurable * Revert new config and address other review comments Co-authored-by: Sagar Sumit <sagarsumit09@gmail.com>
-
[HUDI-2788] Fixing issues w/ Z-order Layout Optimization (apache#4026)
* Simplyfying, tidying up * Fixed packaging for `TestOptimizeTable` * Cleaned up `HoodiFileIndex` file filtering seq; Removed optimization manually reading Parquet table circumventing Spark * Refactored `DataSkippingUtils`: - Fixed checks to validate all statistics cols are present - Fixed some predicates being constructed incorrectly - Rewrote comments for easier comprehension, added more notes - Tidying up * Tidying up tests * `lint` * Fixing compilation * `TestOptimizeTable` > `TestTableLayoutOptimization`; Added assertions to test data skipping paths * Fixed tests to properly hit data-skipping path * Fixed pruned files candidates lookup seq to conservatively included all non-indexed files * Added java-doc * Fixed compilation
Alexey Kudinkin committedNov 24, 2021 -
[HUDI-2766] Cluster update strategy should not be fenced by write con…
…fig (apache#4093) Fix pending clustering rollback test
-
[HUDI-2793] Fixing deltastreamer checkpoint fetch/copy over (apache#4034
Commits on Nov 25, 2021
-
[HUDI-2853] Add JMX deps in hudi utilities and kafka connect bundles (a…
…pache#4108) Co-authored-by: Rajesh Mahindra <rmahindra@Rajeshs-MacBook-Pro.local>
-
[HUDI-2844][CLI] Fixing archived Timeline crashing if timeline contai…
…ns REPLACE_COMMIT (apache#4091)
Alexey Kudinkin committedNov 25, 2021 -
[HUDI-1290] [RFC-39] Deltastreamer avro source for Debezium CDC (apac…
…he#4048) * Add RFC entry for deltastreamer source for debezium * Add RFC for debezium source * Add RFC for debezium source * Add RFC for debezium source * fix hyperlink issue and rebase * Update progress Co-authored-by: Rajesh Mahindra <rmahindra@Rajeshs-MacBook-Pro.local>
-
[HUDI-1290] Add Debezium Source for deltastreamer (apache#4063)
* add source for postgres debezium * Add tests for debezium payload * Fix test * Fix test * Add tests for debezium source * Add tests for debezium source * Fix schema for debezium * Fix checkstyle issues * Fix config issue for schema registry * Add mysql source for debezium * Fix checkstyle issues an tests * Improve code for merging toasted values * Improve code for merging toasted values Co-authored-by: Rajesh Mahindra <rmahindra@Rajeshs-MacBook-Pro.local>
-
[HUDI-2792] Configure metadata payload consistency check (apache#4035)
- Relax metadata payload consistency check to consider spark task failures with spurious deletes
-
[HUDI-2855] Change the default value of 'PAYLOAD_CLASS_NAME' to 'Defa…
…ultHoodieRecordPayload' (apache#4115)
-
[HUDI-2480] FileSlice after pending compaction-requested instant-time… (
apache#3703) * [HUDI-2480] FileSlice after pending compaction-requested instant-time is ignored by MOR snapshot reader * include file slice after a pending compaction for spark reader Co-authored-by: garyli1019 <yanjia.gary.li@gmail.com>
-
-
[HUDI-2800] Remove rdd.isEmpty() validation to prevent CreateHandle b…
…eing called twice (apache#4121)
-
[HUDI-2794] Guarding table service commits within a single lock to co…
…mmit to both data table and metadata table (apache#4037) * Fixing a single lock to commit table services across metadata table and data table * Addressing comments * rebasing with master
-
[HUDI-2671] Making error -> warn logs from timeline server with concu…
…rrent writers for inconsistent state (apache#4088) * Making error -> warn logs from timeline server with concurrent writers for inconsistent state * Fixing bad request response exception for timeline out of sync * Addressing feedback. removed write concurrency mode depedency
-
[HUDI-2840] Fixed DeltaStreaemer to properly respect configuration pa…
…ssed t/h properties file (apache#4090) * Rebased `DFSPropertiesConfiguration` to access Hadoop config in liue of FS to avoid confusion * Fixed `readConfig` to take Hadoop's `Configuration` instead of FS; Fixing usages * Added test for local FS access * Rebase to use `FSUtils.getFs` * Combine properties provided as a file along w/ overrides provided from the CLI * Added helper utilities to `HoodieClusteringConfig`; Make sure corresponding config methods fallback to defaults; * Fixed DeltaStreamer usage to respect properly combined configuration; Abstracted `HoodieClusteringConfig.from` convenience utility to init Clustering config from `Properties` * Tidying up * `lint` * Reverting changes to `HoodieWriteConfig` * Tdiying up * Fixed incorrect merge of the props * Converted `HoodieConfig` to wrap around `Properties` into `TypedProperties` * Fixed compilation * Fixed compilation
Alexey Kudinkin committedNov 25, 2021
Commits on Nov 26, 2021
-
[MINOR] Include hudi-aws in flink bundle jar (apache#4127)
HUDI-2801 makes this jar as required.
-
[HUDI-2852] Table metadata returns empty for non-exist partition (apa…
…che#4117) * [HUDI-2852] Table metadata returns empty for non-exist partition * add unit test * fix code checkstyle Co-authored-by: wangminchao <wangminchao@asinking.com>
-
[HUDI-2850] Fixing Clustering CLI - schedule and run command fixes to…
… avoid NumberFormatException (apache#4101)
-
[HUDI-2814] Addressing issues w/ Z-order Layout Optimization (apache#…
…4060) * `ZCurveOptimizeHelper` > `ZOrderingIndexHelper`; Moved Z-index helper under `hudi.index.zorder` package * Tidying up `ZOrderingIndexHelper` * Fixing compilation * Fixed index new/original table merging sequence to always prefer values from new index; Cleaned up `HoodieSparkUtils` * Added test for `mergeIndexSql` * Abstracted Z-index name composition w/in `ZOrderingIndexHelper`; * Fixed `DataSkippingUtils` to interrupt prunning in case data filter contains non-indexed column reference * Properly handle exceptions origination during pruning in `HoodieFileIndex` * Make sure no errors are logged upon encountering `AnalysisException` * Cleaned up Z-index updating sequence; Tidying up comments, java-docs; * Fixed Z-index to properly handle changes of the list of clustered columns * Tidying up * `lint` * Suppressing `JavaDocStyle` first sentence check * Fixed compilation * Fixing incorrect `DecimalType` conversion * Refactored test `TestTableLayoutOptimization` - Added Z-index table composition test (against fixtures) - Separated out GC test; Tidying up * Fixed tests re-shuffling column order for Z-Index table `DataFrame` to align w/ the one by one loaded from JSON * Scaffolded `DataTypeUtils` to do basic checks of Spark types; Added proper compatibility checking b/w old/new index-tables * Added test for Z-index tables merging * Fixed import being shaded by creating internal `hudi.util` package * Fixed packaging for `TestOptimizeTable` * Revised `updateMetadataIndex` seq to provide Z-index updating process w/ source table schema * Make sure existing Z-index table schema is sync'd to source table's one * Fixed shaded refs * Fixed tests * Fixed type conversion of Parquet provided metadata values into Spark expected schemas * Fixed `composeIndexSchema` utility to propose proper schema * Added more tests for Z-index: - Checking that Z-index table is built correctly - Checking that Z-index tables are merged correctly (during update) * Fixing source table * Fixing tests to read from Parquet w/ proper schema * Refactored `ParquetUtils` utility reading stats from Parquet footers * Fixed incorrect handling of Decimals extracted from Parquet footers * Worked around issues in javac failign to compile stream's collection * Fixed handling of `Date` type * Fixed handling of `DateType` to be parsed as `LocalDate` * Updated fixture; Make sure test loads Z-index fixture using proper schema * Removed superfluous scheme adjusting when reading from Parquet, since Spark is actually able to perfectly restore schema (given Parquet was previously written by Spark as well) * Fixing race-condition in Parquet's `DateStringifier` trying to share `SimpleDataFormat` object which is inherently not thread-safe * Tidying up * Make sure schema is used upon reading to validate input files are in the appropriate format; Tidying up; * Worked around javac (1.8) inability to infer expression type properly * Updated fixtures; Tidying up * Fixing compilation after rebase * Assert clustering have in Z-order layout optimization testing * Tidying up exception messages * XXX * Added test validating Z-index lookup filter correctness * Added more test-cases; Tidying up * Added tests for string expressions * Fixed incorrect Z-index filter lookup translations * Added more test-cases * Added proper handling on complex negations of AND/OR expressions by pushing NOT operator down into inner expressions for appropriate handling * Added `-target:jvm-1.8` for `hudi-spark` module * Adding more tests * Added tests for non-indexed columns * Properly handle non-indexed columns by falling back to a re-write of containing expression as `TrueLiteral` instead * Fixed tests * Removing the parquet test files and disabling corresponding tests Co-authored-by: Vinoth Chandar <vinoth@apache.org>
-
[HUDI-2767] Enabling timeline-server-based marker as default (apache#…
…4112) - Changes the default config of marker type (HoodieWriteConfig.MARKERS_TYPE or hoodie.write.markers.type) from DIRECT to TIMELINE_SERVER_BASED for Spark Engine. - Adds engine-specific marker type configs: Spark -> TIMELINE_SERVER_BASED, Flink -> DIRECT, Java -> DIRECT. - Uses DIRECT markers as well for Spark structured streaming due to timeline server only available for the first mini-batch. - Fixes the marker creation method for non-partitioned table in TimelineServerBasedWriteMarkers. - Adds the fallback to direct markers even when TIMELINE_SERVER_BASED is configured, in WriteMarkersFactory: when HDFS is used, or embedded timeline server is disabled, the fallback to direct markers happens. - Fixes the closing of timeline service. - Fixes tests that depend on markers, mainly by starting the timeline service for each test.
-
[HUDI-2845] Metadata CLI - files/partition file listing fix and new v…
…alidate option (apache#4092) - Co-authored-by: Sivabalan Narayanan <n.siva.b@gmail.com>
-
-
[HUDI-2864] Fix README and scripts with current limitations of hive s…
…ync (apache#4129) * Fix README with current limitations of hive sync * Fix README with current limitations of hive sync * Fix dep issue * Fix Copy on Write flow Co-authored-by: Rajesh Mahindra <rmahindra@Rajeshs-MacBook-Pro.local>
-
[HUDI-2856] Bit cask disk map delete modified (apache#4116)
* modified BitCaskDiskMap_close_function * change iterators location to finally * Update BitCaskDiskMap.java
Commits on Nov 27, 2021
-
[HUDI-2868] Fix skipped HoodieSparkSqlWriterSuite (apache#4125)
- Co-authored-by: Yann Byron <biyan900116@gmail.com>
-
[HUDI-2475] [HUDI-2862] Metadata table creation and avoid bootstrappi…
…ng race for write client & add locking for upgrade (apache#4114) Co-authored-by: Sivabalan Narayanan <n.siva.b@gmail.com>
-
[HUDI-2102] Support hilbert curve for hudi (apache#3952)
Co-authored-by: Y Ethan Guo <ethan.guoyihua@gmail.com>
-
Commits on Nov 28, 2021
Commits on Nov 29, 2021
Commits on Nov 30, 2021
Commits on Dec 1, 2021
-
Revert "[HUDI-2855] Change the default value of 'PAYLOAD_CLASS_NAME' …
…to 'DefaultHoodieRecordPayload' (apache#4115)" (apache#4169) This reverts commit 88067f5.
Alexey Kudinkin committedDec 1, 2021 -
Revert "[HUDI-2856] Bit cask disk map delete modified (apache#4116)" (a…
…pache#4171) This reverts commit 257a6a7.
-
[HUDI-2880] Fixing loading of props from default dir (apache#4167)
* Fixing loading of props from default dir * addressing comments
Commits on Dec 2, 2021
-
[HUDI-2881] Compact the file group with larger log files to reduce wr…
…ite amplification (apache#4152)
-
Fixed partitions produced by layout optimization in case order-by key…
… is composed of a single column (apache#4183)
Alexey Kudinkin committedDec 2, 2021 -
[MINOR] Fix the wrong usage of timestamp length variable bug (apache#…
…4179) Signed-off-by: zzzhy <candle_1667@163.com>
-
[HUDI-2904] Fix metadata table archival overstepping between regular …
…writers and table services (apache#4186) - Co-authored-by: Rajesh Mahindra <rmahindra@Rajeshs-MacBook-Pro.local> - Co-authored-by: Sivabalan Narayanan <n.siva.b@gmail.com>
Commits on Dec 3, 2021
-
[HUDI-2902] Fixing populate meta fields with Hfile writers and Disabl…
…ing virtual keys by default for metadata table (apache#4194)
-
[HUDI-2911] Removing default value for
PARTITIONPATH_FIELD_NAME
res……ulting in incorrect `KeyGenerator` configuration (apache#4195)
Alexey Kudinkin committedDec 3, 2021 -
Revert "[HUDI-2495] Resolve inconsistent key generation for timestamp…
… types by GenericRecord and Row (apache#3944)" (apache#4201)
-
[HUDI-2894][HUDI-2905] Metadata table - avoiding key lookup failures …
…on base files over S3 (apache#4185) - Fetching partition files or all partitions from the metadata table is failing when run over S3. Metadata table uses HFile format for the base files and the record lookup uses HFile.Reader and HFileScanner interfaces to get records by partition keys. When the backing storage is S3, this record lookup from HFiles is failing with IOException, in turn failing the caller commit/update operations. - Metadata table looks up HFile records with positional read enabled so as to perform better for random lookups. But this positional read key lookup is returning with partial read sizes over S3 leading to HFile scanner throwing IOException. This doesn't happen over HDFS. Metadata table though uses the HFile for random key lookups, the positional read is not mandatory as we sort the keys when doing a lookup for multiple keys. - The fix is to disable HFile positional read for all HFile scanner based key lookups.
Commits on Dec 4, 2021
-
Revert "[HUDI-2489]Tuning HoodieROTablePathFilter by caching hoodieTa…
…bleFileSystemView, aiming to reduce unnecessary list/get requests" Co-authored-by: yuezhang <yuezhang@freewheel.tv>
-
[MINOR] Mitigate CI jobs timeout issues (apache#4173)
* skip shutdown zookeeper in `@AfterAll` in TestHBaseIndex * rebalance CI tests
-
-
[HUDI-2890] Kafka Connect: Fix failed writes and avoid table service …
…concurrent operations (apache#4211) * Fix kafka connect readme * Fix handling of errors in write records for kafka connect * By default, ensure we skip error records and keep the pipeline alive * Fix indentation Co-authored-by: Rajesh Mahindra <rmahindra@Rajeshs-MacBook-Pro.local>
-
[HUDI-2923] Fixing metadata table reader when metadata compaction is …
…inflight (apache#4206) * [HUDI-2923] Fixing metadata table reader when metadata compaction is inflight * Fixing retry of pending compaction in metadata table and enhancing tests
-
[HUDI-2935] Remove special casing of clustering in deltastreamer chec…
…kpoint retrival (apache#4216) - We now seek backwards to find the checkpoint - No need to return empty anymore
Commits on Dec 5, 2021
-
[HUDI-2877] Support flink catalog to help user use flink table conven…
…iently (apache#4153) * [HUDI-2877] Support flink catalog to help user use flink table conveniently * Fix comment * fix comment2
-
[HUDI-2937] Introduce a pulsar implementation of hoodie write commit … (
apache#4217) * [HUDI-2937] Introduce a pulsar implementation of hoodie write commit callback * [HUDI-2937] Introduce a pulsar implementation of hoodie write commit callback * [HUDI-2937] Introduce a pulsar implementation of hoodie write commit callback * [HUDI-2937] Introduce a pulsar implementation of hoodie write commit callback * [HUDI-2937] Introduce a pulsar implementation of hoodie write commit callback * [HUDI-2937] Introduce a pulsar implementation of hoodie write commit callback * [HUDI-2937] Introduce a pulsar implementation of hoodie write commit callback
-
[HUDI-2418] Support HiveSchemaProvider (apache#3671)
Co-authored-by: jian.feng <fengjian428@gmial.com>
Commits on Dec 6, 2021
-
-
[HUDI-2900] Fix corrupt block end position (apache#4181)
* [HUDI-2900] Fix corrupt block end position * add a test
-
[HUDI-2876] for hive/presto hudi should remove the temp file which cr…
…eated by HoodieMergedLogRecordSanner when the query finished. (apache#4139)
-
[MINOR] Use maven-shade-plugin version for hudi-timeline-server-bundl…
…e from main pom.xml (apache#4209) Co-authored-by: Wenning Ding <wenningd@amazon.com>
Commits on Dec 7, 2021
Commits on Dec 8, 2021
-
-
[HUDI-2832][RFC-41] Proposal to integrate Hudi on Snowflake platform (a…
…pache#4074) * [HUDI-2832][RFC-40] Proposal to integrate Hudi on Snowflake platform * rebased and addressed review comments
Vinoth Govindarajan committedDec 8, 2021
Commits on Dec 9, 2021
-
[HUDI-2665] Fix overflow of huge log file in HoodieLogFormatWriter (a…
…pache#3912) Co-authored-by: guanziyue.gzy <guanziyue.gzy@bytedance.com>
-
-
[HUDI-2779] Cache BaseDir if HudiTableNotFound Exception thrown (apac…
RexAn committedDec 9, 2021 -
[HUDI-2966] Add TaskCompletionListener for HoodieMergeOnReadRDD to cl…
…ose logScaner when the query finished. (apache#4265) * [HUDI-2966] Add TaskCompletionListener for HoodieMergeOnReadRDD to close logScaner when the query finished.
-
Commits on Dec 10, 2021
-
Revert "Claiming RFC for data skipping index for updated version (apa…
…che#4271)" (apache#4272) This reverts commit 8321d20.
-
[HUDI-2912] Fix CompactionPlanOperator typo (apache#4187)
Co-authored-by: yuzhaojing <yuzhaojing@bytedance.com>
-
[HUDI-2892][BUG] Pending Clustering may stain the ActiveTimeLine and …
…lead to incomplete query results (apache#4172) Co-authored-by: yuezhang <yuezhang@freewheel.tv>
-
-
[HUDI-2814] Make Z-index more generic Column-Stats Index (apache#4106)
Alexey Kudinkin committedDec 10, 2021
Commits on Dec 11, 2021
-
[HUDI-2974] Make the prefix for metrics name configurable (apache#4274)
Co-authored-by: Rajesh Mahindra <rmahindra@Rajeshs-MacBook-Pro.local>
Commits on Dec 12, 2021
-
[HUDI-2946] Upgrade maven plugins to be compatible with higher Java v…
…ersions (apache#4232) Co-authored-by: Wenning Ding <wenningd@amazon.com>
Commits on Dec 13, 2021
-
[HUDI-2994] Add judgement to existed partitionPath in the catch code …
…block for HU… (apache#4294) * [HUDI-2994] Add judgement to existed partition path in the catch code block for HUDI-2743 Co-authored-by: wangminchao <wangminchao@asinking.com>
Commits on Dec 14, 2021
Commits on Dec 15, 2021
-
[HUDI-3022] Fix NPE for isDropPartition method (apache#4319)
* [HUDI-3022] Fix NPE for isDropPartition method
-
[HUDI-3024] Add explicit write handler for flink (apache#4329)
Co-authored-by: wangminchao <wangminchao@asinking.com>
-
[HUDI-3025] Add additional wait time for namenode availability during…
… IT tests initiatialization (apache#4328) - Co-authored-by: Sivabalan Narayanan <n.siva.b@gmail.com>
-
[HUDI-3028] Use blob storage to speed up CI downloads (apache#4331)
Co-authored-by: Sivabalan Narayanan <n.siva.b@gmail.com>
-
[HUDI-2998] claiming rfc number for consistent hashing index (apache#…
…4303) Co-authored-by: xiaoyuwei <xiaoyuwei.yw@alibaba-inc.com>
Commits on Dec 16, 2021
-
[Minor] Catch and ignore all the exceptions in quietDeleteMarkerDir (a…
…pache#4301) Co-authored-by: yuezhang <yuezhang@freewheel.tv>
Commits on Dec 17, 2021
-
[HUDI-3043] Revert async cleaner leak commit to unblock CI failure (a…
…pache#4343) * Revert "[HUDI-2959] Fix the thread leak of cleaning service (apache#4252)" Reverting to unblock CI failure for now. will revisit this with the right fix
-
[HUDI-3046] Claim RFC number for RFC for Compaction / Clustering Serv…
…ice (apache#4347) Co-authored-by: yuzhaojing <yuzhaojing@bytedance.com>
-
[HUDI-2958] Automatically set spark.sql.parquet.writelegacyformat, wh…
…en using bulkinsert to insert data which contains decimalType (apache#4253)
Commits on Dec 18, 2021
-
[HUDI-2962] InProcess lock provider to guard single writer process wi…
…th async table operations (apache#4259) - Adding Local JVM process based lock provider implementation - This local lock provider can be used by a single writer process with async table operations to guard the metadata tabl against concurrent updates.
-
-
[HUDI-3029] Transaction manager: avoid deadlock when doing begin and …
…end transactions (apache#4363) * [HUDI-3029] Transaction manager: avoid deadlock when doing begin and end transactions - Transaction manager has begin and end transactions as synchronized methods. Based on the lock provider implementaion, this can lead to deadlock situation when the underlying lock() calls are blocking or with a long timeout. - Fixing transaction manager begin and end transactions to not get to deadlock and to not assume anything on the lock provider implementation.
-
[HUDI-3029] Transaction manager: avoid deadlock when doing begin and …
…end transactions (apache#4373)
-
[HUDI-3054] Fixing default lock configs for FileSystemBasedLock and f…
…ixing a flaky test (apache#4374)
Commits on Dec 19, 2021
-
-
-
[HUDI-3064][HUDI-3054] FileSystemBasedLockProviderTestClass tryLock f…
…ix and TestHoodieClientMultiWriter test fixes (apache#4384) - Made FileSystemBasedLockProviderTestClass thread safe and fixed the tryLock retry logic. - Made TestHoodieClientMultiWriter. testHoodieClientBasicMultiWriter deterministic in verifying the HoodieWriteConflictException.
Commits on Dec 20, 2021
Commits on Dec 21, 2021
-
-
[HUDI-3070] Add rerunFailingTestsCount for flakly testes (apache#4398)
Co-authored-by: yuezhang <yuezhang@freewheel.tv>
-
Commits on Dec 22, 2021
-
[HUDI-2547] Schedule Flink compaction in service (apache#4254)
Co-authored-by: yuzhaojing <yuzhaojing@bytedance.com>
-
Merge pull request apache#4308 from harsh1231/HUDI-3008
[HUDI-3008] Fixing HoodieFileIndex partition column parsing for nested fields
-
[HUDI-3011] Adding ability to read entire data with HoodieIncrSource …
…with empty checkpoint (apache#4334) * Adding ability to read entire data with HoodieIncrSource with empty checkpoint * Addressing comments
-
-
Commits on Dec 23, 2021
Commits on Dec 24, 2021
Commits on Dec 25, 2021
Commits on Dec 28, 2021
-
[HUDI-2374] Fixing AvroDFSSource does not use the overridden schema t…
…o deserialize Avro binaries (apache#4353)
-
-
-
Commits on Dec 29, 2021
Commits on Dec 30, 2021
-
Revert "[HUDI-3043] Revert async cleaner leak commit to unblock CI fa…
…ilure (apache#4343)" (apache#4465) This reverts commit 7e7ad15.
-
[HUDI-3083] Support component data types for flink bulk_insert (apach…
…e#4470) * [HUDI-3083] Support component data types for flink bulk_insert * add nested row type test
-
[HUDI-3124] Bootstrap when timeline have completed instant (apache#4467)
Co-authored-by: yuzhaojing <yuzhaojing@bytedance.com>
-
[HUDI-1951] Add bucket hash index, compatible with the hive bucket (a…
…pache#3173) * [HUDI-2154] Add index key field to HoodieKey * [HUDI-2157] Add the bucket index and its read/write implemention of Spark engine. * revert HUDI-2154 add index key field to HoodieKey * fix all comments and introduce a new tricky way to get index key at runtime support double insert for bucket index * revert spark read optimizer based on bucket index * add the storage layout * index tag, hash function and add ut * fix ut * address partial comments * Code review feedback * add layout config and docs * fix ut * rename hoodie.layout and rebase master Co-authored-by: Vinoth Chandar <vinoth@apache.org>
Commits on Dec 31, 2021
-
[HUDI-3120] Cache compactionPlan in buffer (apache#4463)
Co-authored-by: yuzhaojing <yuzhaojing@bytedance.com>
-
[HUDI-3095] abstract partition filter logic to enable code reuse (apa…
…che#4454) * [HUDI-3095] abstract partition filter logic to enable code reuse * [HUDI-3095] address reviews
-
[HUDI-3107]Fix HiveSyncTool drop partitions using JDBC or hivesql or …
…hms (apache#4453) * constructDropPartitions when drop partitions using jdbc * done * done * code style * code review Co-authored-by: yuezhang <yuezhang@freewheel.tv>
Commits on Jan 1, 2022
Commits on Jan 2, 2022
Commits on Jan 3, 2022
-
[HUDI-3138] Fix broken UT test for TestHiveSyncTool.testDropPartitions (
apache#4493) Co-authored-by: yuezhang <yuezhang@freewheel.tv>
-
[MINOR] Update README.md (apache#4492)
Update Spark 3 build instructions
Commits on Jan 4, 2022
-
[HUDI-3141] Metadata merged log record reader - avoiding NullPointerE…
…xception when records by keys (apache#4505) - HoodieMetadataMergedLogRecordReader#getRecordsByKeys() and its parent class methods are not thread safe. When multiple queries come in for gettting log records by keys, they all operate on the same log record reader instance provided by HoodieBackedTableMetadata#openReadersIfNeeded() and they trip over each other as they clear/put/get the same class memeber records. - The fix is to streamline the mutatation to class member records. Making HoodieMetadataMergedLogRecordReader#getRecordsByKeys() a synchronized method to avoid concurrent log records readers getting into NPE.
-
[HUDI-3147] Add endpoint_url to dynamodb lock provider (apache#4500)
Co-authored-by: Nicolas Paris <nicolas.paris@adevinta.com>
Commits on Jan 5, 2022
-
[HUDI-2966] Closing LogRecordScanner in compactor (apache#4478)
* Closing LogRecordScanner in compactor * Addressing comments