-
Notifications
You must be signed in to change notification settings - Fork 2
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
FixIncrementalSchema #8
FixIncrementalSchema #8
Commits on Nov 12, 2021
-
[HUDI-2718] ExternalSpillableMap payload size re-estimation throws Ar…
…ithmeticException (apache#3955) - ExternalSpillableMap does the payload/value size estimation on the first put to determine when to spill over to disk map. The payload size re-estimation also happens after a minimum threshold of puts. This size re-estimation goes my the current in-memory map size for calculating average payload size and does attempts divide by zero operation when the map is size is empty. Avoiding the ArithmeticException during the payload size re-estimate by checking the map size upfront.
Configuration menu - View commit details
-
Copy full SHA for 9720820 - Browse repository at this point
Copy the full SHA 9720820View commit details -
Configuration menu - View commit details
-
Copy full SHA for 93fd351 - Browse repository at this point
Copy the full SHA 93fd351View commit details
Commits on Nov 13, 2021
-
Configuration menu - View commit details
-
Copy full SHA for 0e8461e - Browse repository at this point
Copy the full SHA 0e8461eView commit details
Commits on Nov 14, 2021
-
[HUDI-2472] Enabling metadata table in TestHoodieIndex and TestMergeO…
…nReadRollbackActionExecutor (apache#3978) - With rollback after first commit support added to metadata table, these test cases are safe to have metadata table turned on.
Configuration menu - View commit details
-
Copy full SHA for 994922a - Browse repository at this point
Copy the full SHA 994922aView commit details -
Configuration menu - View commit details
-
Copy full SHA for c2f9094 - Browse repository at this point
Copy the full SHA c2f9094View commit details -
Configuration menu - View commit details
-
Copy full SHA for 0bb6d8f - Browse repository at this point
Copy the full SHA 0bb6d8fView commit details
Commits on Nov 15, 2021
-
Configuration menu - View commit details
-
Copy full SHA for a14d104 - Browse repository at this point
Copy the full SHA a14d104View commit details -
[HUDI-2758] remove redundant code in the hoodieRealtimeInputFormatUit…
…ls.getRealtimeSplits (apache#3994)
Configuration menu - View commit details
-
Copy full SHA for a0dae41 - Browse repository at this point
Copy the full SHA a0dae41View commit details -
[MINOR] Fix typo in IntervalTreeBasedGlobalIndexFileFilter (apache#3993)
Co-authored-by: 闫杜峰 <yandufeng@sinochem.com>
Configuration menu - View commit details
-
Copy full SHA for 3c43197 - Browse repository at this point
Copy the full SHA 3c43197View commit details -
[HUDI-2744] Fix parsing of metadadata table compaction timestamp when…
… metrics are enabled (apache#3976)
Configuration menu - View commit details
-
Copy full SHA for 53d2d6a - Browse repository at this point
Copy the full SHA 53d2d6aView commit details -
[HUDI-2683] Parallelize deleting archived hoodie commits (apache#3920)
Co-authored-by: yuezhang <yuezhang@freewheel.tv>
Configuration menu - View commit details
-
Copy full SHA for 38b6934 - Browse repository at this point
Copy the full SHA 38b6934View commit details
Commits on Nov 16, 2021
-
[HUDI-2712] Fixing a bug with rollback of partially failed commit whi…
…ch has new partitions (apache#3947)
Configuration menu - View commit details
-
Copy full SHA for bff8769 - Browse repository at this point
Copy the full SHA bff8769View commit details -
Configuration menu - View commit details
-
Copy full SHA for 6f5e661 - Browse repository at this point
Copy the full SHA 6f5e661View commit details
Commits on Nov 17, 2021
-
[MINOR] Fixed checkstyle config to be based off Maven root-dir (requi…
…res Maven >=3.3.1 to work properly); (apache#4009) Updated README
Alexey Kudinkin committedNov 17, 2021 Configuration menu - View commit details
-
Copy full SHA for cbcbec4 - Browse repository at this point
Copy the full SHA cbcbec4View commit details -
Configuration menu - View commit details
-
Copy full SHA for 04eb5fd - Browse repository at this point
Copy the full SHA 04eb5fdView commit details -
[HUDI-2151] Part3 Enabling marker based rollback as default rollback …
…strategy (apache#3950) * Enabling timeline server based markers * Enabling timeline server based markers and marker based rollback * Removing constraint that timeline server can be enabled only for hdfs * Fixing tests
Configuration menu - View commit details
-
Copy full SHA for ce7d233 - Browse repository at this point
Copy the full SHA ce7d233View commit details -
Check --source-avro-schema-path parameter (apache#3987)
Co-authored-by: 0x3E6 <dragon1996>
Configuration menu - View commit details
-
Copy full SHA for aec5d11 - Browse repository at this point
Copy the full SHA aec5d11View commit details -
Configuration menu - View commit details
-
Copy full SHA for 4d884bd - Browse repository at this point
Copy the full SHA 4d884bdView commit details -
[MINOR] Add the Schema for GooseFS to StorageSchemes (apache#3982)
Co-authored-by: lubo <bollu@tencent.com>
Configuration menu - View commit details
-
Copy full SHA for 826414c - Browse repository at this point
Copy the full SHA 826414cView commit details -
[HUDI-2314] Add support for DynamoDb based lock provider (apache#3486)
- Co-authored-by: Wenning Ding <wenningd@amazon.com> - Co-authored-by: Sivabalan Narayanan <n.siva.b@gmail.com>
Configuration menu - View commit details
-
Copy full SHA for 1ee12cf - Browse repository at this point
Copy the full SHA 1ee12cfView commit details -
Configuration menu - View commit details
-
Copy full SHA for f715cf6 - Browse repository at this point
Copy the full SHA f715cf6View commit details -
Configuration menu - View commit details
-
Copy full SHA for 2d3f2a3 - Browse repository at this point
Copy the full SHA 2d3f2a3View commit details
Commits on Nov 18, 2021
-
Configuration menu - View commit details
-
Copy full SHA for 71a2ae0 - Browse repository at this point
Copy the full SHA 71a2ae0View commit details -
Configuration menu - View commit details
-
Copy full SHA for 8772cec - Browse repository at this point
Copy the full SHA 8772cecView commit details -
[HUDI-2362] Add external config file support (apache#3416)
Co-authored-by: Wenning Ding <wenningd@amazon.com>
Configuration menu - View commit details
-
Copy full SHA for 24def0b - Browse repository at this point
Copy the full SHA 24def0bView commit details -
[HUDI-2641] Avoid deleting all inflight commits heartbeats while roll…
…ing back failed writes (apache#3956)
Configuration menu - View commit details
-
Copy full SHA for 4e067ca - Browse repository at this point
Copy the full SHA 4e067caView commit details
Commits on Nov 19, 2021
-
Configuration menu - View commit details
-
Copy full SHA for 7a00f86 - Browse repository at this point
Copy the full SHA 7a00f86View commit details -
Configuration menu - View commit details
-
Copy full SHA for bf00876 - Browse repository at this point
Copy the full SHA bf00876View commit details -
Configuration menu - View commit details
-
Copy full SHA for eba354e - Browse repository at this point
Copy the full SHA eba354eView commit details -
[HUDI-2593] Virtual keys support for metadata table (apache#3968)
- Metadata table today has virtual keys disabled, thereby populating the metafields for each record written out and increasing the overall storage space used. Hereby adding virtual keys support for metadata table so that metafields are disabled for metadata table records. - Adding a custom KeyGenerator for Metadata table so as to not rely on the default Base/SimpleKeyGenerators which currently look for record key and partition field set in the table config. - AbstractHoodieLogRecordReader's version of processing next data block and createHoodieRecord() will be a generic version and making the derived class HoodieMetadataMergedLogRecordReader take care of the special creation of records from explictly passed in partition names.
Configuration menu - View commit details
-
Copy full SHA for 459b342 - Browse repository at this point
Copy the full SHA 459b342View commit details
Commits on Nov 20, 2021
-
[HUDI-2472] Enabling metadata table for TestHoodieMergeOnReadTable an…
…d TestHoodieCompactor (apache#4023)
Configuration menu - View commit details
-
Copy full SHA for c8617d9 - Browse repository at this point
Copy the full SHA c8617d9View commit details -
[HUDI-2796] Metadata table support for Restore action to first commit (…
…apache#4039) - Adding support for the metadata table to restore to first commit and take proper action for the bootstrap on subequent commits.
Configuration menu - View commit details
-
Copy full SHA for 0230d40 - Browse repository at this point
Copy the full SHA 0230d40View commit details -
[HUDI-2242] Add configuration inference logic for few options (apache…
…#3359) Co-authored-by: Wenning Ding <wenningd@amazon.com>
Configuration menu - View commit details
-
Copy full SHA for 3dc6262 - Browse repository at this point
Copy the full SHA 3dc6262View commit details -
Configuration menu - View commit details
-
Copy full SHA for 6cc97cc - Browse repository at this point
Copy the full SHA 6cc97ccView commit details -
[HUDI-2742] Added S3 object filter to support multiple S3EventsHoodie…
…IncrSources single S3 meta table (apache#4025)
Configuration menu - View commit details
-
Copy full SHA for f4b974a - Browse repository at this point
Copy the full SHA f4b974aView commit details -
[HUDI-2795] Add mechanism to safely update,delete and recover table p…
…roperties (apache#4038) * [HUDI-2795] Add mechanism to safely update,delete and recover table properties - Fail safe mechanism, that lets queries succeed off a backup file - Readers who are not upgraded to this version of code will just fail until recovery is done. - Added unit tests that exercises all these scenarios. - Adding CLI for recovery, updation to table command. - [Pending] Add some hash based verfication to ensure any rare partial writes for HDFS * Fixing upgrade/downgrade infrastructure to use new updation method
Configuration menu - View commit details
-
Copy full SHA for ae0c67d - Browse repository at this point
Copy the full SHA ae0c67dView commit details
Commits on Nov 21, 2021
-
Configuration menu - View commit details
-
Copy full SHA for 1a5484d - Browse repository at this point
Copy the full SHA 1a5484dView commit details -
[MINOR] optimize in constructor of inputbatch class (apache#4040)
Co-authored-by: 闫杜峰 <yandufeng@sinochem.com>
Configuration menu - View commit details
-
Copy full SHA for 305d160 - Browse repository at this point
Copy the full SHA 305d160View commit details -
Configuration menu - View commit details
-
Copy full SHA for 74b59a4 - Browse repository at this point
Copy the full SHA 74b59a4View commit details -
Configuration menu - View commit details
-
Copy full SHA for 0411f73 - Browse repository at this point
Copy the full SHA 0411f73View commit details -
Configuration menu - View commit details
-
Copy full SHA for 520538b - Browse repository at this point
Copy the full SHA 520538bView commit details -
[HUDI-1932] Update Hive sync timestamp when change detected (apache#3053
) * Update Hive sync timestamp when change detected Only update the last commit timestamp on the Hive table when the table schema has changed or a partition is created/updated. When using AWS Glue Data Catalog as the metastore for Hive this will ensure that table versions are substantive (including schema and/or partition changes). Prior to this change when a Hive sync is performed without schema or partition changes the table in the Glue Data Catalog would have a new version published with the only change being the timestamp property. https://issues.apache.org/jira/browse/HUDI-1932 * add conditional sync flag * fix testSyncWithoutDiffs * fix HiveSyncConfig Co-authored-by: Raymond Xu <2701446+xushiyan@users.noreply.github.com>
Configuration menu - View commit details
-
Copy full SHA for 887787e - Browse repository at this point
Copy the full SHA 887787eView commit details -
Configuration menu - View commit details
-
Copy full SHA for 2533a9c - Browse repository at this point
Copy the full SHA 2533a9cView commit details
Commits on Nov 22, 2021
-
Configuration menu - View commit details
-
Copy full SHA for 8281cbf - Browse repository at this point
Copy the full SHA 8281cbfView commit details -
[HUDI-1870] Add more Spark CI build tasks (apache#4022)
* [HUDI-1870] Add more Spark CI build tasks - build for spark3.0.x - build for spark-shade-unbundle-avro - fix build failures - delete unnecessary assertion for spark 3.0.x - use AvroConversionUtils#convertAvroSchemaToStructType instead of calling SchemaConverters#toSqlType directly to solve the compilation failures with spark-shade-unbundle-avro (#5) Co-authored-by: Yann <biyan900116@gmail.com>
Configuration menu - View commit details
-
Copy full SHA for 02f7ca2 - Browse repository at this point
Copy the full SHA 02f7ca2View commit details -
[HUDI-2533] New option for hoodieClusteringJob to check, rollback and…
… re-execute the last failed clustering job (apache#3765) * coding finished and need to do uts * add uts * code review * code review Co-authored-by: yuezhang <yuezhang@freewheel.tv>
Configuration menu - View commit details
-
Copy full SHA for a2c91a7 - Browse repository at this point
Copy the full SHA a2c91a7View commit details -
[HUDI-2472] Enabling metadata table for TestHoodieIndex test case (ap…
…ache#4045) - Enablng the metadata table for testSimpleGlobalIndexTagLocationWhenShouldUpdatePartitionPath. This is more of a test issue.
Configuration menu - View commit details
-
Copy full SHA for 7f3b89f - Browse repository at this point
Copy the full SHA 7f3b89fView commit details -
Configuration menu - View commit details
-
Copy full SHA for 8945206 - Browse repository at this point
Copy the full SHA 8945206View commit details -
[HUDI-2559] Converting commit timestamp format to millisecs (apache#4024
Configuration menu - View commit details
-
Copy full SHA for fc9ca6a - Browse repository at this point
Copy the full SHA fc9ca6aView commit details -
Configuration menu - View commit details
-
Copy full SHA for fe57e9b - Browse repository at this point
Copy the full SHA fe57e9bView commit details
Commits on Nov 23, 2021
-
[HUDI-2550] Expand File-Group candidates list for appending for MOR t…
…ables (apache#3986)
Alexey Kudinkin committedNov 23, 2021 Configuration menu - View commit details
-
Copy full SHA for 3bdab01 - Browse repository at this point
Copy the full SHA 3bdab01View commit details -
[HUDI-2737] Use earliest instant by default for async compaction and …
…clustering jobs (apache#3991) Address review comments Fix test failures Co-authored-by: Sagar Sumit <sagarsumit09@gmail.com>
Configuration menu - View commit details
-
Copy full SHA for 772af93 - Browse repository at this point
Copy the full SHA 772af93View commit details -
Configuration menu - View commit details
-
Copy full SHA for 0d1e7ec - Browse repository at this point
Copy the full SHA 0d1e7ecView commit details -
[HUDI-1937] Rollback unfinished replace commit to allow updates (apac…
…he#3869) * [HUDI-1937] Rollback unfinished replace commit to allow updates while clustering * Revert and delete requested replacecommit too * Rollback pending clustering instants transactionally * No double locking and add a config to enable rollback * Update config to be clear about rollback only on conflict
Configuration menu - View commit details
-
Copy full SHA for e22150f - Browse repository at this point
Copy the full SHA e22150fView commit details -
[MINOR] Add more configuration to Kafka setup script (apache#3992)
* [MINOR] Add more configuration to Kafka setup script * Add option to reuse Kafka topic * Minor fixes to README
Configuration menu - View commit details
-
Copy full SHA for 6aa710e - Browse repository at this point
Copy the full SHA 6aa710eView commit details -
[HUDI-2743] Assume path exists and defer fs.exists() in AbstractTable…
…FileSystemView (apache#4002)
Configuration menu - View commit details
-
Copy full SHA for c88c2af - Browse repository at this point
Copy the full SHA c88c2afView commit details -
[HUDI-2778] Optimize statistics collection related codes and add some…
… docs for z-order add fix some bugs (apache#4013) * [HUDI-2778] Optimize statistics collection related codes and add more docs for z-order. * add test code for multi-thread parquet footer read
Configuration menu - View commit details
-
Copy full SHA for 9de9951 - Browse repository at this point
Copy the full SHA 9de9951View commit details -
[HUDI-2409] Using HBase shaded jars in Hudi presto bundle (apache#3623)
* using hbase-shaded-jars-in-hudi-presto-hundle * test * add hudi-common-bundle * code review * code review * code review * code review * test * test Co-authored-by: yuezhang <yuezhang@freewheel.tv>
Configuration menu - View commit details
-
Copy full SHA for 9ed28b1 - Browse repository at this point
Copy the full SHA 9ed28b1View commit details -
[HUDI-2332] Add clustering and compaction in Kafka Connect Sink (apac…
…he#3857) * [HUDI-2332] Add clustering and compaction in Kafka Connect Sink * Disable validation check on instant time for compaction and adjust configs * Add javadocs * Add clustering and compaction config * Fix transaction causing missing records in the target table * Add debugging logs * Fix kafka offset sync in participant * Adjust how clustering and compaction are configured in kafka-connect * Fix clustering strategy * Remove irrelevant changes from other published PRs * Update clustering logic and others * Update README * Fix test failures * Fix indentation * Fix clustering config * Add JavaCustomColumnsSortPartitioner and make async compaction enabled by default * Add test for JavaCustomColumnsSortPartitioner * Add more changes after IDE sync * Update README with clarification * Fix clustering logic after rebasing * Remove unrelated changes
Configuration menu - View commit details
-
Copy full SHA for ca9bfa2 - Browse repository at this point
Copy the full SHA ca9bfa2View commit details -
Configuration menu - View commit details
-
Copy full SHA for 969a5bf - Browse repository at this point
Copy the full SHA 969a5bfView commit details -
[HUDI-2325] Add hive sync support to kafka connect (apache#3660)
Co-authored-by: Rajesh Mahindra <rmahindra@Rajeshs-MacBook-Pro.local>
Configuration menu - View commit details
-
Copy full SHA for fbff079 - Browse repository at this point
Copy the full SHA fbff079View commit details
Commits on Nov 24, 2021
-
[HUDI-2831] Securing usages of
SimpleDateFormat
to be thread-safe (a…Alexey Kudinkin committedNov 24, 2021 Configuration menu - View commit details
-
Copy full SHA for 18cf595 - Browse repository at this point
Copy the full SHA 18cf595View commit details -
Configuration menu - View commit details
-
Copy full SHA for 5078d29 - Browse repository at this point
Copy the full SHA 5078d29View commit details -
Configuration menu - View commit details
-
Copy full SHA for 0cf2f10 - Browse repository at this point
Copy the full SHA 0cf2f10View commit details -
Revert "[HUDI-2799] Fix the classloader of flink write task (apache#4042
)" (apache#4069) This reverts commit 8281cbf.
Configuration menu - View commit details
-
Copy full SHA for 323be33 - Browse repository at this point
Copy the full SHA 323be33View commit details -
Configuration menu - View commit details
-
Copy full SHA for 0bb506f - Browse repository at this point
Copy the full SHA 0bb506fView commit details -
[HUDI-2759] extract HoodieCatalogTable to coordinate spark catalog ta…
…ble and hoodie table (apache#3998)
Configuration menu - View commit details
-
Copy full SHA for a234833 - Browse repository at this point
Copy the full SHA a234833View commit details -
Configuration menu - View commit details
-
Copy full SHA for 9af219b - Browse repository at this point
Copy the full SHA 9af219bView commit details -
[HUDI-2671] Fix kafka offset handling in Kafka Connect protocol (apac…
…he#4021) Co-authored-by: Rajesh Mahindra <rmahindra@Rajeshs-MacBook-Pro.local>
Configuration menu - View commit details
-
Copy full SHA for 90f2ea2 - Browse repository at this point
Copy the full SHA 90f2ea2View commit details -
[HUDI-2443] Hudi KVComparator for all HFile writer usages (apache#3889)
* [HUDI-2443] Hudi KVComparator for all HFile writer usages - Hudi relies on custom class shading for Hbase's KeyValue.KVComparator to avoid versioning and class loading issues. There are few places which are still using the Hbase's comparator class directly and version upgrades would make them obsolete. Refactoring the HoodieKVComparator and making all HFile writer creation using the same shaded class. * [HUDI-2443] Hudi KVComparator for all HFile writer usages - Moving HoodieKVComparator from common.bootstrap.index to common.util * [HUDI-2443] Hudi KVComparator for all HFile writer usages - Retaining the old HoodieKVComparatorV2 for boostrap case. Adding the new comparator as HoodieKVComparatorV2 to differentiate from the old one. * [HUDI-2443] Hudi KVComparator for all HFile writer usages - Renamed HoodieKVComparatorV2 to HoodieMetadataKVComparator and moved it under the package org.apache.hudi.metadata. * Make comparator classname configurable * Revert new config and address other review comments Co-authored-by: Sagar Sumit <sagarsumit09@gmail.com>
Configuration menu - View commit details
-
Copy full SHA for 973f78f - Browse repository at this point
Copy the full SHA 973f78fView commit details -
[HUDI-2788] Fixing issues w/ Z-order Layout Optimization (apache#4026)
* Simplyfying, tidying up * Fixed packaging for `TestOptimizeTable` * Cleaned up `HoodiFileIndex` file filtering seq; Removed optimization manually reading Parquet table circumventing Spark * Refactored `DataSkippingUtils`: - Fixed checks to validate all statistics cols are present - Fixed some predicates being constructed incorrectly - Rewrote comments for easier comprehension, added more notes - Tidying up * Tidying up tests * `lint` * Fixing compilation * `TestOptimizeTable` > `TestTableLayoutOptimization`; Added assertions to test data skipping paths * Fixed tests to properly hit data-skipping path * Fixed pruned files candidates lookup seq to conservatively included all non-indexed files * Added java-doc * Fixed compilation
Alexey Kudinkin committedNov 24, 2021 Configuration menu - View commit details
-
Copy full SHA for 60b23b9 - Browse repository at this point
Copy the full SHA 60b23b9View commit details -
[HUDI-2766] Cluster update strategy should not be fenced by write con…
…fig (apache#4093) Fix pending clustering rollback test
Configuration menu - View commit details
-
Copy full SHA for ff94d92 - Browse repository at this point
Copy the full SHA ff94d92View commit details -
[HUDI-2793] Fixing deltastreamer checkpoint fetch/copy over (apache#4034
Configuration menu - View commit details
-
Copy full SHA for 435ea15 - Browse repository at this point
Copy the full SHA 435ea15View commit details
Commits on Nov 25, 2021
-
[HUDI-2853] Add JMX deps in hudi utilities and kafka connect bundles (a…
…pache#4108) Co-authored-by: Rajesh Mahindra <rmahindra@Rajeshs-MacBook-Pro.local>
Configuration menu - View commit details
-
Copy full SHA for 7286b56 - Browse repository at this point
Copy the full SHA 7286b56View commit details -
[HUDI-2844][CLI] Fixing archived Timeline crashing if timeline contai…
…ns REPLACE_COMMIT (apache#4091)
Alexey Kudinkin committedNov 25, 2021 Configuration menu - View commit details
-
Copy full SHA for 5129773 - Browse repository at this point
Copy the full SHA 5129773View commit details -
Configuration menu - View commit details
-
Copy full SHA for bef373f - Browse repository at this point
Copy the full SHA bef373fView commit details -
[HUDI-1290] [RFC-39] Deltastreamer avro source for Debezium CDC (apac…
…he#4048) * Add RFC entry for deltastreamer source for debezium * Add RFC for debezium source * Add RFC for debezium source * Add RFC for debezium source * fix hyperlink issue and rebase * Update progress Co-authored-by: Rajesh Mahindra <rmahindra@Rajeshs-MacBook-Pro.local>
Configuration menu - View commit details
-
Copy full SHA for abc0175 - Browse repository at this point
Copy the full SHA abc0175View commit details -
[HUDI-1290] Add Debezium Source for deltastreamer (apache#4063)
* add source for postgres debezium * Add tests for debezium payload * Fix test * Fix test * Add tests for debezium source * Add tests for debezium source * Fix schema for debezium * Fix checkstyle issues * Fix config issue for schema registry * Add mysql source for debezium * Fix checkstyle issues an tests * Improve code for merging toasted values * Improve code for merging toasted values Co-authored-by: Rajesh Mahindra <rmahindra@Rajeshs-MacBook-Pro.local>
Configuration menu - View commit details
-
Copy full SHA for 83f8ed2 - Browse repository at this point
Copy the full SHA 83f8ed2View commit details -
[HUDI-2792] Configure metadata payload consistency check (apache#4035)
- Relax metadata payload consistency check to consider spark task failures with spurious deletes
Configuration menu - View commit details
-
Copy full SHA for a9bd208 - Browse repository at this point
Copy the full SHA a9bd208View commit details -
[HUDI-2855] Change the default value of 'PAYLOAD_CLASS_NAME' to 'Defa…
…ultHoodieRecordPayload' (apache#4115)
Configuration menu - View commit details
-
Copy full SHA for 88067f5 - Browse repository at this point
Copy the full SHA 88067f5View commit details -
[HUDI-2480] FileSlice after pending compaction-requested instant-time… (
apache#3703) * [HUDI-2480] FileSlice after pending compaction-requested instant-time is ignored by MOR snapshot reader * include file slice after a pending compaction for spark reader Co-authored-by: garyli1019 <yanjia.gary.li@gmail.com>
Configuration menu - View commit details
-
Copy full SHA for a2eb2b0 - Browse repository at this point
Copy the full SHA a2eb2b0View commit details -
Configuration menu - View commit details
-
Copy full SHA for 264e1ce - Browse repository at this point
Copy the full SHA 264e1ceView commit details -
[HUDI-2800] Remove rdd.isEmpty() validation to prevent CreateHandle b…
…eing called twice (apache#4121)
Configuration menu - View commit details
-
Copy full SHA for b972aa5 - Browse repository at this point
Copy the full SHA b972aa5View commit details -
[HUDI-2794] Guarding table service commits within a single lock to co…
…mmit to both data table and metadata table (apache#4037) * Fixing a single lock to commit table services across metadata table and data table * Addressing comments * rebasing with master
Configuration menu - View commit details
-
Copy full SHA for 7bb90e8 - Browse repository at this point
Copy the full SHA 7bb90e8View commit details -
[HUDI-2671] Making error -> warn logs from timeline server with concu…
…rrent writers for inconsistent state (apache#4088) * Making error -> warn logs from timeline server with concurrent writers for inconsistent state * Fixing bad request response exception for timeline out of sync * Addressing feedback. removed write concurrency mode depedency
Configuration menu - View commit details
-
Copy full SHA for f692078 - Browse repository at this point
Copy the full SHA f692078View commit details -
Configuration menu - View commit details
-
Copy full SHA for 6a0f079 - Browse repository at this point
Copy the full SHA 6a0f079View commit details -
Configuration menu - View commit details
-
Copy full SHA for 8e13793 - Browse repository at this point
Copy the full SHA 8e13793View commit details -
Configuration menu - View commit details
-
Copy full SHA for e0125a7 - Browse repository at this point
Copy the full SHA e0125a7View commit details -
[HUDI-2840] Fixed DeltaStreaemer to properly respect configuration pa…
…ssed t/h properties file (apache#4090) * Rebased `DFSPropertiesConfiguration` to access Hadoop config in liue of FS to avoid confusion * Fixed `readConfig` to take Hadoop's `Configuration` instead of FS; Fixing usages * Added test for local FS access * Rebase to use `FSUtils.getFs` * Combine properties provided as a file along w/ overrides provided from the CLI * Added helper utilities to `HoodieClusteringConfig`; Make sure corresponding config methods fallback to defaults; * Fixed DeltaStreamer usage to respect properly combined configuration; Abstracted `HoodieClusteringConfig.from` convenience utility to init Clustering config from `Properties` * Tidying up * `lint` * Reverting changes to `HoodieWriteConfig` * Tdiying up * Fixed incorrect merge of the props * Converted `HoodieConfig` to wrap around `Properties` into `TypedProperties` * Fixed compilation * Fixed compilation
Alexey Kudinkin committedNov 25, 2021 Configuration menu - View commit details
-
Copy full SHA for 6f5d8d0 - Browse repository at this point
Copy the full SHA 6f5d8d0View commit details -
Configuration menu - View commit details
-
Copy full SHA for 8340ccb - Browse repository at this point
Copy the full SHA 8340ccbView commit details
Commits on Nov 26, 2021
-
Configuration menu - View commit details
-
Copy full SHA for 38585e4 - Browse repository at this point
Copy the full SHA 38585e4View commit details -
[MINOR] Include hudi-aws in flink bundle jar (apache#4127)
HUDI-2801 makes this jar as required.
Configuration menu - View commit details
-
Copy full SHA for f5da9b5 - Browse repository at this point
Copy the full SHA f5da9b5View commit details -
[HUDI-2852] Table metadata returns empty for non-exist partition (apa…
…che#4117) * [HUDI-2852] Table metadata returns empty for non-exist partition * add unit test * fix code checkstyle Co-authored-by: wangminchao <wangminchao@asinking.com>
Configuration menu - View commit details
-
Copy full SHA for e554c7f - Browse repository at this point
Copy the full SHA e554c7fView commit details -
Configuration menu - View commit details
-
Copy full SHA for e9efbdb - Browse repository at this point
Copy the full SHA e9efbdbView commit details -
[HUDI-2850] Fixing Clustering CLI - schedule and run command fixes to…
… avoid NumberFormatException (apache#4101)
Configuration menu - View commit details
-
Copy full SHA for 3d75aca - Browse repository at this point
Copy the full SHA 3d75acaView commit details -
[HUDI-2814] Addressing issues w/ Z-order Layout Optimization (apache#…
…4060) * `ZCurveOptimizeHelper` > `ZOrderingIndexHelper`; Moved Z-index helper under `hudi.index.zorder` package * Tidying up `ZOrderingIndexHelper` * Fixing compilation * Fixed index new/original table merging sequence to always prefer values from new index; Cleaned up `HoodieSparkUtils` * Added test for `mergeIndexSql` * Abstracted Z-index name composition w/in `ZOrderingIndexHelper`; * Fixed `DataSkippingUtils` to interrupt prunning in case data filter contains non-indexed column reference * Properly handle exceptions origination during pruning in `HoodieFileIndex` * Make sure no errors are logged upon encountering `AnalysisException` * Cleaned up Z-index updating sequence; Tidying up comments, java-docs; * Fixed Z-index to properly handle changes of the list of clustered columns * Tidying up * `lint` * Suppressing `JavaDocStyle` first sentence check * Fixed compilation * Fixing incorrect `DecimalType` conversion * Refactored test `TestTableLayoutOptimization` - Added Z-index table composition test (against fixtures) - Separated out GC test; Tidying up * Fixed tests re-shuffling column order for Z-Index table `DataFrame` to align w/ the one by one loaded from JSON * Scaffolded `DataTypeUtils` to do basic checks of Spark types; Added proper compatibility checking b/w old/new index-tables * Added test for Z-index tables merging * Fixed import being shaded by creating internal `hudi.util` package * Fixed packaging for `TestOptimizeTable` * Revised `updateMetadataIndex` seq to provide Z-index updating process w/ source table schema * Make sure existing Z-index table schema is sync'd to source table's one * Fixed shaded refs * Fixed tests * Fixed type conversion of Parquet provided metadata values into Spark expected schemas * Fixed `composeIndexSchema` utility to propose proper schema * Added more tests for Z-index: - Checking that Z-index table is built correctly - Checking that Z-index tables are merged correctly (during update) * Fixing source table * Fixing tests to read from Parquet w/ proper schema * Refactored `ParquetUtils` utility reading stats from Parquet footers * Fixed incorrect handling of Decimals extracted from Parquet footers * Worked around issues in javac failign to compile stream's collection * Fixed handling of `Date` type * Fixed handling of `DateType` to be parsed as `LocalDate` * Updated fixture; Make sure test loads Z-index fixture using proper schema * Removed superfluous scheme adjusting when reading from Parquet, since Spark is actually able to perfectly restore schema (given Parquet was previously written by Spark as well) * Fixing race-condition in Parquet's `DateStringifier` trying to share `SimpleDataFormat` object which is inherently not thread-safe * Tidying up * Make sure schema is used upon reading to validate input files are in the appropriate format; Tidying up; * Worked around javac (1.8) inability to infer expression type properly * Updated fixtures; Tidying up * Fixing compilation after rebase * Assert clustering have in Z-order layout optimization testing * Tidying up exception messages * XXX * Added test validating Z-index lookup filter correctness * Added more test-cases; Tidying up * Added tests for string expressions * Fixed incorrect Z-index filter lookup translations * Added more test-cases * Added proper handling on complex negations of AND/OR expressions by pushing NOT operator down into inner expressions for appropriate handling * Added `-target:jvm-1.8` for `hudi-spark` module * Adding more tests * Added tests for non-indexed columns * Properly handle non-indexed columns by falling back to a re-write of containing expression as `TrueLiteral` instead * Fixed tests * Removing the parquet test files and disabling corresponding tests Co-authored-by: Vinoth Chandar <vinoth@apache.org>
Configuration menu - View commit details
-
Copy full SHA for 5755ff2 - Browse repository at this point
Copy the full SHA 5755ff2View commit details -
Configuration menu - View commit details
-
Copy full SHA for a88691f - Browse repository at this point
Copy the full SHA a88691fView commit details -
Configuration menu - View commit details
-
Copy full SHA for f8e0176 - Browse repository at this point
Copy the full SHA f8e0176View commit details -
[HUDI-2767] Enabling timeline-server-based marker as default (apache#…
…4112) - Changes the default config of marker type (HoodieWriteConfig.MARKERS_TYPE or hoodie.write.markers.type) from DIRECT to TIMELINE_SERVER_BASED for Spark Engine. - Adds engine-specific marker type configs: Spark -> TIMELINE_SERVER_BASED, Flink -> DIRECT, Java -> DIRECT. - Uses DIRECT markers as well for Spark structured streaming due to timeline server only available for the first mini-batch. - Fixes the marker creation method for non-partitioned table in TimelineServerBasedWriteMarkers. - Adds the fallback to direct markers even when TIMELINE_SERVER_BASED is configured, in WriteMarkersFactory: when HDFS is used, or embedded timeline server is disabled, the fallback to direct markers happens. - Fixes the closing of timeline service. - Fixes tests that depend on markers, mainly by starting the timeline service for each test.
Configuration menu - View commit details
-
Copy full SHA for d1e83e4 - Browse repository at this point
Copy the full SHA d1e83e4View commit details -
[HUDI-2845] Metadata CLI - files/partition file listing fix and new v…
…alidate option (apache#4092) - Co-authored-by: Sivabalan Narayanan <n.siva.b@gmail.com>
Configuration menu - View commit details
-
Copy full SHA for 445208a - Browse repository at this point
Copy the full SHA 445208aView commit details -
Configuration menu - View commit details
-
Copy full SHA for 8402cac - Browse repository at this point
Copy the full SHA 8402cacView commit details -
[HUDI-2864] Fix README and scripts with current limitations of hive s…
…ync (apache#4129) * Fix README with current limitations of hive sync * Fix README with current limitations of hive sync * Fix dep issue * Fix Copy on Write flow Co-authored-by: Rajesh Mahindra <rmahindra@Rajeshs-MacBook-Pro.local>
Configuration menu - View commit details
-
Copy full SHA for 9028e6e - Browse repository at this point
Copy the full SHA 9028e6eView commit details -
[HUDI-2856] Bit cask disk map delete modified (apache#4116)
* modified BitCaskDiskMap_close_function * change iterators location to finally * Update BitCaskDiskMap.java
Configuration menu - View commit details
-
Copy full SHA for 257a6a7 - Browse repository at this point
Copy the full SHA 257a6a7View commit details
Commits on Nov 27, 2021
-
[MINOR] Follow ups from HUDI-2861 (re-use same rollback instant for f…
…ailed rollback) (apache#4133)
Configuration menu - View commit details
-
Copy full SHA for 9c059ef - Browse repository at this point
Copy the full SHA 9c059efView commit details -
[HUDI-2868] Fix skipped HoodieSparkSqlWriterSuite (apache#4125)
- Co-authored-by: Yann Byron <biyan900116@gmail.com>
Configuration menu - View commit details
-
Copy full SHA for 3a8d64e - Browse repository at this point
Copy the full SHA 3a8d64eView commit details -
[HUDI-2475] [HUDI-2862] Metadata table creation and avoid bootstrappi…
…ng race for write client & add locking for upgrade (apache#4114) Co-authored-by: Sivabalan Narayanan <n.siva.b@gmail.com>
Configuration menu - View commit details
-
Copy full SHA for 2c7656c - Browse repository at this point
Copy the full SHA 2c7656cView commit details -
[HUDI-2102] Support hilbert curve for hudi (apache#3952)
Co-authored-by: Y Ethan Guo <ethan.guoyihua@gmail.com>
Configuration menu - View commit details
-
Copy full SHA for 780a2ac - Browse repository at this point
Copy the full SHA 780a2acView commit details -
Configuration menu - View commit details
-
Copy full SHA for a1d0ff4 - Browse repository at this point
Copy the full SHA a1d0ff4View commit details
Commits on Nov 28, 2021
-
Configuration menu - View commit details
-
Copy full SHA for eca1693 - Browse repository at this point
Copy the full SHA eca1693View commit details
Commits on Nov 29, 2021
-
Configuration menu - View commit details
-
Copy full SHA for 52aae36 - Browse repository at this point
Copy the full SHA 52aae36View commit details -
Configuration menu - View commit details
-
Copy full SHA for 38e75ea - Browse repository at this point
Copy the full SHA 38e75eaView commit details -
Configuration menu - View commit details
-
Copy full SHA for 536af4b - Browse repository at this point
Copy the full SHA 536af4bView commit details
Commits on Nov 30, 2021
-
Configuration menu - View commit details
-
Copy full SHA for 3433f00 - Browse repository at this point
Copy the full SHA 3433f00View commit details -
Configuration menu - View commit details
-
Copy full SHA for a398aad - Browse repository at this point
Copy the full SHA a398aadView commit details -
Configuration menu - View commit details
-
Copy full SHA for ea009b5 - Browse repository at this point
Copy the full SHA ea009b5View commit details
Commits on Dec 1, 2021
-
Revert "[HUDI-2855] Change the default value of 'PAYLOAD_CLASS_NAME' …
…to 'DefaultHoodieRecordPayload' (apache#4115)" (apache#4169) This reverts commit 88067f5.
Alexey Kudinkin committedDec 1, 2021 Configuration menu - View commit details
-
Copy full SHA for 24380c2 - Browse repository at this point
Copy the full SHA 24380c2View commit details -
Revert "[HUDI-2856] Bit cask disk map delete modified (apache#4116)" (a…
…pache#4171) This reverts commit 257a6a7.
Configuration menu - View commit details
-
Copy full SHA for 9b254b6 - Browse repository at this point
Copy the full SHA 9b254b6View commit details -
[HUDI-2880] Fixing loading of props from default dir (apache#4167)
* Fixing loading of props from default dir * addressing comments
Configuration menu - View commit details
-
Copy full SHA for f4c25ba - Browse repository at this point
Copy the full SHA f4c25baView commit details
Commits on Dec 2, 2021
-
[HUDI-2881] Compact the file group with larger log files to reduce wr…
…ite amplification (apache#4152)
Configuration menu - View commit details
-
Copy full SHA for 5284730 - Browse repository at this point
Copy the full SHA 5284730View commit details -
Fixed partitions produced by layout optimization in case order-by key…
… is composed of a single column (apache#4183)
Alexey Kudinkin committedDec 2, 2021 Configuration menu - View commit details
-
Copy full SHA for 772f5ca - Browse repository at this point
Copy the full SHA 772f5caView commit details -
[MINOR] Fix the wrong usage of timestamp length variable bug (apache#…
…4179) Signed-off-by: zzzhy <candle_1667@163.com>
Configuration menu - View commit details
-
Copy full SHA for 61a03bc - Browse repository at this point
Copy the full SHA 61a03bcView commit details -
[HUDI-2904] Fix metadata table archival overstepping between regular …
…writers and table services (apache#4186) - Co-authored-by: Rajesh Mahindra <rmahindra@Rajeshs-MacBook-Pro.local> - Co-authored-by: Sivabalan Narayanan <n.siva.b@gmail.com>
Configuration menu - View commit details
-
Copy full SHA for 91d2e61 - Browse repository at this point
Copy the full SHA 91d2e61View commit details
Commits on Dec 3, 2021
-
Configuration menu - View commit details
-
Copy full SHA for 934fe54 - Browse repository at this point
Copy the full SHA 934fe54View commit details -
Configuration menu - View commit details
-
Copy full SHA for f74b3d1 - Browse repository at this point
Copy the full SHA f74b3d1View commit details -
Configuration menu - View commit details
-
Copy full SHA for 0699521 - Browse repository at this point
Copy the full SHA 0699521View commit details -
Configuration menu - View commit details
-
Copy full SHA for ca42724 - Browse repository at this point
Copy the full SHA ca42724View commit details -
[HUDI-2902] Fixing populate meta fields with Hfile writers and Disabl…
…ing virtual keys by default for metadata table (apache#4194)
Configuration menu - View commit details
-
Copy full SHA for e483f7c - Browse repository at this point
Copy the full SHA e483f7cView commit details -
[HUDI-2911] Removing default value for
PARTITIONPATH_FIELD_NAME
res……ulting in incorrect `KeyGenerator` configuration (apache#4195)
Alexey Kudinkin committedDec 3, 2021 Configuration menu - View commit details
-
Copy full SHA for bed7f98 - Browse repository at this point
Copy the full SHA bed7f98View commit details -
Revert "[HUDI-2495] Resolve inconsistent key generation for timestamp…
… types by GenericRecord and Row (apache#3944)" (apache#4201)
Configuration menu - View commit details
-
Copy full SHA for 2f96f43 - Browse repository at this point
Copy the full SHA 2f96f43View commit details -
[HUDI-2894][HUDI-2905] Metadata table - avoiding key lookup failures …
…on base files over S3 (apache#4185) - Fetching partition files or all partitions from the metadata table is failing when run over S3. Metadata table uses HFile format for the base files and the record lookup uses HFile.Reader and HFileScanner interfaces to get records by partition keys. When the backing storage is S3, this record lookup from HFiles is failing with IOException, in turn failing the caller commit/update operations. - Metadata table looks up HFile records with positional read enabled so as to perform better for random lookups. But this positional read key lookup is returning with partial read sizes over S3 leading to HFile scanner throwing IOException. This doesn't happen over HDFS. Metadata table though uses the HFile for random key lookups, the positional read is not mandatory as we sort the keys when doing a lookup for multiple keys. - The fix is to disable HFile positional read for all HFile scanner based key lookups.
Configuration menu - View commit details
-
Copy full SHA for 383d5ed - Browse repository at this point
Copy the full SHA 383d5edView commit details
Commits on Dec 4, 2021
-
Revert "[HUDI-2489]Tuning HoodieROTablePathFilter by caching hoodieTa…
…bleFileSystemView, aiming to reduce unnecessary list/get requests" Co-authored-by: yuezhang <yuezhang@freewheel.tv>
Configuration menu - View commit details
-
Copy full SHA for 5616830 - Browse repository at this point
Copy the full SHA 5616830View commit details -
[MINOR] Mitigate CI jobs timeout issues (apache#4173)
* skip shutdown zookeeper in `@AfterAll` in TestHBaseIndex * rebalance CI tests
Configuration menu - View commit details
-
Copy full SHA for a799fae - Browse repository at this point
Copy the full SHA a799faeView commit details -
Configuration menu - View commit details
-
Copy full SHA for 0fd6b2d - Browse repository at this point
Copy the full SHA 0fd6b2dView commit details -
[HUDI-2890] Kafka Connect: Fix failed writes and avoid table service …
…concurrent operations (apache#4211) * Fix kafka connect readme * Fix handling of errors in write records for kafka connect * By default, ensure we skip error records and keep the pipeline alive * Fix indentation Co-authored-by: Rajesh Mahindra <rmahindra@Rajeshs-MacBook-Pro.local>
Configuration menu - View commit details
-
Copy full SHA for 94f45e9 - Browse repository at this point
Copy the full SHA 94f45e9View commit details -
[HUDI-2923] Fixing metadata table reader when metadata compaction is …
…inflight (apache#4206) * [HUDI-2923] Fixing metadata table reader when metadata compaction is inflight * Fixing retry of pending compaction in metadata table and enhancing tests
Configuration menu - View commit details
-
Copy full SHA for 1d4fb82 - Browse repository at this point
Copy the full SHA 1d4fb82View commit details -
Configuration menu - View commit details
-
Copy full SHA for 568181a - Browse repository at this point
Copy the full SHA 568181aView commit details -
[HUDI-2935] Remove special casing of clustering in deltastreamer chec…
…kpoint retrival (apache#4216) - We now seek backwards to find the checkpoint - No need to return empty anymore
Configuration menu - View commit details
-
Copy full SHA for 36b69d8 - Browse repository at this point
Copy the full SHA 36b69d8View commit details
Commits on Dec 5, 2021
-
[HUDI-2877] Support flink catalog to help user use flink table conven…
…iently (apache#4153) * [HUDI-2877] Support flink catalog to help user use flink table conveniently * Fix comment * fix comment2
Configuration menu - View commit details
-
Copy full SHA for a8fb696 - Browse repository at this point
Copy the full SHA a8fb696View commit details -
[HUDI-2937] Introduce a pulsar implementation of hoodie write commit … (
apache#4217) * [HUDI-2937] Introduce a pulsar implementation of hoodie write commit callback * [HUDI-2937] Introduce a pulsar implementation of hoodie write commit callback * [HUDI-2937] Introduce a pulsar implementation of hoodie write commit callback * [HUDI-2937] Introduce a pulsar implementation of hoodie write commit callback * [HUDI-2937] Introduce a pulsar implementation of hoodie write commit callback * [HUDI-2937] Introduce a pulsar implementation of hoodie write commit callback * [HUDI-2937] Introduce a pulsar implementation of hoodie write commit callback
Configuration menu - View commit details
-
Copy full SHA for 63b1560 - Browse repository at this point
Copy the full SHA 63b1560View commit details -
[HUDI-2418] Support HiveSchemaProvider (apache#3671)
Co-authored-by: jian.feng <fengjian428@gmial.com>
Configuration menu - View commit details
-
Copy full SHA for 734c9f5 - Browse repository at this point
Copy the full SHA 734c9f5View commit details
Commits on Dec 6, 2021
-
Configuration menu - View commit details
-
Copy full SHA for f0e46bf - Browse repository at this point
Copy the full SHA f0e46bfView commit details -
[HUDI-2900] Fix corrupt block end position (apache#4181)
* [HUDI-2900] Fix corrupt block end position * add a test
Configuration menu - View commit details
-
Copy full SHA for 84b531a - Browse repository at this point
Copy the full SHA 84b531aView commit details -
[HUDI-2876] for hive/presto hudi should remove the temp file which cr…
…eated by HoodieMergedLogRecordSanner when the query finished. (apache#4139)
Configuration menu - View commit details
-
Copy full SHA for 57c4bf8 - Browse repository at this point
Copy the full SHA 57c4bf8View commit details -
Configuration menu - View commit details
-
Copy full SHA for 2d66451 - Browse repository at this point
Copy the full SHA 2d66451View commit details -
[MINOR] Use maven-shade-plugin version for hudi-timeline-server-bundl…
…e from main pom.xml (apache#4209) Co-authored-by: Wenning Ding <wenningd@amazon.com>
Configuration menu - View commit details
-
Copy full SHA for 4a437f2 - Browse repository at this point
Copy the full SHA 4a437f2View commit details
Commits on Dec 7, 2021
-
[MINOR] Remove redundant and conflicting spark-hive dependency (apach…
…e#4228) Disable TestHiveSchemaProvider
Configuration menu - View commit details
-
Copy full SHA for 6dab307 - Browse repository at this point
Copy the full SHA 6dab307View commit details -
Configuration menu - View commit details
-
Copy full SHA for e8473b9 - Browse repository at this point
Copy the full SHA e8473b9View commit details
Commits on Dec 8, 2021
-
Configuration menu - View commit details
-
Copy full SHA for c9e18d1 - Browse repository at this point
Copy the full SHA c9e18d1View commit details -
Configuration menu - View commit details
-
Copy full SHA for c56d93e - Browse repository at this point
Copy the full SHA c56d93eView commit details -
[HUDI-2832][RFC-41] Proposal to integrate Hudi on Snowflake platform (a…
…pache#4074) * [HUDI-2832][RFC-40] Proposal to integrate Hudi on Snowflake platform * rebased and addressed review comments
Vinoth Govindarajan committedDec 8, 2021 Configuration menu - View commit details
-
Copy full SHA for 082faa3 - Browse repository at this point
Copy the full SHA 082faa3View commit details
Commits on Dec 9, 2021
-
Configuration menu - View commit details
-
Copy full SHA for 7c3f077 - Browse repository at this point
Copy the full SHA 7c3f077View commit details -
Configuration menu - View commit details
-
Copy full SHA for bd08470 - Browse repository at this point
Copy the full SHA bd08470View commit details -
[HUDI-2665] Fix overflow of huge log file in HoodieLogFormatWriter (a…
…pache#3912) Co-authored-by: guanziyue.gzy <guanziyue.gzy@bytedance.com>
Configuration menu - View commit details
-
Copy full SHA for 9c8ad0f - Browse repository at this point
Copy the full SHA 9c8ad0fView commit details -
Configuration menu - View commit details
-
Copy full SHA for 5ac9ce7 - Browse repository at this point
Copy the full SHA 5ac9ce7View commit details -
[HUDI-2779] Cache BaseDir if HudiTableNotFound Exception thrown (apac…
RexAn committedDec 9, 2021 Configuration menu - View commit details
-
Copy full SHA for f612a20 - Browse repository at this point
Copy the full SHA f612a20View commit details -
[HUDI-2966] Add TaskCompletionListener for HoodieMergeOnReadRDD to cl…
…ose logScaner when the query finished. (apache#4265) * [HUDI-2966] Add TaskCompletionListener for HoodieMergeOnReadRDD to close logScaner when the query finished.
Configuration menu - View commit details
-
Copy full SHA for 68f8597 - Browse repository at this point
Copy the full SHA 68f8597View commit details -
Configuration menu - View commit details
-
Copy full SHA for 3fb2f97 - Browse repository at this point
Copy the full SHA 3fb2f97View commit details
Commits on Dec 10, 2021
-
Configuration menu - View commit details
-
Copy full SHA for 8321d20 - Browse repository at this point
Copy the full SHA 8321d20View commit details -
Revert "Claiming RFC for data skipping index for updated version (apa…
…che#4271)" (apache#4272) This reverts commit 8321d20.
Configuration menu - View commit details
-
Copy full SHA for ea154bc - Browse repository at this point
Copy the full SHA ea154bcView commit details -
Configuration menu - View commit details
-
Copy full SHA for 456d74c - Browse repository at this point
Copy the full SHA 456d74cView commit details -
Configuration menu - View commit details
-
Copy full SHA for c7473a7 - Browse repository at this point
Copy the full SHA c7473a7View commit details -
Configuration menu - View commit details
-
Copy full SHA for f194566 - Browse repository at this point
Copy the full SHA f194566View commit details -
Configuration menu - View commit details
-
Copy full SHA for be36826 - Browse repository at this point
Copy the full SHA be36826View commit details -
[HUDI-2912] Fix CompactionPlanOperator typo (apache#4187)
Co-authored-by: yuzhaojing <yuzhaojing@bytedance.com>
Configuration menu - View commit details
-
Copy full SHA for 3ad9b12 - Browse repository at this point
Copy the full SHA 3ad9b12View commit details -
Configuration menu - View commit details
-
Copy full SHA for 3ce0526 - Browse repository at this point
Copy the full SHA 3ce0526View commit details -
[HUDI-2892][BUG] Pending Clustering may stain the ActiveTimeLine and …
…lead to incomplete query results (apache#4172) Co-authored-by: yuezhang <yuezhang@freewheel.tv>
Configuration menu - View commit details
-
Copy full SHA for 3ba2909 - Browse repository at this point
Copy the full SHA 3ba2909View commit details -
Configuration menu - View commit details
-
Copy full SHA for 72901a3 - Browse repository at this point
Copy the full SHA 72901a3View commit details -
[HUDI-2814] Make Z-index more generic Column-Stats Index (apache#4106)
Alexey Kudinkin committedDec 10, 2021 Configuration menu - View commit details
-
Copy full SHA for 2d864f7 - Browse repository at this point
Copy the full SHA 2d864f7View commit details
Commits on Dec 11, 2021
-
Configuration menu - View commit details
-
Copy full SHA for c48a2a1 - Browse repository at this point
Copy the full SHA c48a2a1View commit details -
[HUDI-2974] Make the prefix for metrics name configurable (apache#4274)
Co-authored-by: Rajesh Mahindra <rmahindra@Rajeshs-MacBook-Pro.local>
Configuration menu - View commit details
-
Copy full SHA for 9797fdf - Browse repository at this point
Copy the full SHA 9797fdfView commit details -
Configuration menu - View commit details
-
Copy full SHA for 9bdcee0 - Browse repository at this point
Copy the full SHA 9bdcee0View commit details -
Configuration menu - View commit details
-
Copy full SHA for 2dcb3f0 - Browse repository at this point
Copy the full SHA 2dcb3f0View commit details -
Configuration menu - View commit details
-
Copy full SHA for b5f05fd - Browse repository at this point
Copy the full SHA b5f05fdView commit details -
Configuration menu - View commit details
-
Copy full SHA for 8dd0444 - Browse repository at this point
Copy the full SHA 8dd0444View commit details
Commits on Dec 12, 2021
-
[HUDI-2946] Upgrade maven plugins to be compatible with higher Java v…
…ersions (apache#4232) Co-authored-by: Wenning Ding <wenningd@amazon.com>
Configuration menu - View commit details
-
Copy full SHA for 15444c9 - Browse repository at this point
Copy the full SHA 15444c9View commit details -
Configuration menu - View commit details
-
Copy full SHA for b22c2c6 - Browse repository at this point
Copy the full SHA b22c2c6View commit details
Commits on Dec 13, 2021
-
Configuration menu - View commit details
-
Copy full SHA for dd96129 - Browse repository at this point
Copy the full SHA dd96129View commit details -
[HUDI-2994] Add judgement to existed partitionPath in the catch code …
…block for HU… (apache#4294) * [HUDI-2994] Add judgement to existed partition path in the catch code block for HUDI-2743 Co-authored-by: wangminchao <wangminchao@asinking.com>
Configuration menu - View commit details
-
Copy full SHA for 46de25d - Browse repository at this point
Copy the full SHA 46de25dView commit details
Commits on Dec 14, 2021
-
[HUDI-2996] Flink streaming reader 'skip_compaction' option does not …
…work (apache#4304) close apache#4304
Configuration menu - View commit details
-
Copy full SHA for 29bc5fd - Browse repository at this point
Copy the full SHA 29bc5fdView commit details -
Configuration menu - View commit details
-
Copy full SHA for c8d6bd8 - Browse repository at this point
Copy the full SHA c8d6bd8View commit details -
[HUDI-2995] Enabling metadata table by default (apache#4295)
- Enabling metadata table by default
Configuration menu - View commit details
-
Copy full SHA for bc8bf04 - Browse repository at this point
Copy the full SHA bc8bf04View commit details
Commits on Dec 15, 2021
-
[HUDI-3022] Fix NPE for isDropPartition method (apache#4319)
* [HUDI-3022] Fix NPE for isDropPartition method
Configuration menu - View commit details
-
Copy full SHA for dbec6c5 - Browse repository at this point
Copy the full SHA dbec6c5View commit details -
[HUDI-3024] Add explicit write handler for flink (apache#4329)
Co-authored-by: wangminchao <wangminchao@asinking.com>
Configuration menu - View commit details
-
Copy full SHA for 9a2030a - Browse repository at this point
Copy the full SHA 9a2030aView commit details -
[HUDI-3025] Add additional wait time for namenode availability during…
… IT tests initiatialization (apache#4328) - Co-authored-by: Sivabalan Narayanan <n.siva.b@gmail.com>
Configuration menu - View commit details
-
Copy full SHA for 3b89457 - Browse repository at this point
Copy the full SHA 3b89457View commit details -
[HUDI-3028] Use blob storage to speed up CI downloads (apache#4331)
Co-authored-by: Sivabalan Narayanan <n.siva.b@gmail.com>
Configuration menu - View commit details
-
Copy full SHA for 27907de - Browse repository at this point
Copy the full SHA 27907deView commit details -
[HUDI-2998] claiming rfc number for consistent hashing index (apache#…
…4303) Co-authored-by: xiaoyuwei <xiaoyuwei.yw@alibaba-inc.com>
Configuration menu - View commit details
-
Copy full SHA for f5b07a7 - Browse repository at this point
Copy the full SHA f5b07a7View commit details
Commits on Dec 16, 2021
-
Configuration menu - View commit details
-
Copy full SHA for ea2eba1 - Browse repository at this point
Copy the full SHA ea2eba1View commit details -
[Minor] Catch and ignore all the exceptions in quietDeleteMarkerDir (a…
…pache#4301) Co-authored-by: yuezhang <yuezhang@freewheel.tv>
Configuration menu - View commit details
-
Copy full SHA for a8a192a - Browse repository at this point
Copy the full SHA a8a192aView commit details -
Configuration menu - View commit details
-
Copy full SHA for 294d712 - Browse repository at this point
Copy the full SHA 294d712View commit details
Commits on Dec 17, 2021
-
[HUDI-3043] Revert async cleaner leak commit to unblock CI failure (a…
…pache#4343) * Revert "[HUDI-2959] Fix the thread leak of cleaning service (apache#4252)" Reverting to unblock CI failure for now. will revisit this with the right fix
Configuration menu - View commit details
-
Copy full SHA for 7e7ad15 - Browse repository at this point
Copy the full SHA 7e7ad15View commit details -
Configuration menu - View commit details
-
Copy full SHA for d0087d4 - Browse repository at this point
Copy the full SHA d0087d4View commit details -
[HUDI-3046] Claim RFC number for RFC for Compaction / Clustering Serv…
…ice (apache#4347) Co-authored-by: yuzhaojing <yuzhaojing@bytedance.com>
Configuration menu - View commit details
-
Copy full SHA for e4cfb42 - Browse repository at this point
Copy the full SHA e4cfb42View commit details -
[HUDI-2958] Automatically set spark.sql.parquet.writelegacyformat, wh…
…en using bulkinsert to insert data which contains decimalType (apache#4253)
Configuration menu - View commit details
-
Copy full SHA for 9246b16 - Browse repository at this point
Copy the full SHA 9246b16View commit details -
Configuration menu - View commit details
-
Copy full SHA for 6eba834 - Browse repository at this point
Copy the full SHA 6eba834View commit details
Commits on Dec 18, 2021
-
[HUDI-2962] InProcess lock provider to guard single writer process wi…
…th async table operations (apache#4259) - Adding Local JVM process based lock provider implementation - This local lock provider can be used by a single writer process with async table operations to guard the metadata tabl against concurrent updates.
Configuration menu - View commit details
-
Copy full SHA for 7784249 - Browse repository at this point
Copy the full SHA 7784249View commit details -
Configuration menu - View commit details
-
Copy full SHA for 4785244 - Browse repository at this point
Copy the full SHA 4785244View commit details -
[HUDI-3029] Transaction manager: avoid deadlock when doing begin and …
…end transactions (apache#4363) * [HUDI-3029] Transaction manager: avoid deadlock when doing begin and end transactions - Transaction manager has begin and end transactions as synchronized methods. Based on the lock provider implementaion, this can lead to deadlock situation when the underlying lock() calls are blocking or with a long timeout. - Fixing transaction manager begin and end transactions to not get to deadlock and to not assume anything on the lock provider implementation.
Configuration menu - View commit details
-
Copy full SHA for d1d48ed - Browse repository at this point
Copy the full SHA d1d48edView commit details -
[HUDI-3029] Transaction manager: avoid deadlock when doing begin and …
…end transactions (apache#4373)
Configuration menu - View commit details
-
Copy full SHA for 733732b - Browse repository at this point
Copy the full SHA 733732bView commit details -
Configuration menu - View commit details
-
Copy full SHA for dc40397 - Browse repository at this point
Copy the full SHA dc40397View commit details -
[HUDI-3054] Fixing default lock configs for FileSystemBasedLock and f…
…ixing a flaky test (apache#4374)
Configuration menu - View commit details
-
Copy full SHA for 77abb5c - Browse repository at this point
Copy the full SHA 77abb5cView commit details
Commits on Dec 19, 2021
-
Configuration menu - View commit details
-
Copy full SHA for f57e28f - Browse repository at this point
Copy the full SHA f57e28fView commit details -
Configuration menu - View commit details
-
Copy full SHA for bb99836 - Browse repository at this point
Copy the full SHA bb99836View commit details -
Configuration menu - View commit details
-
Copy full SHA for 478f9f3 - Browse repository at this point
Copy the full SHA 478f9f3View commit details -
Configuration menu - View commit details
-
Copy full SHA for 03f71ef - Browse repository at this point
Copy the full SHA 03f71efView commit details -
[HUDI-3064][HUDI-3054] FileSystemBasedLockProviderTestClass tryLock f…
…ix and TestHoodieClientMultiWriter test fixes (apache#4384) - Made FileSystemBasedLockProviderTestClass thread safe and fixed the tryLock retry logic. - Made TestHoodieClientMultiWriter. testHoodieClientBasicMultiWriter deterministic in verifying the HoodieWriteConflictException.
Configuration menu - View commit details
-
Copy full SHA for 4a48f99 - Browse repository at this point
Copy the full SHA 4a48f99View commit details
Commits on Dec 20, 2021
-
Configuration menu - View commit details
-
Copy full SHA for 3ca9210 - Browse repository at this point
Copy the full SHA 3ca9210View commit details -
Configuration menu - View commit details
-
Copy full SHA for f166dda - Browse repository at this point
Copy the full SHA f166ddaView commit details
Commits on Dec 21, 2021
-
Configuration menu - View commit details
-
Copy full SHA for 982ae3d - Browse repository at this point
Copy the full SHA 982ae3dView commit details -
[HUDI-3070] Add rerunFailingTestsCount for flakly testes (apache#4398)
Co-authored-by: yuezhang <yuezhang@freewheel.tv>
Configuration menu - View commit details
-
Copy full SHA for f3f6112 - Browse repository at this point
Copy the full SHA f3f6112View commit details -
Configuration menu - View commit details
-
Copy full SHA for 32a44bb - Browse repository at this point
Copy the full SHA 32a44bbView commit details -
Configuration menu - View commit details
-
Copy full SHA for 7d046f9 - Browse repository at this point
Copy the full SHA 7d046f9View commit details -
Configuration menu - View commit details
-
Copy full SHA for 92f54ce - Browse repository at this point
Copy the full SHA 92f54ceView commit details
Commits on Dec 22, 2021
-
Configuration menu - View commit details
-
Copy full SHA for f1286c2 - Browse repository at this point
Copy the full SHA f1286c2View commit details -
[HUDI-2547] Schedule Flink compaction in service (apache#4254)
Co-authored-by: yuzhaojing <yuzhaojing@bytedance.com>
Configuration menu - View commit details
-
Copy full SHA for 15eb7e8 - Browse repository at this point
Copy the full SHA 15eb7e8View commit details -
Merge pull request apache#4308 from harsh1231/HUDI-3008
[HUDI-3008] Fixing HoodieFileIndex partition column parsing for nested fields
Configuration menu - View commit details
-
Copy full SHA for b5890cd - Browse repository at this point
Copy the full SHA b5890cdView commit details -
[HUDI-3011] Adding ability to read entire data with HoodieIncrSource …
…with empty checkpoint (apache#4334) * Adding ability to read entire data with HoodieIncrSource with empty checkpoint * Addressing comments
Configuration menu - View commit details
-
Copy full SHA for 1a5f869 - Browse repository at this point
Copy the full SHA 1a5f869View commit details
Commits on Jan 5, 2022
-
Configuration menu - View commit details
-
Copy full SHA for fc05a75 - Browse repository at this point
Copy the full SHA fc05a75View commit details -
Configuration menu - View commit details
-
Copy full SHA for 38479fb - Browse repository at this point
Copy the full SHA 38479fbView commit details