[SUPPORT] Process Fallout: Unable to upsert due to continues Failure in Compaction

[ketkidev]

Describe the problem you faced

Unable to process data into the Hudi tables as inline Compaction is failing.

To Reproduce

Previous hoodie.compact.inline.max.delta.commits was 6. We changed it to 10.

Hudi MOR Table configuration:
{ 'hoodie.table.name': asset, 'hoodie.datasource.write.recordkey.field': id, 'hoodie.datasource.write.table.name': asset, 'hoodie.upsert.shuffle.parallelism': 400, 'hoodie.keep.max.commits': 1000, 'hoodie.keep.min.commits': 990, 'hoodie.compact.inline.max.delta.commits': 10, 'hoodie.clean.automatic': 'false', 'hoodie.clean.async': 'false' 'hoodie.datasource.write.operation': 'upsert', 'hoodie.datasource.write.precombine.field': 'cdc_timestamp', 'hoodie.datasource.write.table.type': 'MERGE_ON_READ', 'hoodie.cleaner.policy': 'KEEP_LATEST_COMMITS', 'hoodie.schema.on.read.enable' : "true", 'hoodie.datasource.write.reconcile.schema' : "true", 'hoodie.compact.inline' : "true", 'hoodie.datasource.write.keygenerator.class': 'org.apache.hudi.keygen.NonpartitionedKeyGenerator' }
Error:
org.apache.hudi.exception.HoodieCompactionException: Could not compact s3a://*************************/hudi_tables/assets/asset

Environment Description

• Hudi version : 0.13.0

• Spark version : 3.3.1

• Storage (HDFS/S3/GCS..) : S3

• Running on Docker? (yes/no) : no

Additional context

Add any other context about the problem here.

Stacktrace

HudiCompactionIssue20231003.txt

[ad1happy2go]

@ketkidev Just after updating hoodie.compact.inline.max.delta.commits , you started getting this error? Is there any other config you updated?

[ketkidev]

@ad1happy2go It didn't happen quickly. We changed on 29th Sept and issue came on 3rd Oct. No other configuration changes.

[ad1happy2go]

Discussed with @ketkidev offline. Was using the wrong config for S3 max connection. Asked to use fs.s3a.connection.maximum .

[ketkidev]

@ad1happy2go As disucssed, we are getting another error which suggesting that some file is missing:
: org.apache.hudi.exception.HoodieCompactionException: Could not compact s3a://***/hudi_tables/assets/asset . . . Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Task 17 in stage 376.0 failed 1 times, most recent failure: Lost task 17.0 in stage 376.0 (TID 6008) (ip-10-208-6-95.us-west-2.compute.internal executor driver): org.apache.hudi.exception.HoodieException: Exception when reading log file . . . Caused by: org.apache.hudi.exception.HoodieIOException: unable to initialize read with log file . . . Caused by: java.io.FileNotFoundException: No such file or directory: s3a://***/hudi_tables/assets/asset/.10df2f04-d4d4-4fc1-95f2-f645e89d3d61-0_20231003100633082.log.7_95-442-4962

HudiCompactionIssue20231005_00.txt

[ketkidev]

@ad1happy2go We removed all inflight requests for compaction. Still unable to process further.

[ad1happy2go]

@ketkidev Did you again faced the same exception with some other file? or any other issues.