[SUPPORT] Duplicates on MOR table

[eyjian]

Hudi: 0.12.1
Flink: 0.15
Spark: 3.1

Duplicates span multiple files within the same partitionpath:

ds	ts	transcode	actid	level	_hoodie_file_name	_hoodie_partition_path	_hoodie_record_key	_hoodie_commit_time
20230509	2023/5/10 20:11	1486092202305095800051572	300108	3	00000007-6d54-4918-9abb-6f8d48843bba_4-10-0_20230514101617270.parquet	20230509	transcode:1486092202305095800051572,actid:300148,level:3	20230510200812817
20230509	2023/5/14 10:11	1486092202305095800051572	300108	3	00000001-7f3b-4b8c-9b46-dff2c4987f72_4-10-0_20230514101617270.parquet	20230509	transcode:1486092202305095800051572,actid:300148,level:3	20230514100809825

CREATE TABLE `t_test_01` (
  `_hoodie_commit_time` STRING,
  `_hoodie_commit_seqno` STRING,
  `_hoodie_record_key` STRING,
  `_hoodie_partition_path` STRING,
  `_hoodie_file_name` STRING,
  `ds` BIGINT,
  `dt` STRING,
  `ts` STRING,
  `transcode` STRING,
  `actid` BIGINT,
  `level` BIGINT
)
USING hudi
OPTIONS (
  `hoodie.query.as.ro.table` 'false'
)
PARTITIONED BY (ds)
LOCATION 'hdfs://user/root/warehouse/test_db.db/t_test_01'
TBLPROPERTIES (
  'type' = 'mor',
  'primaryKey' = 'transcode,actid,level',
  'preCombineField' = 'ts',
  'connector' = 'hudi',
  'hoodie.bucket.index.num.buckets' = '19',
  'hoodie.index.type' = 'BUCKET',
  'hoodie.datasource.write.recordkey.field' = 'transcode,actid,level',
  'hoodie.table.name' = 't_test',
  'hoodie.table.precombine.field' = 'ts'
);

INSERT INTO t_test_01
SELECT
ds,
dt,
DATE_FORMAT(CURRENT_TIMESTAMP,'yyyy-MM-dd HH:mm:ss') AS ts,
transcode,
actid,
level
FROM t_test_02;

Can not use the records deduplicate command to fix for no permissions.

[eyjian]

Problem causes: drop a table and create the table again, the table path is not deleted, so two files at the same time.

[danny0405]

Is the table a managed table or external table then?

[eyjian]

Is the table a managed table or external table then?

A hudi table created by flink,it's not an external table.

[danny0405]

How do you drop the table, if it is a managed table and if you use the Flink Hudi Hive catalog, the table path would be deleted.

[eyjian]

Drop the hudi table using SparkSQL

[danny0405]

There is a param named purge:

case class DropHoodieTableCommand(
    tableIdentifier: TableIdentifier,
    ifExists: Boolean,
    isView: Boolean,
    purge: Boolean) extends HoodieLeafRunnableCommand {

If it is true, the directory on fs would be deleted recursively.

[eyjian]

There is a param named purge:

case class DropHoodieTableCommand(
    tableIdentifier: TableIdentifier,
    ifExists: Boolean,
    isView: Boolean,
    purge: Boolean) extends HoodieLeafRunnableCommand {

If it is true, the directory on fs would be deleted recursively.

Thanks, it's good.