[SUPPORT] Issues with URL_ENCODE_PARTITIONING_OPT_KEY in hudi 0.11.0

[bkosuru]

Hello,

We created hudi table in hdfs using version 0.8.0. We are planning to upgrade to hudi 0.11.0. There seems to be a difference in url encoding. We are using
option(URL_ENCODE_PARTITIONING_OPT_KEY, value = true)

For partition

0.8.0 encoding is
%3Chttp%3A%2F%2Fpurl.obolibrary.org%2Fobo%2Fuberon.owl%3E

0.11.0 encoding is
<http%3A%2F%2Fpurl.obolibrary.org%2Fobo%2Fuberon.owl>

When we insert new data with hudi 0.11.0 it creates a different partition.
/data/spog/g=%3Chttp%3A%2F%2Fpurl.obolibrary.org%2Fobo%2Fuberon.owl%3E
/data/spog/g=<http%3A%2F%2Fpurl.obolibrary.org%2Fobo%2Fuberon.owl>

Is this a bug? Any workaround?

Thanks,
Bindu

[nsivabalan]

do you happened to know what do you see w/ local FS ? I will give it a try.

[bkosuru]

I did not check. I tested it on the cluster. We are not able upgrade to 0.11.0 because of this issue. Thanks

[bkosuru]

I see the same difference on the local FS

[nsivabalan]

I could not able to reproduce atleast w/ local FS.

import org.apache.hudi.QuickstartUtils._
import scala.collection.JavaConversions._
import org.apache.spark.sql.SaveMode._
import org.apache.hudi.DataSourceReadOptions._
import org.apache.hudi.DataSourceWriteOptions._
import org.apache.hudi.config.HoodieWriteConfig._

val tableName = "hudi_trips_cow"
val basePath = "file:///tmp/hudi_trips_cow"
val dataGen = new DataGenerator

// spark-shell
val inserts = convertToStringList(dataGen.generateInserts(10))
val df = spark.read.json(spark.sparkContext.parallelize(inserts, 2))

import org.apache.spark.sql.functions.lit;


df.withColumn("ppath",lit("http://purl.obolibrary.org/obo/uberon.owl")).write.format("hudi").
  options(getQuickstartWriteConfigs).
  option(PRECOMBINE_FIELD_OPT_KEY, "ts").
  option(RECORDKEY_FIELD_OPT_KEY, "uuid").
  option(PARTITIONPATH_FIELD_OPT_KEY, "ppath").
  option(TABLE_NAME, tableName).
  option(URL_ENCODE_PARTITIONING_OPT_KEY,"true").
  mode(Append).
  save(basePath)

I tired above script for 0.8.0, 0.10.0 and 0.11.0 and its all same. infact, I same file group was updated for every commit which I tried w/ newer versions.

[nsivabalan]

nsb$ ls -ltr /tmp/hudi_trips_cow/
total 0
drwxr-xr-x  10 nsb  wheel  320 May 13 14:39 http%3A%2F%2Fpurl.obolibrary.org%2Fobo%2Fuberon.owl
nsb$ ls -ltr /tmp/hudi_trips_cow/http%3A%2F%2Fpurl.obolibrary.org%2Fobo%2Fuberon.owl/
total 2592
-rw-r--r--  1 nsb  wheel  438813 May 13 14:37 9a5cf8d4-029c-4389-9184-0c43d1b22d13-0_0-84-106_20220513143702.parquet
-rw-r--r--  1 nsb  wheel  439521 May 13 14:38 9a5cf8d4-029c-4389-9184-0c43d1b22d13-0_0-27-30_20220513143842591.parquet
-rw-r--r--  1 nsb  wheel  440341 May 13 14:39 9a5cf8d4-029c-4389-9184-0c43d1b22d13-0_0-36-43_20220513143934780.parquet

[bkosuru]

Please add angle brackets on each side of the input partition. The angle brackets got deleted in the plain text. Below is the screenshot. I updated the description as well. Thanks

[bkosuru]

[bettermouse]

    //hudi-client/hudi-client-common/src/main/java/org/apache/hudi/keygen/KeyGenUtils.java   getRecordPartitionPath
    //https://github.com/apache/hudi/pull/2645
    String encodeVersion08 = URLEncoder.encode("<http://purl.obolibrary.org/obo/uberon.owl>", StandardCharsets.UTF_8.toString());
    String encodeVersion11 = PartitionPathEncodeUtils.escapePathName("<http://purl.obolibrary.org/obo/uberon.owl>");

    System.out.println(encodeVersion08);
    System.out.println(encodeVersion11);

//%3Chttp%3A%2F%2Fpurl.obolibrary.org%2Fobo%2Fuberon.owl%3E
//<http%3A%2F%2Fpurl.obolibrary.org%2Fobo%2Fuberon.owl>

It seems like " Partition Path Encode " in different way in version 0.8 and 0.11.
may need to rebuild table by hudi 0.11?

[xushiyan]

@nsivabalan the problem is some newer encoding logic introduced in after 0.8 and not encoding < and >, which resulted in the difference. We need to make a call to fix this in minor release.