[WIP][HUDI-3625][RFC-60] OSSStorageStrategy POC by zhangyue19921010 · Pull Request #12460 · apache/hudi

zhangyue19921010 · 2024-12-11T03:26:26Z

Change Logs

OSS Storage strategy POC

For local testing, write data using Spark and query data with Spark, using UT as an example. Assume /tmp/bucketA/ is the user's S3 bucket. The final data distribution is as follows

UT can pass directly.

hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/dml/TestInsertTable.scala
test("Test Insert Into with values and OSS storage")

The distribution of metadata

zhangyue61@ZBMac-C02FN40EMD6P ~ % cd /private/var/folders/ww/dc150vl50815wqgbsx_98w9w0000gp/T/spark-8212a35d-7784-47fc-a496-ce499753edf1
zhangyue61@ZBMac-C02FN40EMD6P spark-8212a35d-7784-47fc-a496-ce499753edf1 % ll
total 0
drwxr-xr-x    3 zhangyue61  staff     96 12 11 11:12 ./
drwx------@ 317 zhangyue61  staff  10144 12 11 11:12 ../
drwxr-xr-x    3 zhangyue61  staff     96 12 11 11:12 h1/
zhangyue61@ZBMac-C02FN40EMD6P spark-8212a35d-7784-47fc-a496-ce499753edf1 % tree -a
.
└── h1
    └── .hoodie
        ├── .aux
        │   └── .bootstrap
        │       ├── .fileids
        │       └── .partitions
        ├── .hoodie.properties.crc
        ├── .schema
        ├── .temp
        ├── hoodie.properties
        ├── metadata
        │   ├── .hoodie
        │   │   ├── .aux
        │   │   │   └── .bootstrap
        │   │   │       ├── .fileids
        │   │   │       └── .partitions
        │   │   ├── .hoodie.properties.crc
        │   │   ├── .schema
        │   │   ├── .temp
        │   │   ├── hoodie.properties
        │   │   └── timeline
        │   │       ├── .00000000000000000.deltacommit.inflight.crc
        │   │       ├── .00000000000000000.deltacommit.requested.crc
        │   │       ├── .00000000000000000_20241211031254712.deltacommit.crc
        │   │       ├── .20241211031249856.deltacommit.inflight.crc
        │   │       ├── .20241211031249856.deltacommit.requested.crc
        │   │       ├── .20241211031249856_20241211031302018.deltacommit.crc
        │   │       ├── 00000000000000000.deltacommit.inflight
        │   │       ├── 00000000000000000.deltacommit.requested
        │   │       ├── 00000000000000000_20241211031254712.deltacommit
        │   │       ├── 20241211031249856.deltacommit.inflight
        │   │       ├── 20241211031249856.deltacommit.requested
        │   │       ├── 20241211031249856_20241211031302018.deltacommit
        │   │       └── history
        │   └── files
        │       ├── ..files-0000-0_00000000000000000.log.1_0-0-0.crc
        │       ├── ..files-0000-0_20241211031249856.log.1_0-18-229.crc
        │       ├── ..hoodie_partition_metadata.crc
        │       ├── .files-0000-0_0-5-4_00000000000000000.hfile.crc
        │       ├── .files-0000-0_00000000000000000.log.1_0-0-0
        │       ├── .files-0000-0_20241211031249856.log.1_0-18-229
        │       ├── .hoodie_partition_metadata
        │       └── files-0000-0_0-5-4_00000000000000000.hfile
        └── timeline
            ├── .20241211031249856.commit.requested.crc
            ├── .20241211031249856.inflight.crc
            ├── .20241211031249856_20241211031302213.commit.crc
            ├── 20241211031249856.commit.requested
            ├── 20241211031249856.inflight
            ├── 20241211031249856_20241211031302213.commit
            └── history

22 directories, 30 files

The distribution of data

zhangyue61@ZBMac-C02FN40EMD6P spark-8212a35d-7784-47fc-a496-ce499753edf1 % cd /tmp/bucketA/
zhangyue61@ZBMac-C02FN40EMD6P bucketA % ll
total 0
drwxr-xr-x   5 zhangyue61  wheel   160 12 11 11:12 ./
drwxrwxrwt  70 root        wheel  2240 12 11 11:12 ../
drwxr-xr-x   3 zhangyue61  wheel    96 12 11 11:12 1103196347/
drwxr-xr-x   3 zhangyue61  wheel    96 12 11 11:12 1782571635/
drwxr-xr-x   3 zhangyue61  wheel    96 12 11 11:12 538548677/
zhangyue61@ZBMac-C02FN40EMD6P bucketA % tree -a
.
├── 1103196347
│   └── private
│       └── var
│           └── folders
│               └── ww
│                   └── dc150vl50815wqgbsx_98w9w0000gp
│                       └── T
│                           └── spark-8212a35d-7784-47fc-a496-ce499753edf1
│                               └── h1
│                                   └── dt=2021-01-06
│                                       ├── ..hoodie_partition_metadata.crc
│                                       ├── .977a6c38-41a9-4423-921b-b96ef57c853b-0_1-11-18_20241211031249856.parquet.crc
│                                       ├── .hoodie_partition_metadata
│                                       └── 977a6c38-41a9-4423-921b-b96ef57c853b-0_1-11-18_20241211031249856.parquet
├── 1782571635
│   └── private
│       └── var
│           └── folders
│               └── ww
│                   └── dc150vl50815wqgbsx_98w9w0000gp
│                       └── T
│                           └── spark-8212a35d-7784-47fc-a496-ce499753edf1
│                               └── h1
│                                   └── dt=2021-01-05
│                                       ├── ..hoodie_partition_metadata.crc
│                                       ├── .7d71eaa5-72e3-4ffd-8d90-ee09c27b0675-0_0-11-17_20241211031249856.parquet.crc
│                                       ├── .hoodie_partition_metadata
│                                       └── 7d71eaa5-72e3-4ffd-8d90-ee09c27b0675-0_0-11-17_20241211031249856.parquet
└── 538548677
    └── private
        └── var
            └── folders
                └── ww
                    └── dc150vl50815wqgbsx_98w9w0000gp
                        └── T
                            └── spark-8212a35d-7784-47fc-a496-ce499753edf1
                                └── h1
                                    └── dt=2021-01-07
                                        ├── ..hoodie_partition_metadata.crc
                                        ├── .1bd01552-e01e-4696-a7cb-6dba97dfe2b1-0_2-11-19_20241211031249856.parquet.crc
                                        ├── .hoodie_partition_metadata
                                        └── 1bd01552-e01e-4696-a7cb-6dba97dfe2b1-0_2-11-19_20241211031249856.parquet

31 directories, 12 files

Base Path

/private/var/folders/ww/dc150vl50815wqgbsx_98w9w0000gp/T/spark-8212a35d-7784-47fc-a496-ce499753edf1/h1/.hoodie

Data Path

/tmp/bucketA/1103196347/private/var/folders/ww/dc150vl50815wqgbsx_98w9w0000gp/T/spark-8212a35d-7784-47fc-a496-ce499753edf1/h1/dt=2021-01-06/977a6c38-41a9-4423-921b-b96ef57c853b-0_1-11-18_20241211031249856.parquet

Impact

no

Risk level (write none, low medium or high below)

low

Documentation Update

Describe any necessary documentation update if there is any new feature, config, or user-facing change. If not, put "none".

The config description must be updated if new configs are added or the default value of the configs are changed
Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
ticket number here and follow the instruction to make
changes to the website.

Contributor's checklist

Read through contributor's guide
Change Logs and Impact were stated clearly
Adequate tests were added if applicable
CI passed

This reverts commit fc67ac2.

This reverts commit e18fc63.

CTTY · 2024-12-12T02:33:54Z

hudi-io/src/main/java/org/apache/hudi/storage/HoodieOSSStorageStrategy.java

+  public static final String FILE_ID_KEY = "hoodie_file_id";
+  public static final String TABLE_BASE_PATH = "hoodie_table_base_path";
+  public static final String TABLE_NAME = "hoodie_table_name";
+  public static final String TABLE_STORAGE_PATH = "hoodie_storage_path";


We should reuse existing config names

Sure thing, just for poc quick test

CTTY · 2024-12-12T03:17:03Z

hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieWriteHandle.java

    return FSUtils.makeWriteToken(getPartitionId(), getStageId(), getAttemptId());
  }

+  protected StoragePath getPartitionPath(String partitionPath) {


nit: we can change the method name to toPhysicalPath

make sense!

CTTY · 2024-12-12T03:17:59Z

hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieCreateHandle.java

      HoodiePartitionMetadata partitionMetadata = new HoodiePartitionMetadata(storage, instantTime,
-          new StoragePath(config.getBasePath()),
-          FSUtils.constructAbsolutePath(config.getBasePath(), partitionPath),
+          new StoragePath(config.getBasePath()), getPartitionPath(partitionPath),


Why do we need to store physical path in partition metadata?

CTTY · 2024-12-12T03:19:27Z

hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieWriteHandle.java

+
  public StoragePath makeNewPath(String partitionPath) {
-    StoragePath path = FSUtils.constructAbsolutePath(config.getBasePath(), partitionPath);
+    StoragePath path = getPartitionPath(partitionPath);


I assume this change is just for the POC? Ideally the conversion from logical path to physical path should happen within HoodieStorage at L134

CTTY · 2024-12-12T03:20:38Z

hudi-common/src/main/java/org/apache/hudi/common/table/view/AbstractTableFileSystemView.java

          List<Pair<String, StoragePath>> absolutePartitionPathList = partitionSet.stream()
              .map(partition -> Pair.of(
-                  partition, FSUtils.constructAbsolutePath(metaClient.getBasePath(), partition)))
+                  partition, storage.getAllLocations(partition, config).stream().findFirst().get()))


Just curious, would findFirst() work here even in the production?

CTTY · 2024-12-12T03:45:28Z

hudi-common/src/main/java/org/apache/hudi/metadata/HoodieMetadataPayload.java

+    HoodieStorage storage = dataMetaClient.getStorage();
+    String tableName = dataMetaClient.getTableConfig().getTableName();
+    StoragePath dataBasePath = dataMetaClient.getBasePath();
    long blockSize = storage.getDefaultBlockSize(partitionPath);


How would this line leverage storage strategy?

zhangyue143 added 3 commits December 9, 2024 11:21

flink schedule compaction with incremental partitions

e18fc63

fix checkstyle

fc67ac2

poc finished

8eab655

zhangyue19921010 mentioned this pull request Dec 11, 2024

[HUDI-3625][RFC-60] Update RFC-60 With Federated Storage Layout #12407

Open

4 tasks

zhangyue143 added 2 commits December 11, 2024 11:29

Revert "fix checkstyle"

c56b6a3

This reverts commit fc67ac2.

Revert "flink schedule compaction with incremental partitions"

13986f8

This reverts commit e18fc63.

zhangyue19921010 requested a review from yihua December 11, 2024 03:36

github-actions bot added the size:M PR with lines of changes in (100, 300] label Dec 11, 2024

add getAllLocations api

9109ba8

CTTY reviewed Dec 12, 2024

View reviewed changes

vinothchandar added the rfc Request for comments label Feb 19, 2025

yihua self-assigned this Feb 28, 2025

hudi-bot mentioned this pull request Dec 9, 2025

[RFC-60] Optimized storage layout for cloud object stores #15062

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP][HUDI-3625][RFC-60] OSSStorageStrategy POC#12460

[WIP][HUDI-3625][RFC-60] OSSStorageStrategy POC#12460
zhangyue19921010 wants to merge 6 commits intomasterfrom
rfc-60-ossstorage-poc

zhangyue19921010 commented Dec 11, 2024 •

edited

Loading

Uh oh!

CTTY Dec 12, 2024

Uh oh!

zhangyue19921010 Dec 12, 2024

Uh oh!

CTTY Dec 12, 2024

Uh oh!

zhangyue19921010 Dec 12, 2024

Uh oh!

CTTY Dec 12, 2024

Uh oh!

CTTY Dec 12, 2024

Uh oh!

CTTY Dec 12, 2024

Uh oh!

CTTY Dec 12, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

zhangyue19921010 commented Dec 11, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Change Logs

Impact

Risk level (write none, low medium or high below)

Documentation Update

Contributor's checklist

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

zhangyue19921010 commented Dec 11, 2024 •

edited

Loading