Sync write through worker #17502

JiamingMai · 2023-05-26T08:50:57Z

This PR implements the "sync write through worker" feature that supports writing data to Alluxio DORA and UFS in "sync" way.

… sync-write-through-worker

Signed-off-by: Huang Hua <huanghua78@msn.com>

dora/core/client/fs/src/main/java/alluxio/client/file/DoraFileOutStream.java

dora/core/server/worker/src/main/java/alluxio/worker/dora/PagedFileWriter.java

dora/core/client/fs/src/main/java/alluxio/client/file/dora/netty/NettyDataWriter.java

jiacheliu3

Added a few comments, PTAL thanks!

dora/core/client/fs/src/main/java/alluxio/client/file/DoraFileOutStream.java

dora/core/client/fs/src/main/java/alluxio/client/file/DoraFileOutStreamBak.java

dora/core/client/fs/src/main/java/alluxio/client/file/dora/netty/NettyDataWriter.java

dora/core/server/worker/src/main/java/alluxio/worker/netty/FileWriteHandler.java

dora/core/server/worker/src/main/java/alluxio/worker/netty/PacketReadTaskStateMachine.java

dora/core/server/worker/src/main/java/alluxio/worker/page/PagedBlockMetaStore.java

jiacheliu3

Added a few comments, PTAL thanks!

dora/core/client/fs/src/main/java/alluxio/client/file/DoraFileOutStream.java

dbw9580 · 2023-05-31T09:26:25Z

dora/core/client/fs/src/main/java/alluxio/client/file/DoraFileOutStream.java

+      if (!mUnderStorageType.isSyncPersist()) {
+        mUnderStorageOutputStream = null;
+      } else { // Write is through to the under storage, create mUnderStorageOutputStream.
+        mUnderStorageOutputStream = ufsOutStream;
+      }


We only deal with syncPersist writes currently, so this check is unnecessary, and mUnderStorageOutputStream must be non-null.

dora/core/client/fs/src/main/java/alluxio/client/file/DoraFileOutStream.java

Signed-off-by: Huang Hua <huanghua78@msn.com>

LuQQiu · 2023-06-01T00:11:20Z

@JiamingMai Does DoraFileOutStream has some simple tests?

LuQQiu · 2023-06-01T01:42:52Z

Some high level suggestions

Write to UFS first, write to Dora later <-- avoid data corruption
If write to UFS succeed, write to Dora failed should be tolerated, return success to user

Signed-off-by: Huang Hua <huanghua78@msn.com>

JiamingMai · 2023-06-01T08:21:50Z

Some high level suggestions

Write to UFS first, write to Dora later <-- avoid data corruption

If write to UFS succeed, write to Dora failed should be tolerated, return success to user

Good idea!

…t master any more

…ster

FIXME: the mNettyDataWriter.close() is stuck if no data is written. Signed-off-by: Huang Hua <huanghua78@msn.com>

JiamingMai · 2023-06-02T16:40:06Z

@JiamingMai Does DoraFileOutStream has some simple tests?

I think the existing tests have already covered this class.

beinan · 2023-06-02T19:30:48Z

alluxio-bot, merge this please.

JiamingMai added 2 commits May 25, 2023 09:53

add files

ffb4d58

add logic of committing file

2e4d3b8

huanghua78 mentioned this pull request May 29, 2023

Add initial framework for write support in Dora #17505

Merged

JiamingMai force-pushed the sync-write-through-worker branch from a405189 to 2e4d3b8 Compare May 29, 2023 02:56

JiamingMai and others added 7 commits May 29, 2023 15:29

add logic of committing metadata

3b6aae0

add getStoreDirOfFile method

564b656

Merge remote-tracking branch 'jiaming/sync-write-through-worker' into…

37f586e

… sync-write-through-worker

bugfix for committing metadata

77781f4

merge main

9086877

completed sync write through worker

b6d202d

format code style

2d492a7

JiamingMai changed the title ~~[WIP] Sync write through worker~~ Sync write through worker May 30, 2023

JiamingMai added the type-feature This issue is a feature request label May 30, 2023

JiamingMai self-assigned this May 30, 2023

JiamingMai requested review from dbw9580, huanghua78, beinan and LuQQiu May 30, 2023 08:38

Use full ufs path

7e54f39

Signed-off-by: Huang Hua <huanghua78@msn.com>

huanghua78 reviewed May 31, 2023

View reviewed changes

dora/core/client/fs/src/main/java/alluxio/client/file/DoraFileOutStream.java Outdated Show resolved Hide resolved

huanghua78 reviewed May 31, 2023

View reviewed changes

dora/core/server/worker/src/main/java/alluxio/worker/dora/PagedFileWriter.java Show resolved Hide resolved

huanghua78 reviewed May 31, 2023

View reviewed changes

dora/core/server/worker/src/main/java/alluxio/worker/dora/PagedFileWriter.java Outdated Show resolved Hide resolved

huanghua78 reviewed May 31, 2023

View reviewed changes

dora/core/client/fs/src/main/java/alluxio/client/file/dora/netty/NettyDataWriter.java Outdated Show resolved Hide resolved

huanghua78 reviewed May 31, 2023

View reviewed changes

dora/core/client/fs/src/main/java/alluxio/client/file/dora/netty/NettyDataWriter.java Show resolved Hide resolved

jiacheliu3 requested changes May 31, 2023

View reviewed changes

JiamingMai added 4 commits May 31, 2023 14:06

delete redundant class

cb7d9e9

update TODO list

26808be

have a clean up

b6ce17b

merge main

f1fb513

JiamingMai added 3 commits May 31, 2023 15:33

add metric for counting bytes written to alluxio by the client

fac42d8

divide a large buf into smaller ones

9b37862

wrap ByteBuf from ByteBuffer to avoid copying and duplicated codes

a10ab39

dbw9580 reviewed May 31, 2023

View reviewed changes

Unit Test and enhanced write.

265a753

Signed-off-by: Huang Hua <huanghua78@msn.com>

huanghua78 added 2 commits June 1, 2023 12:37

Support overwrite: invalidating cached data and commit new file

aa7fe47

Signed-off-by: Huang Hua <huanghua78@msn.com>

fix maven build problems

c6e9e64

Signed-off-by: Huang Hua <huanghua78@msn.com>

JiamingMai and others added 9 commits June 1, 2023 19:30

fix FuseFileSystemDataTest by enabling netty

147e9a2

make the createFile test method deprecated as we don't need to reques…

482184c

…t master any more

mark some methods as deprecated

5898411

mark some tests as deprecated because metadata no longer exists in ma…

1128021

…ster

mark some tests as deprecated as metadata no longer exists in master

d118b13

mark some tests as deprecated

bcfcd54

Add a switch to enable/disable dora write.

be9ca7b

FIXME: the mNettyDataWriter.close() is stuck if no data is written. Signed-off-by: Huang Hua <huanghua78@msn.com>

mark PermissionCheckerTest as ignore

2a399a7

replace MUST_CACHE configuration with CACHE_THROUGH in tests

c948fd8

beinan approved these changes Jun 2, 2023

View reviewed changes

alluxio-bot merged commit 61d24a7 into Alluxio:main Jun 2, 2023

JiamingMai deleted the sync-write-through-worker branch June 25, 2023 15:00

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Sync write through worker #17502

Sync write through worker #17502

JiamingMai commented May 26, 2023 •

edited

Loading

jiacheliu3 left a comment

jiacheliu3 left a comment

dbw9580 May 31, 2023

LuQQiu commented Jun 1, 2023

LuQQiu commented Jun 1, 2023

JiamingMai commented Jun 1, 2023

JiamingMai commented Jun 2, 2023 •

edited

Loading

beinan commented Jun 2, 2023

Sync write through worker #17502

Sync write through worker #17502

Conversation

JiamingMai commented May 26, 2023 • edited Loading

jiacheliu3 left a comment

Choose a reason for hiding this comment

jiacheliu3 left a comment

Choose a reason for hiding this comment

dbw9580 May 31, 2023

Choose a reason for hiding this comment

LuQQiu commented Jun 1, 2023

LuQQiu commented Jun 1, 2023

JiamingMai commented Jun 1, 2023

JiamingMai commented Jun 2, 2023 • edited Loading

beinan commented Jun 2, 2023

JiamingMai commented May 26, 2023 •

edited

Loading

JiamingMai commented Jun 2, 2023 •

edited

Loading