feat : linkis-storage support s3 filesystem support write/read launch log and resultSet in s3(#4185) #4435

sjgllgh · 2023-04-03T17:02:19Z

What is the purpose of the change

linkis-storge support s3 filesystem;
support write/read launch log in s3 bucket;
support write/read resultSet in s3 bucket.

Related issues/PRs

Related issues: #4185
Related pr: #4185

Brief change log

add S3FileSystem class which extends from FileSystem class, in order to operate s3.
add BuildS3System class to build S3FileSystem.
modify StorageUtils class to fix an error when reading a resultSet from s3 in S3ObjectInputStream.
modify StorageConfiguration class.

How to use s3 file system

modify configs in conf/linkis-cg-entrance.properties.

wds.linkis.resultSet.store.path=s3://linkis/tmp/
wds.linkis.filesystem.hdfs.root.path=s3://linkis/tmp/

add configs in conf/linkis.properties.

wds.linkis.storage.s3.access.key=kyrO2anrUJQLtS2B
wds.linkis.storage.s3.secret.key=3BO2O9YHW110A64lx4hqh0MNsGHfIpZP
wds.linkis.storage.s3.endpoint=http://localhost:9000/
wds.linkis.storage.s3.region=cn-hb-1
wds.linkis.storage.s3.bucket=bucket-1

Checklist

I have read the Contributing Guidelines on pull requests.
I have explained the need for this PR and the problem it solves
I have explained the changes or the new features added to this PR
I have added tests corresponding to this change
I have updated the documentation to reflect this change
I have verified that this change is backward compatible (If not, please discuss on the Linkis mailing list first)
If this is a code change: I have written unit tests to fully verify the new behavior.

linkis-dist/package/conf/linkis.properties

peacewong · 2023-04-04T09:33:51Z

linkis-commons/linkis-storage/src/main/java/org/apache/linkis/storage/fs/impl/S3FileSystem.java

+      return ret;
+    } else {
+      logger.warn("File or folder does not exist or file name is garbled(文件或者文件夹不存在或者文件名乱码)");
+      throw new StorageWarnException(TO_BE_UNKNOW.getErrorCode(), "文件或者文件夹不存在或者文件名乱码");


should add English

Thanks for your advice.I've added the English comments.

peacewong · 2023-04-04T09:58:25Z

...s/linkis-storage/src/main/java/org/apache/linkis/storage/factory/impl/BuildS3FileSystem.java

+    } catch (IOException e) {
+      LOG.warn("get file system failed", e);
+    }
+    fs.setUser(user);


should use proxyUser?

Thanks for the suggestion, but I don't think need a proxyUser in there. If I am wrong, please let me know.

peacewong · 2023-04-04T10:02:09Z

linkis-commons/linkis-storage/src/main/java/org/apache/linkis/storage/fs/impl/S3FileSystem.java

+  }
+
+  @Override
+  public void close() throws IOException {}


Whether there are resources that need to be released when close

Thanks for your advice.I checked the code and there were no resources to close.

Thanks for your advice.I checked the code and there were no resources to close.

ok, thanks!

peacewong · 2023-04-04T10:03:56Z

linkis-commons/linkis-storage/src/main/java/org/apache/linkis/storage/fs/impl/S3FileSystem.java

+    this.label = label;
+  }
+
+  public String buildPath(String path)   {


should Use org.apache.linkis.common.io.FsPath#getSchemaPath

Thanks for your advice. I use this method because the subdirectories I get through s3client don't have a "/" prefix. I tested this at minIo. The method you mentioned does not meet my needs.

Thanks for your advice. I use this method because the subdirectories I get through s3client don't have a "/" prefix. I tested this at minIo. The method you mentioned does not meet my needs.

ok, thanks!

peacewong · 2023-04-04T10:04:24Z

...ons/linkis-storage/src/main/scala/org/apache/linkis/storage/utils/StorageConfiguration.scala

@@ -83,4 +83,15 @@ object StorageConfiguration {
  val FS_CHECKSUM_DISBALE =
    CommonVars[java.lang.Boolean]("linkis.fs.hdfs.impl.disable.checksum", false)

+  val S3_ACCESS_KEY = CommonVars("wds.linkis.storage.s3.access.key", "")


need to remove wds prefix

Thanks for your advice.I've dropped the prefix.

peacewong

LGTM.

peacewong · 2023-04-10T07:19:34Z

@sjgllgh code needs to be formatted

peacewong

LGTM.

peacewong · 2023-04-10T12:40:59Z

ping @jackxu2011

jackxu2011

LGTM

linkis-storage support S3 filesystem

89d5c4d

jackxu2011 requested changes Apr 4, 2023

View reviewed changes

linkis-dist/package/conf/linkis.properties Outdated Show resolved Hide resolved

peacewong reviewed Apr 4, 2023

View reviewed changes

sjgllgh added 4 commits April 6, 2023 09:16

format code

63058ad

add comment

20badfe

remove wds.

47de2e0

merge dev-1.4.0 and fix conflicts

5b4899a

peacewong previously approved these changes Apr 10, 2023

View reviewed changes

1. format code 2. add dependant license

84899b4

sjgllgh dismissed peacewong’s stale review via 84899b4 April 10, 2023 07:45

sjgllgh added 3 commits April 10, 2023 17:20

format linkis-storage.pom

262f897

format linkis-storage.pom

c02b610

format code

ae43eae

peacewong approved these changes Apr 10, 2023

View reviewed changes

peacewong requested a review from jackxu2011 April 10, 2023 12:40

jackxu2011 approved these changes Apr 10, 2023

View reviewed changes

jackxu2011 merged commit f0f6657 into apache:dev-1.4.0 Apr 10, 2023
10 checks passed

sjgllgh deleted the dev-1.4.0-storge-support-s3-filesystem branch April 13, 2023 08:32

sjgllgh mentioned this pull request May 10, 2023

[Feature ]Support docking with FS such as AWS S3 #4185

Closed

aiceflower mentioned this pull request May 31, 2023

[Feature] Linkis1.4.0 requirements #4590

Open

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat : linkis-storage support s3 filesystem support write/read launch log and resultSet in s3(#4185) #4435

feat : linkis-storage support s3 filesystem support write/read launch log and resultSet in s3(#4185) #4435

sjgllgh commented Apr 3, 2023

peacewong Apr 4, 2023

sjgllgh Apr 7, 2023

peacewong Apr 4, 2023

sjgllgh Apr 7, 2023

peacewong Apr 4, 2023

sjgllgh Apr 7, 2023

peacewong Apr 10, 2023

peacewong Apr 4, 2023

sjgllgh Apr 7, 2023

peacewong Apr 10, 2023

peacewong Apr 4, 2023

sjgllgh Apr 7, 2023

peacewong left a comment

peacewong commented Apr 10, 2023

peacewong left a comment

peacewong commented Apr 10, 2023

jackxu2011 left a comment

feat : linkis-storage support s3 filesystem support write/read launch log and resultSet in s3(#4185) #4435

feat : linkis-storage support s3 filesystem support write/read launch log and resultSet in s3(#4185) #4435

Conversation

sjgllgh commented Apr 3, 2023

What is the purpose of the change

Related issues/PRs

Brief change log

How to use s3 file system

Checklist

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

peacewong left a comment

Choose a reason for hiding this comment

peacewong commented Apr 10, 2023

peacewong left a comment

Choose a reason for hiding this comment

peacewong commented Apr 10, 2023

jackxu2011 left a comment

Choose a reason for hiding this comment