[Feature-13428][Resource Center] Add support for Google Cloud Storage in the resource center

[rickchengx]

Purpose of the pull request

• close: [Feature][Resource Center] Add support for Google Cloud Storage in the resource center #13428

Brief change log

• Add support for writing task logs to Google Cloud Storage

Verify this pull request

UT and manually tested

1. configure in the common.properties

resource.storage.type=GCS
# the location of the google cloud credential, required if you set resource.storage.type=GCS
resource.google.cloud.storage.credential=/path/to/credential
# gcs bucket name, required if you set resource.storage.type=GCS
resource.google.cloud.storage.bucket.name=<your-bucket>

2. Upload files and directory in the resource center

3. Check files in the GCS

[codecov-commenter]

Codecov Report

Merging #13562 (4d0300d) into dev (2bd65fb) will decrease coverage by 0.05%.
The diff coverage is 33.75%.

❗ Current head 4d0300d differs from pull request most recent head a5a00e2. Consider uploading reports for the commit a5a00e2 to get more accurate results

@@             Coverage Diff              @@
##                dev   #13562      +/-   ##
============================================
- Coverage     39.47%   39.43%   -0.05%     
- Complexity     4382     4401      +19     
============================================
  Files          1102     1104       +2     
  Lines         41437    41675     +238     
  Branches       4733     4754      +21     
============================================
+ Hits          16359    16436      +77     
- Misses        23262    23413     +151     
- Partials       1816     1826      +10     

Impacted Files	Coverage Δ
...e/dolphinscheduler/common/constants/Constants.java	75.00% <ø> (ø)
...ler/server/master/event/TaskStateEventHandler.java	0.00% <0.00%> (ø)
.../plugin/storage/gcs/GcsStorageOperatorFactory.java	0.00% <0.00%> (ø)
...heduler/plugin/storage/gcs/GcsStorageOperator.java	35.39% <35.39%> (ø)
...e/dolphinscheduler/common/enums/ResUploadType.java	100.00% <100.00%> (ø)
...r/plugin/task/sqoop/parameter/SqoopParameters.java	28.57% <0.00%> (-9.53%)	⬇️
...org/apache/dolphinscheduler/remote/utils/Host.java	42.55% <0.00%> (-2.13%)	⬇️
...e/dolphinscheduler/remote/NettyRemotingClient.java	51.38% <0.00%> (-1.39%)	⬇️
...lphinscheduler/plugin/task/sqoop/SqoopJobType.java	88.88% <0.00%> (+88.88%)	⬆️

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

[sonarcloud]

SonarCloud Quality Gate failed.   

1 Bug
0 Vulnerabilities
0 Security Hotspots
27 Code Smells

37.2% Coverage
7.2% Duplication

[rickchengx]

Hi, @zhongjiajie , @SbloodyS , @EricGao888 ,could you please help review this?

[zhongjiajie]

it seems the parameter recursive not use in this method, so GCS does not support delete recursive in this method?

[rickchengx]

Hi, @zhongjiajie , thanks a lot for your kind review. Perhaps for some historical reasons, recursive is not used (also in the S3StorageOperator and OssStorageOperator).

Another interface below is called when deleting with a childrenPathList.

@Override
    public boolean delete(String fullName, List<String> childrenPathList, boolean recursive) throws IOException {
        // append the resource fullName to the list for deletion.
        childrenPathList.add(fullName);

        boolean result = true;
        for (String filePath : childrenPathList) {
            if (!delete(filePath, recursive)) {
                result = false;
            }
        }

        return result;
    }