[FLINK-11012] Introduce abstract superclass for filesystem IT cases #7384

wujinhu · 2018-12-30T12:28:10Z

What is the purpose of the change

IT cases for filesystems are somewhat similar as they are using write/read cycles and querying meta data like directory listing. We could avoid code duplication now and for the future by introducing an abstract superclass for those it cases that outlines the tests.

Brief change log

Add AbstractHadoopFileSystemITTest as the superclass
Move common test cases to the superclass and remove them from flink-oss-fs-hadoop/flink-s3-fs-hadoop/flink-s3-fs-presto
Add HadoopOSSFileSystemBehaviorITCase for flink-oss-fs-hadoop

Verifying this change

This change does not change the tests logic, so they should be verified as before

Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): (yes / no)
The public API, i.e., is any changed class annotated with @Public(Evolving): (yes / no)
The serializers: (yes / no / don't know)
The runtime per-record code paths (performance sensitive): (yes / no / don't know)
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (yes / no / don't know)
The S3 file system connector: (yes / no / don't know)

Documentation

Does this pull request introduce a new feature? (yes / no)
If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)

wujinhu · 2018-12-30T13:58:24Z

@StefanRRichter Please help to review this PR, thanks.
I will upload test results after you review this PR.
Notice: I do not change tests of flink-swift-fs-hadoop because I do not have the corresponding environment. I think we can leave it to its owner.

wujinhu · 2019-01-07T03:50:53Z

@StefanRRichter Please help to review this PR, thanks :)

wujinhu · 2019-01-14T01:40:09Z

@StefanRRichter Would you pls review this PR? It has been pending for half a month, thanks.

StefanRRichter

Thanks for your contribution, I think it looks very good. I will partition this into three commits, before merging (1 code fix, 1 doc fix, the test rewrites). 👍

StefanRRichter · 2019-01-14T17:33:03Z

...systems/flink-hadoop-fs/src/main/java/org/apache/flink/runtime/fs/hdfs/HadoopFileSystem.java


-		if (scheme.startsWith("s3") || scheme.startsWith("emr")) {
-			// the Amazon S3 storage
+		if (scheme.startsWith("s3") || scheme.startsWith("emr") || scheme.startsWith("oss")) {


Maybe this change should go into s separate commit, either with a different jira or as hotfix.

Sounds good, I think hotfix will be better.

wujinhu · 2019-01-15T01:53:33Z

Thanks @StefanRRichter
I will upload test results if you think it's time now.

StefanRRichter · 2019-01-15T10:00:23Z

Ok, please upload the result and I will merge after that.

wujinhu · 2019-01-16T02:53:17Z

@StefanRRichter
I found an issue when I run mvn clean install with s3 credentials, it fails.
[INFO]
[INFO] force-shading ...................................... SUCCESS [ 1.742 s]
[INFO] flink .............................................. SUCCESS [ 8.802 s]
[INFO] flink-annotations .................................. SUCCESS [ 1.359 s]
[INFO] flink-shaded-hadoop ................................ SUCCESS [ 0.173 s]
[INFO] flink-shaded-hadoop2 ............................... SUCCESS [ 12.045 s]
[INFO] flink-shaded-hadoop2-uber .......................... SUCCESS [ 17.859 s]
[INFO] flink-shaded-yarn-tests ............................ SUCCESS [ 12.220 s]
[INFO] flink-shaded-curator ............................... SUCCESS [ 0.812 s]
[INFO] flink-metrics ...................................... SUCCESS [ 0.108 s]
[INFO] flink-metrics-core ................................. SUCCESS [ 2.664 s]
[INFO] flink-test-utils-parent ............................ SUCCESS [ 0.124 s]
[INFO] flink-test-utils-junit ............................. SUCCESS [ 1.834 s]
[INFO] flink-core ......................................... SUCCESS [01:15 min]
[INFO] flink-java ......................................... SUCCESS [ 29.085 s]
[INFO] flink-queryable-state .............................. SUCCESS [ 0.105 s]
[INFO] flink-queryable-state-client-java .................. SUCCESS [ 2.713 s]
[INFO] flink-filesystems .................................. SUCCESS [ 0.125 s]
[INFO] flink-hadoop-fs .................................... SUCCESS [ 11.096 s]
[INFO] flink-runtime ...................................... SUCCESS [12:00 min]
[INFO] flink-scala ........................................ SUCCESS [01:09 min]
[INFO] flink-mapr-fs ...................................... SUCCESS [ 3.911 s]
[INFO] flink-filesystems :: flink-fs-hadoop-shaded ........ SUCCESS [ 5.436 s]
[INFO] flink-s3-fs-base ................................... SUCCESS [ 10.906 s]
[INFO] flink-s3-fs-hadoop ................................. SUCCESS [04:04 min]
[INFO] flink-s3-fs-presto ................................. FAILURE [ 22.152 s]
[INFO] flink-swift-fs-hadoop .............................. SKIPPED

Error message:
[ERROR] Errors:
[ERROR] PrestoS3FileSystemBehaviorITCase>FileSystemBehaviorTestSuite.cleanup:81 » NoSuchMethod
[ERROR] PrestoS3FileSystemBehaviorITCase>FileSystemBehaviorTestSuite.cleanup:81 » NoSuchMethod
[ERROR] PrestoS3FileSystemBehaviorITCase>FileSystemBehaviorTestSuite.cleanup:81 » NoSuchMethod
[ERROR] org.apache.flink.fs.s3presto.PrestoS3FileSystemBehaviorITCase.testMkdirsFailsForExistingFile(org.apache.flink.fs.s3presto.PrestoS3FileSystemBehaviorITCase)
[ERROR] Run 1: PrestoS3FileSystemBehaviorITCase>FileSystemBehaviorTestSuite.testMkdirsFailsForExistingFile:154->FileSystemBehaviorTestSuite.assumeNotObjectStore:207 » AssumptionViolated
[ERROR] Run 2: PrestoS3FileSystemBehaviorITCase>FileSystemBehaviorTestSuite.cleanup:81 » NoSuchMethod
[INFO]
[ERROR] org.apache.flink.fs.s3presto.PrestoS3FileSystemBehaviorITCase.testMkdirsFailsWithExistingParentFile(org.apache.flink.fs.s3presto.PrestoS3FileSystemBehaviorITCase)
[ERROR] Run 1: PrestoS3FileSystemBehaviorITCase>FileSystemBehaviorTestSuite.testMkdirsFailsWithExistingParentFile:172->FileSystemBehaviorTestSuite.assumeNotObjectStore:207 » AssumptionViolated
[ERROR] Run 2: PrestoS3FileSystemBehaviorITCase>FileSystemBehaviorTestSuite.cleanup:81 » NoSuchMethod
[INFO]
[ERROR] org.apache.flink.fs.s3presto.PrestoS3FileSystemBehaviorITCase.testMkdirsReturnsTrueForExistingDirectory(org.apache.flink.fs.s3presto.PrestoS3FileSystemBehaviorITCase)
[ERROR] Run 1: PrestoS3FileSystemBehaviorITCase>FileSystemBehaviorTestSuite.testMkdirsReturnsTrueForExistingDirectory:145->FileSystemBehaviorTestSuite.createRandomFileInDirectory:203->FileSystemBehaviorTestSuite.createFile:196 » NoSuchMethod
[ERROR] Run 2: PrestoS3FileSystemBehaviorITCase>FileSystemBehaviorTestSuite.cleanup:81 » NoSuchMethod
[INFO]
[ERROR] PrestoS3FileSystemBehaviorITCase>FileSystemBehaviorTestSuite.cleanup:81 » NoSuchMethod
[ERROR] PrestoS3FileSystemBehaviorITCase>FileSystemBehaviorTestSuite.cleanup:81 » NoSuchMethod
[ERROR] PrestoS3FileSystemITCase.testConfigKeysForwarding:103 » NoSuchMethod com.googl...
[ERROR] PrestoS3FileSystemITCase.testConfigKeysForwarding:103 » NoSuchMethod com.googl...
[ERROR] PrestoS3FileSystemITCase.testDirectoryListing:191 » NoSuchMethod com.google.co...
[ERROR] PrestoS3FileSystemITCase.testDirectoryListing:191 » NoSuchMethod com.google.co...
[ERROR] PrestoS3FileSystemITCase.testSimpleFileWriteAndRead:171 » NoSuchMethod com.goo...
[ERROR] PrestoS3FileSystemITCase.testSimpleFileWriteAndRead:171 » NoSuchMethod com.goo...
[INFO]
[ERROR] Tests run: 14, Failures: 0, Errors: 14, Skipped: 0

However, when I enter flink-systems/flink-s3-fs-presto and ran the same command, it's OK.

I found flink master branch also has this issue. Do you think I should fire another issue fix this first, or am I missing sth?

wujinhu · 2019-01-16T04:12:22Z

Test results from this PR:

Ran tests except flink-s3-fs-presto moudle
MyMacBook:flink-filesystems wujinhu$ IT_CASE_S3_BUCKET=...IT_CASE_S3_ACCESS_KEY=.... mvn clean install -pl '!flink-s3-fs-presto'
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] flink-filesystems .................................. SUCCESS [ 3.019 s]
[INFO] flink-hadoop-fs .................................... SUCCESS [ 10.914 s]
[INFO] flink-mapr-fs ...................................... SUCCESS [ 3.593 s]
[INFO] flink-filesystems :: flink-fs-hadoop-shaded ........ SUCCESS [ 6.652 s]
[INFO] flink-s3-fs-base ................................... SUCCESS [ 10.916 s]
[INFO] flink-s3-fs-hadoop ................................. SUCCESS [03:58 min]
[INFO] flink-swift-fs-hadoop .............................. SUCCESS [ 19.842 s]
[INFO] flink-oss-fs-hadoop ................................ SUCCESS [ 15.325 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 05:09 min
[INFO] Finished at: 2019-01-16T12:10:22+08:00
[INFO] Final Memory: 101M/1495M
[INFO] ------------------------------------------------------------------------
Run tests for flink-s3-fs-presto moudle
MyMacBook:flink-s3-fs-presto wujinhu$ IT_CASE_S3_BUCKET=...IT_CASE_S3_ACCESS_KEY=.... mvn clean install

[INFO] -------------------------------------------------------
[INFO] T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.apache.flink.fs.s3presto.PrestoS3FileSystemBehaviorITCase
[INFO] Running org.apache.flink.fs.s3presto.PrestoS3FileSystemITCase
[WARNING] Tests run: 8, Failures: 0, Errors: 0, Skipped: 2, Time elapsed: 11.915 s - in org.apache.flink.fs.s3presto.PrestoS3FileSystemBehaviorITCase
[INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 41.194 s - in org.apache.flink.fs.s3presto.PrestoS3FileSystemITCase
[INFO]
[INFO] Results:
[INFO]
[WARNING] Tests run: 14, Failures: 0, Errors: 0, Skipped: 2
[INFO]
[INFO]
.........
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 01:03 min
[INFO] Finished at: 2019-01-16T12:11:47+08:00
[INFO] Final Memory: 66M/1512M
[INFO] ------------------------------------------------------------------------

StefanRRichter · 2019-01-16T12:38:10Z

LGTM 👍 Merging.

wujinhu · 2019-01-16T12:42:05Z

@StefanRRichter Thanks, but what about the issue I mentioned? Flink master branch tests fail if I execute command in flink-s3-fs-presto's parent directory.

StefanRRichter · 2019-01-16T12:43:22Z

But you said this was the same in the master, so it is would not be a regression from this PR and can be fixed independently.

wujinhu · 2019-01-16T12:44:11Z

But you said this was the same in the master, so it is would not be a regression from this PR and can be fixed independently.
@StefanRRichter
OK, I think I can fire another jira to fix this.

StefanRRichter · 2019-01-16T12:45:10Z

Thanks!

…cases. This closes apache#7384.

wujinhu added 2 commits December 30, 2018 20:06

Introduce abstract superclass for filesystem IT cases

979d642

remove one comment

4b8da69

StefanRRichter approved these changes Jan 14, 2019

View reviewed changes

asfgit closed this in c08db3f Jan 16, 2019

tisonkun pushed a commit to tisonkun/flink that referenced this pull request Jan 17, 2019

[FLINK-11012][tests] Introduce abstract superclass for filesystem IT …

53b732e

…cases. This closes apache#7384.

rmetzger added the component=FileSystems label Mar 18, 2019

[FLINK-11012] Introduce abstract superclass for filesystem IT cases #7384

[FLINK-11012] Introduce abstract superclass for filesystem IT cases #7384

Uh oh!

Conversation

wujinhu commented Dec 30, 2018

What is the purpose of the change

Brief change log

Verifying this change

Does this pull request potentially affect one of the following parts:

Documentation

Uh oh!

wujinhu commented Dec 30, 2018

Uh oh!

wujinhu commented Jan 7, 2019

Uh oh!

wujinhu commented Jan 14, 2019

Uh oh!

StefanRRichter left a comment

Choose a reason for hiding this comment

Uh oh!

StefanRRichter Jan 14, 2019

Choose a reason for hiding this comment

Uh oh!

wujinhu Jan 15, 2019

Choose a reason for hiding this comment

Uh oh!

wujinhu commented Jan 15, 2019

Uh oh!

StefanRRichter commented Jan 15, 2019

Uh oh!

wujinhu commented Jan 16, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

wujinhu commented Jan 16, 2019

Uh oh!

StefanRRichter commented Jan 16, 2019

Uh oh!

wujinhu commented Jan 16, 2019

Uh oh!

StefanRRichter commented Jan 16, 2019

Uh oh!

wujinhu commented Jan 16, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

StefanRRichter commented Jan 16, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

wujinhu commented Jan 16, 2019 •

edited

Loading

wujinhu commented Jan 16, 2019 •

edited

Loading