HADOOP-17308. WASB : PageBlobOutputStream succeeding flush even when …

[anoopsjohn]

…underlying flush to storage failed

NOTICE

Please create an issue in ASF JIRA before opening a pull request,
and you need to set the title of the pull request which starts with
the corresponding JIRA issue number. (e.g. HADOOP-XXXXX. Fix a typo in YYY.)
For more details, please see https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute

[ramkrish86]

+1 - nonbinding. Good catch.

[steveloughran]

usual process: test Azure zone, command line for test run?

[steveloughran]

make sure that hsync is happy too

[anoopsjohn]

make sure that hsync is happy too

Done. Added a test

[anoopsjohn]

usual process: test Azure zone, command line for test run?

test on East US region standard storage account
mvn test -Dtest="org.apache.hadoop.fs.azure.Test*"
Tests run: 244, Failures: 0, Errors: 0, Skipped: 11
mvn test -Dtest="org.apache.hadoop.fs.azure.ITest*"
Tests run: 530, Failures: 1, Errors: 0, Skipped: 93
Failure seems not related. Flakey?
ITestWasbUriAndConfiguration.testCanonicalServiceName:656 Expecting java.lang.IllegalArgumentException with text java.net.UnknownHostException but got : "void"

Ran tests which complained in QA
TestNativeAzureFileSystemContractMocked
Tests run: 43, Failures: 0, Errors: 0, Skipped: 5

TestNativeAzureFileSystemMocked
Tests run: 48, Failures: 0, Errors: 0, Skipped: 0

TestNativeAzureFileSystemFileNameCheck
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0

TestWasbFsck
Tests run: 2, Failures: 0, Errors: 0, Skipped: 1

TestNativeAzureFileSystemOperationsMocked
Tests run: 50, Failures: 0, Errors: 0, Skipped: 0

TestBlobMetadata
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0

TestOutOfBandAzureBlobOperations
Tests run: 7, Failures: 0, Errors: 0, Skipped: 0,

TestNativeAzureFileSystemConcurrency
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0

[hadoop-yetus]

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	42m 57s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚		0m 0s	test4tests	The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	36m 48s		trunk passed
+1 💚	compile	0m 35s		trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1
+1 💚	compile	0m 42s		trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01
+1 💚	checkstyle	0m 33s		trunk passed
+1 💚	mvnsite	0m 34s		trunk passed
+1 💚	shadedclient	18m 13s		branch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 26s		trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1
+1 💚	javadoc	0m 24s		trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01
+0 🆗	spotbugs	0m 57s		Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	0m 54s		trunk passed
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 27s		the patch passed
+1 💚	compile	0m 30s		the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1
+1 💚	javac	0m 30s		the patch passed
+1 💚	compile	0m 25s		the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01
+1 💚	javac	0m 25s		the patch passed
+1 💚	checkstyle	0m 16s		the patch passed
+1 💚	mvnsite	0m 29s		the patch passed
+1 💚	whitespace	0m 0s		The patch has no whitespace issues.
+1 💚	shadedclient	16m 17s		patch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 23s		the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1
+1 💚	javadoc	0m 21s		the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01
+1 💚	findbugs	0m 59s		the patch passed
			_ Other Tests _
-1 ❌	unit	1m 16s	/patch-unit-hadoop-tools_hadoop-azure.txt	hadoop-azure in the patch passed.
+1 💚	asflicense	0m 28s		The patch does not generate ASF License warnings.
		126m 6s

Reason	Tests
Failed junit tests	hadoop.fs.azure.TestNativeAzureFileSystemContractMocked
	hadoop.fs.azure.TestNativeAzureFileSystemMocked
	hadoop.fs.azure.TestNativeAzureFileSystemFileNameCheck
	hadoop.fs.azure.TestWasbFsck
	hadoop.fs.azure.TestNativeAzureFileSystemOperationsMocked
	hadoop.fs.azure.TestBlobMetadata
	hadoop.fs.azure.TestOutOfBandAzureBlobOperations
	hadoop.fs.azure.TestNativeAzureFileSystemConcurrency

Subsystem	Report/Notes
Docker	ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2392/2/artifact/out/Dockerfile
GITHUB PR	#2392
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux f4e900b82633 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / 88a9f42
Default Java	Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2392/2/testReport/
Max. process+thread count	308 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2392/2/console
versions	git=2.17.1 maven=3.6.0 findbugs=4.1.3
Powered by	Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

[vishwajeetdusane]

Thank you @anoopsjohn for the fix.

[steveloughran]

+1, merged to trunk

backporting to branch-3.3, where I'll do a compilation before I push up the result. If someone else could do the actual test run, that would be good

[steveloughran]

Update: merged to 3.3.x, didn't retest. If someone wants to try testing that and cherrypicking to 3.2.x...