Backport: ABFS. HADOOP-17296 to 17404

[sumangala-patki]

Cherry-picked the following commits:

HADOOP-17296. ABFS: Force reads to be always of buffer size. (PR, JIRA)
HADOOP-17413. ABFS: Release elastic byte buffer pool at close (PR, JIRA)
HADOOP-17422. ABFS: Set default ListMaxResults to max server limit (PR, JIRA)
HADOOP-17407. ABFS: Fix NPE on delete idempotency flow (PR, JIRA)
HADOOP-17347. ABFS: Read optimizations (PR, JIRA)
HADOOP-17404. ABFS: Small write - Merge append and flush (PR, JIRA)

New configs overall: fs.azure.read.alwaysReadBufferSize, fs.azure.read.readahead.blocksize, fs.azure.read.smallfilescompletely, fs.azure.read.optimizefooterread, fs.azure.write.enableappendwithflush

[hadoop-yetus]

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	29m 20s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 1s	No case conflicting files found.
+0 🆗	markdownlint	0m 0s	markdownlint was not available.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s	The patch appears to include 15 new or modified test files.
		_ branch-3.3 Compile Tests _
+1 💚	mvninstall	32m 34s	branch-3.3 passed
+1 💚	compile	0m 36s	branch-3.3 passed
+1 💚	checkstyle	0m 28s	branch-3.3 passed
+1 💚	mvnsite	0m 36s	branch-3.3 passed
+1 💚	shadedclient	16m 27s	branch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 27s	branch-3.3 passed
+0 🆗	spotbugs	0m 58s	Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	0m 56s	branch-3.3 passed
		_ Patch Compile Tests _
+1 💚	mvninstall	0m 29s	the patch passed
+1 💚	compile	0m 27s	the patch passed
+1 💚	javac	0m 27s	the patch passed
+1 💚	checkstyle	0m 19s	the patch passed
+1 💚	mvnsite	0m 31s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	xml	0m 3s	The patch has no ill-formed XML file.
+1 💚	shadedclient	15m 38s	patch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 23s	the patch passed
+1 💚	findbugs	0m 59s	the patch passed
		_ Other Tests _
+1 💚	unit	1m 28s	hadoop-azure in the patch passed.
+1 💚	asflicense	0m 33s	The patch does not generate ASF License warnings.
		103m 57s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2618/1/artifact/out/Dockerfile
GITHUB PR	#2618
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml findbugs checkstyle markdownlint
uname	Linux 52c65b4cc2eb 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	branch-3.3 / 75f34a1
Default Java	Private Build-1.8.0_275-8u275-b01-0ubuntu1~16.04-b01
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2618/1/testReport/
Max. process+thread count	538 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2618/1/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

[sumangala-patki]

TEST RESULTS

HNS Account Location: East US 2
NonHNS Account Location: East US 2, Central US

HNS OAuth

[INFO] Tests run: 89, Failures: 0, Errors: 0, Skipped: 0
[WARNING] Tests run: 461, Failures: 0, Errors: 0, Skipped: 68
[WARNING] Tests run: 208, Failures: 0, Errors: 0, Skipped: 24

HNS SharedKey

[INFO] Tests run: 89, Failures: 0, Errors: 0, Skipped: 0
[WARNING] Tests run: 461, Failures: 0, Errors: 0, Skipped: 24
[WARNING] Tests run: 208, Failures: 0, Errors: 0, Skipped: 16

Non-HNS SharedKey
[INFO] Tests run: 89, Failures: 0, Errors: 0, Skipped: 0
[WARNING] Tests run: 461, Failures: 0, Errors: 0, Skipped: 249
[WARNING] Tests run: 208, Failures: 0, Errors: 0, Skipped: 16

[steveloughran]

thanks.Checked out the branch locally and did the merge of individual changes; github only lets us squash a commit