Hdfs 17834 #8079
Closed
Hdfs 17834 #8079
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…pache#6394) Signed-off-by: Takanobu Asanuma <tasanuma@apache.org> (cherry picked from commit 6a05376)
…buted by Shilun Fan. Reviewed-by: Steve Loughran <stevel@apache.org> Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org> (cherry picked from commit 0f8b74b)
…ibuted by He Xiaoqiao. Signed-off-by: Shuyan Zhang <zhangshuyan@apache.org> Signed-off-by: Shilun Fan <slfan1989@apache.org> Signed-off-by: Ayush Saxena <ayushsaxena@apache.org> (cherry picked from commit 9634bd3)
…sSystemImpl changes. (apache#6450) Contributed by Shilun Fan. Reviewed-by: Steve Loughran <stevel@cloudera.com> Signed-off-by: Shilun Fan <slfan1989@apache.org>
Co-authored-by: slfan1989 <slfan1989@apache.org>
apache#2998)" (apache#6457) Contributed by Shilun Fan. This reverts commit c1bf3cb. Reviewed-by: Takanobu Asanuma <tasanuma@apache.org> Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org> Reviewed-by: Ayush Saxena <ayushsaxena@apache.org> Reviewed-by: Viraj Jasani <vjasani@apache.org> Signed-off-by: Shilun Fan <slfan1989@apache.org>
…buted by Shilun Fan. Reviewed-by: Steve Loughran <stevel@cloudera.com> Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org> Signed-off-by: Shilun Fan <slfan1989@apache.org>
… by PJ Fanning. Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org> Reviewed-by: Steve Loughran <stevel@cloudera.com> Signed-off-by: Shilun Fan <slfan1989@apache.org>
…ibuted by Ayush Saxena." This reverts commit c04a17f. Reverted from Branch-3.4, since this commit is relevant only for trunk.
…che#6324) Move to the new auth flow based signers for aws. * Implement a new Signer Initialization Chain * Add a new instantiation method * Add a new test * Fix Reflection Code for SignerInitialization Contributed by Harshit Gupta
…xceptions (apache#6425) Differentiate from "EOF out of range/end of GET" from "EOF channel problems" through two different subclasses of EOFException and input streams to always retry on http channel errors; out of range GET requests are not retried. Currently an EOFException is always treated as a fail-fast call in read() This allows for all existing external code catching EOFException to handle both, but S3AInputStream to cleanly differentiate range errors (map to -1) from channel errors (retry) - HttpChannelEOFException is subclass of EOFException, so all code which catches EOFException is still happy. retry policy: connectivityFailure - RangeNotSatisfiableEOFException is the subclass of EOFException raised on 416 GET range errors. retry policy: fail - Method ErrorTranslation.maybeExtractChannelException() to create this from shaded/unshaded NoHttpResponseException, using string match to avoid classpath problems. - And do this for SdkClientExceptions with OpenSSL error code WFOPENSSL0035. We believe this is the OpenSSL equivalent. - ErrorTranslation.maybeExtractIOException() to perform this translation as appropriate. S3AInputStream.reopen() code retries on EOF, except on RangeNotSatisfiableEOFException, which is converted to a -1 response to the caller as is done historically. S3AInputStream knows to handle these with read(): HttpChannelEOFException: stream aborting close then retry lazySeek(): Map RangeNotSatisfiableEOFException to -1, but do not map any other EOFException class raised. This means that * out of range reads map to -1 * channel problems in reopen are retried * channel problems in read() abort the failed http connection so it isn't recycled Tests for this using/abusing mocking. Testing through actually raising 416 exceptions and verifying that readFully(), char read() and vector reads are all good. There is no attempt to recover within a readFully(); there's a boolean constant switch to turn this on, but if anyone does it a test will spin forever as the inner PositionedReadable.read(position, buffer, len) downgrades all EOF exceptions to -1. A new method would need to be added which controls whether to downgrade/rethrow exceptions. What does that mean? Possibly reduced resilience to non-retried failures on the inner stream, even though more channel exceptions are retried on. Contributed by Steve Loughran
…points (apache#6277) Adds a new option `fs.s3a.endpoint.fips` to switch the SDK client to use FIPS endpoints, as an alternative to explicitly declaring them. * The option is available as a path capability for probes. * SDK v2 itself doesn't know that some regions don't have FIPS endpoints * SDK only fails with endpoint + fips flag as a retried exception; wit this change the S3A client should fail fast. PR fails fast. * Adds a new "connecting.md" doc; moves existing docs there and restructures. * New Tests in ITestS3AEndpointRegion bucket-info command support: * added to list of path capabilities * added -fips flag and test for explicit probe * also now prints bucket region * and removed some of the obsolete s3guard options * updated docs Contributed by Steve Loughran
…isk of Timeout waiting for connection from pool. (apache#6372) HADOOP-19015. Increase fs.s3a.connection.maximum to 500 to minimize the risk of Timeout waiting for connection from the pool Contributed By: Mukund Thakur
…= false (apache#6441) Add new option fs.s3a.checksum.validation, default false, which is used when creating s3 clients to enable/disable checksum validation. When false, GET response processing is measurably faster. Contributed by Steve Loughran.
…pache#6462) Contributed by Shilun Fan. Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org> Signed-off-by: Shilun Fan <slfan1989@apache.org>
…r the release 3.4.0 (apache#6500) Contributed by Benjamin Teke. Signed-off-by: Shilun Fan <slfan1989@apache.org>
…e#6467) This update ensures that the timeout set in fs.s3a.connection.request.timeout is passed down to calls to CreateSession made in the AWS SDK to get S3 Express session tokens. Contributed by Steve Loughran
… server calls (apache#6022) Address JDK bug JDK-8314978 related to handling of HTTP 100 responses. https://bugs.openjdk.org/browse/JDK-8314978 In the AbfsHttpOperation, after sendRequest() we call processResponse() method from AbfsRestOperation. Even if the conn.getOutputStream() fails due to expect-100 error, we consume the exception and let the code go ahead. This may call getHeaderField() / getHeaderFields() / getHeaderFieldLong() after getOutputStream() has failed. These invocation all lead to server calls. This commit aims to prevent this. If connection.getOutputStream() fails due to an Expect-100 error, the ABFS client does not invoke getHeaderField(), getHeaderFields(), getHeaderFieldLong() or getInputStream(). getResponseCode() is safe as on the failure it sets the responseCode variable in HttpUrlConnection object. Contributed by Pranav Saxena
…#6470) New test ITestCreateSessionTimeout to verify that the duration set in fs.s3a.connection.request.timeout is passed all the way down. This is done by adding a sleep() in a custom signer and verifying that it is interrupted and that an AWSApiCallTimeoutException is raised. + Fix testRequestTimeout() * doesn't skip if considered cross-region * sets a minimum duration of 0 before invocation * resets the minimum afterwards Contributed by Steve Loughran
Cut out S3 Select * leave public/unstable constants alone * s3guard tool will fail with error * s3afs. path capability will fail * openFile() will fail with specific error * s3 select doc updated * Cut eventstream jar * New test: ITestSelectUnsupported verifies new failure handling above Contributed by Steve Loughran
…ll packets. (apache#6503). Contributed by farmmamba. Signed-off-by: Takanobu Asanuma <tasanuma@apache.org> (cherry picked from commit 4f4b846)
Contributed by Dieter De Paepe
Improves region handling in the S3A connector, including enabling cross-region support when that is considered necessary. Consult the documentation in connecting.md/connecting.html for the current resolution process. Contributed by Viraj Jasani
…ache#6488) Contributed by Jia Fan
…adoop-hdfs-rbf (apache#6522) (cherry picked from commit 9a7eead)
This is a followup to PR: HADOOP-19045. S3A: Validate CreateSession Timeout Propagation (apache#6470) Remove all declarations of fs.s3a.connection.request.timeout in - hadoop-common/src/main/resources/core-default.xml - hadoop-aws/src/test/resources/core-site.xml New test in TestAwsClientConfig to verify that the value defined in fs.s3a.Constants class is used. This is brittle to someone overriding it in their test setups, but as this test is intended to verify that the option is not explicitly set, there's no workaround. Contributed by Steve Loughran
The option fs.s3a.classloader.isolation (default: true) can be set to false to disable s3a classloader isolation; This can assist in using custom credential providers and other extension points. Contributed by Antonio Murgia
Contributed by Adnan Hemani
…che#6406) Contributed by Viraj Jasani
* HttpReferrerAuditHeader is thread safe, copying the lists/maps passed in and using synchronized methods when necessary. * All exceptions raised when building referrer header are caught and swallowed. * The first such error is logged at warn, * all errors plus stack are logged at debug Contributed by Steve Loughran
…t is set (apache#7093) (apache#7067) Adds new option s3a.cross.region.access.enabled Which is true by default This enables cross region access as a separate config and enable/disables it irrespective of region/endpoint is set. This commit contains (ADDENDUM) (apache#7098) Contributed by Syed Shameerur Rahman
…7089) (apache#7100) This sets a different timeout for data upload PUT/POST calls to all other requests, so that slow block uploads do not trigger timeouts as rapidly as normal requests. This was always the behavior in the V1 AWS SDK; for V2 we have to explicitly set it on the operations we want to give extended timeouts. Option: fs.s3a.connection.part.upload.timeout Default: 15m Contributed by Steve Loughran
…ase v2 (apache#6629). (apache#7017) This removes support for HBase 1 in the timeline service, moving to an up-to-date HBase 2 version instead. This does not affect the ability of HBase 1 to run on this version of Hadoop -it just removes it from our own redistributables, along with all copies of protobuf-2.5 Contributed by Ayush Saxena
…7101) ChecksumFileSystem creates the chunked ranges based on the checksum chunk size and then calls readVectored on Raw Local which may lead to overlapping ranges in some cases. Contributed by: Mukund Thakur
apache#7091) Contributed by Cheng Pan. Reviewed-by: Steve Loughran <stevel@apache.org> Signed-off-by: Shilun Fan <slfan1989@apache.org>
…ainer for CapacityScheduler (apache#7065). Contributed by Tao Yang. Reviewed-by: Syed Shameerur Rahman <syedthameem1@gmail.com> Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
… (apache#7118) * HADOOP-19310. Add JPMS options required by Java 17+ (apache#7114) Contributed by Cheng Pan. Reviewed-by: Attila Doroszlai <adoroszlai@apache.org> Signed-off-by: Shilun Fan <slfan1989@apache.org>
…ache#7006) (apache#7097) This moves Hadoop to Apache commons-collections4. Apache commons-collections has been removed and is completely banned from the source code. Contributed by Nihal Jain
…file does not contain file scheme (apache#7113) Contributed by Syed Shameerur Rahman
…e#7149) Contributed by Sebastian Klemke
…pache#7125) (apache#7147) Contributed by Min Yan. Signed-off-by: Shilun Fan <slfan1989@apache.org>
github-actions
bot
added
YARN
MapReduce
HDFS
build
DistCp
trunk
Common
RBF
TOOLS
AWS
ABFS
Infra
Native
Maven-Plugins
DYNAMOMETER
labels
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
48 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description of PR
This PR fixes HDFS-17834: HDFS NameNode Web UI generates invalid HTTP links for DataNodes in IPv6 environment.
Problem
The NameNode Web UI incorrectly handles IPv6 addresses in DataNode links:
[]as required by RFC 2732Solution
Example
Before:
https://datanodeHostname:30(invalid)After:
https://[1172:30:0:0:0:0:8a34]:50010(valid)How was this patch tested?
For code changes:
Note: This PR only contains code changes and no new dependencies were added.