Skip to content

Conversation

@hellozepp
Copy link

Description of PR

How was this patch tested?

For code changes:

  • Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
  • Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
  • If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
  • If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

mukund-thakur and others added 30 commits July 5, 2021 11:23
…#3110)

Introducing fs.azure.readahead.range parameter which can be set by the user.
Data will be populated in buffer for random reads as well which leads to fewer
remote calls.

This patch also changes the seek implementation to perform a lazy seek. The
actual seek is done when a read is initiated and data is not present in the buffer else
data is returned from the buffer thus reducing the number of remote storage calls.

Contributed By: Mukund Thakur

Change-Id: Ib920eedd0087caa150afa4d4c23e89df56b29e83
…y Bilwa S T (BilwaST)

(cherry picked from commit 7581413)
…ords for creates, deletes and Renames. (apache#3188)

(cherry picked from commit 6450c1e)
… occasionally. Contributed by Daniel Ma

(cherry picked from commit b6c06c4)
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit ba325a8)
…y tomscut.

Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
…cessors) to const (apache#3106)

* HADOOP-17596. ABFS: Change default Readahead Queue Depth from num(processors) to const (apache#2795)
. Contributed by Sumangala Patki.

(cherry picked from commit 76d92eb)
…3187)

Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
Reviewed-by: Chris Nauroth <cnauroth@apache.org>
(cherry picked from commit c81f82e)
…fsCompatibility tests fail when triggered with default configs (apache#3035) (apache#3126)

(cherry picked from commit 35e4c31)
…ily. Contributed by Abhishek Das (apache#2260)

(cherry picked from commit 1dd03cc)
…n Simon metrics registry. Contributed by Eric Payne.

(cherry picked from commit 632f64c)
…a request outside an audit span (apache#3207)


Followup to HADOOP-17511. "Add audit/telemetry logging to S3A connector"

Contributed by Mehakmeet Singh
…S3AFS (apache#3202)

Contributed by: Mehakmeet Singh.

Change-Id: I26c2a85ef6bbfd1b8269a23fc44d9a55d7fa091c
… failure or data loss. Contributed by Kihwal Lee.

(cherry picked from commit 4700271)
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit e1d00ad)
Signed-off-by: Sean Busbey <busbey@apache.org>
(cherry picked from commit 98412ce)
…rent length than the expected" as EOFException (apache#3040)

Some network exceptions can raise SdkClientException with message
`Data read has a different length than the expected`.

These should be recoverable.

Contributed by Bogdan Stolojan

Change-Id: Ia22fd77d90971e9e02b4f947398a4749eebe5909
…old version MR tarball work with new version (3.0) of hadoop. (apache#3227)

Revert "HDFS-10845. Change defaults in hdfs-site.xml to match timeunit type. Contributed by Yiqun Lin"

This reverts commit b6d839a.

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml

(cherry picked from commit 2f2f822)
Bumps [http-proxy](https://github.com/http-party/node-http-proxy) from 1.18.0 to 1.18.1.
- [Release notes](https://github.com/http-party/node-http-proxy/releases)
- [Changelog](https://github.com/http-party/node-http-proxy/blob/master/CHANGELOG.md)
- [Commits](http-party/node-http-proxy@1.18.0...1.18.1)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit b7431c3)
erichadoop and others added 29 commits January 4, 2022 14:37
… method violates its general contract. Contributed by Andras Gyori (gandras) and Qi Zhu (zhuqi).

(cherry picked from commit e2d6fd0)
…ng events to timeline server in batch (apache#3793)

Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 00e2405)
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit c8725de)
apache#3823)

Reviewed-by: Fei Hui <feihui.ustc@gmail.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit a5659fe)
Reviewed-by: Fei Hui <feihui.ustc@gmail.com>
(cherry picked from commit dba139c)
…es (apache#3836)

Co-authored-by: xuzq <xuzengqiang@kuaishou.com>
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit caab29e)
…es (apache#3836)

Co-authored-by: xuzq <xuzengqiang@kuaishou.com>
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit caab29e)
Change-Id: Ifd3fb9ec6ebfc8874bb799bc198219511fe55a2f

Update pom.xml

Update pom.xml

(cherry picked from commit 7950548)
…ell class. (apache#3846)

Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 7398a0f)
…sExceptionsReturnsCorrectValue (apache#3835)

Reviewed-by: Fei Hui <feihui.ustc@gmail.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 9eea0e2)
…y striped blocks. (apache#3880)

Reviewed-by: litao <tomleescut@gmail.com>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
…3825). Contributed by Ayush Saxena.

Reviewed-by: tomscut <litao@bigo.sg>
Reviewed-by: Steve Loughran <stevel@apache.org>
…tory encryption object store (apache#3870)

Contributed by Monthon Klongklaew

Change-Id: Ib275c9690bbc90170c6a442ded198fe006c20bc1
…isy. (apache#3860)

Downgrading warn logs to debug in case of InterruptedException

Contributed By: Mukund Thakur
…pt (apache#3879)

Contributed by Ahmar Suhail

Change-Id: I6a33043831a059325c58b0f76c925e52c6ae14f7
…chronously (apache#3882).  Contributed by Xiangyi Zhu.

Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
See HADOOP-18091. S3A auditing leaks memory through ThreadLocal references

* Adds a new option fs.s3a.audit.enabled to controls whether or not auditing
is enabled. This is false by default.

* When false, the S3A auditing manager is NoopAuditManagerS3A,
which was formerly only used for unit tests and
during filsystem initialization.

* When true, ActiveAuditManagerS3A is used for managing auditing,
allowing auditing events to be reported.

* updates documentation and tests.

This patch does not fix the underlying leak. When auditing is enabled,
long-lived threads will retain references to the audit managers
of S3A filesystem instances which have already been closed.

Contributed by Steve Loughran.

Change-Id: I671e594cd59e8ca77a1f65be791ad0ae9530b8d9
…pache#3955)

Add support for S3 Access Points. This provides extra security as it
ensures applications are not working with buckets belong to third parties.

To bind a bucket to an access point, set the access point (ap) ARN,
which must be done for each specific bucket, using the pattern

fs.s3a.bucket.$BUCKET.accesspoint.arn = ARN

* The global/bucket option `fs.s3a.accesspoint.required` to
mandate that buckets must declare their access point.
* This is not compatible with S3Guard.

Consult the documentation for further details.
…er than deleting. (apache#3940). Contributed by Ayush Saxena.

Reviewed-by: Steve Loughran <stevel@apache.org>
Reviewed-by: stack <stack@apache.org>
…lication catalog webapp (apache#2591)

Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
(cherry picked from commit 9cb535c)
Contributed by Steve Loughran

Change-Id: I3856afa821dbc8c2e3cb1cbe33793ec1734e2e24
@hellozepp hellozepp closed this Feb 2, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.