Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

sync #14

Merged
merged 56 commits into from
Apr 11, 2022
Merged

sync #14

merged 56 commits into from
Apr 11, 2022

Conversation

GuoPhilipse
Copy link
Owner

sync

szilard-nemeth and others added 30 commits March 21, 2022 14:18
@szilard-nemeth
YARN-11089. Fix typo in RM audit log. Contributed by Junfan Zhang.
4537b34
@szilard-nemeth
YARN-11086. Add space in debug log of ParentQueue. Contributed by Jun…
e2701e2 
…fan Zhang.
@9uapaw
YARN-10565. Refactor CS queue initialization to simplify weight mode …
c3124a3 
…calculation. Contributed by Benjamin Teke.
@omalley
HDFS-13248: Namenode needs to use the actual client IP when going thr…
1d5650c 
…ough the

RBF proxy. There is a new configuration knob dfs.namenode.ip-proxy-users that configures
the list of users than can set their client ip address using the client context.

Fixes #4081
@9uapaw
YARN-11087. Introduce the config to control the refresh interval in R…
2beb729 
…MDelegatedNodeLabelsUpdater. Contributed by Junfan Zhang.
@steveloughran
HADOOP-13704. Optimized S3A getContentSummary()
708a0ce 
Optimize the scan for s3 by performing a deep tree listing,
inferring directory counts from the paths returned.

Contributed by Ahmar Suhail.

Change-Id: I26ffa8c6f65fd11c68a88d6e2243b0eac6ffd024
@Hexiaoqiao
HDFS-14617. Improve oiv tool to parse fsimage file in parallel with d…
8897549 
…elimited format. (#2918). Contributed by Hongbing Wang.

Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
@andreAmorimF
HADOOP-18160 Avoid shading wildfly.openssl runtime dependency (#4074)
59d07bd 
Contributed by André Fonseca.
@GauthamBanasandra
HDFS-16471. Make HDFS ls tool cross platform (#4086)
81879eb 
* The source files for hdfs_ls
   uses getopt for parsing the
   command line arguments.
* getopt is available only on
   Linux and thus, isn't cross
   platform.
* Thus, we need to replace
   getopt with
   boost::program_options to
   make this tool cross platform.
@Hexiaoqiao
Revert "HDFS-14617. Improve oiv tool to parse fsimage file in paralle…
26ba384 
…l with delimited format. (#2918). Contributed by Hongbing Wang."

This reverts commit 8897549.
@Hexiaoqiao
HDFS-15987. Improve oiv tool to parse fsimage file in parallel with d…
ef8bff0 
…elimited format. (#2918). Contributed by Hongbing Wang.

Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
@liubingxing
HDFS-16501. Print the exception when reporting a bad block (#4062)
45ce1cc 
Reviewed-by: tomscut <litao@bigo.sg>
@9uapaw
YARN-11084. Introduce new config to specify AM default node-label whe…
921267c 
…n not specified. Contributed by Junfan Zhang.
@dannycjones
HADOOP-14661. Add S3 requester pays bucket support to S3A (#3962)
9edfe30 
Adds the option fs.s3a.requester.pays.enabled, which, if set to true, allows
the client to access S3 buckets where the requester is billed for the IO.

Contributed by Daniel Carl Jones
@9uapaw
YARN-10547. Decouple job parsing logic from SLSRunner. Contributed by…
077c6c6 
… Szilard Nemeth.
@9uapaw
YARN-10552. Eliminate code duplication in SLSCapacityScheduler and SL…
5261424 
…SFairScheduler. Contributed by Szilard Nemeth.
@bteke
YARN-11094. Follow up changes for YARN-10547. Contributed by Szilard …
ffa0eab 
…Nemeth
@tomscut
HDFS-16434. Add opname to read/write lock for remaining operations (#…
565e848 
…3915)
@9uapaw
YARN-10548. Decouple AM runner logic from SLSRunner. Contributed by S…
08a77a7 
…zilard Nemeth.
@tomicooler @brumi1024
YARN-11069. Dynamic Queue ACL handling in Legacy and Flexible Auto Cr…
da09d68 
…eated Queues. Contributed by Tamas Domok
@pjfanning
HADOOP-13386. Upgrade Avro to 1.9.2 (#3990)
61e809b 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
@9uapaw
YARN-11100. Fix StackOverflowError in SLS scheduler event handling. C…
adbaf48 
…ontributed by Szilard Nemeth.
@GuoPhilipse @tclxgf13871
HDFS-16355. Improve the description of dfs.block.scanner.volume.bytes…
046a620 
….per.second (#3724)

Co-authored-by: gf13871 <gf13871@ly.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
@iwasakims
Make upstream aware of 3.2.3 release.
1087633
@iwasakims
Make upstream aware of 3.2.3 release.
0fbd96a
@omalley
HDFS-16517 Distance metric is wrong for non-DN machines in 2.10. Fixe…
eb16421 
…d in HADOOP-16161, but

this test case adds value to ensure the two getWeight methods stay in sync.

Fixes #4091

Signed-off-by: Owen O'Malley <oomalley@linkedin.com>
@li-leyang @omalley
HDFS-16518: Add shutdownhook to invalidate the KeyProviders in the cache
a9b4396 
Fixes #4100
Signed-off-by: Owen O'Malley <oomalley@linkedin.com>
@9uapaw
YARN-10549. Decouple RM runner logic from SLSRunner. Contributed by S…
e386d6a 
…zilard Nemeth.
@aajisaka
HDFS-16523. Fix dependency error in hadoop-hdfs on M1 Mac (#4112)
4e32318 
The version of hawtjni-runtime has been fixed to 1.11 to avoid build failures

Contributed by Akira Ajisaka
@Hexiaoqiao
HDFS-16498. Fix NPE for checkBlockReportLease #4057. Contributed by t…
6eea28c 
…omscut.

Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
smallzhongfeng and others added 26 commits March 30, 2022 12:42
@smallzhongfeng
HADOOP-18145. Fileutil's unzip method causes unzipped files to lose t…
08e6d0c 
…heir original permissions (#4036)


Contributed by jingxiong zhong
@sekikn
MAPREDUCE-7373. Building MapReduce NativeTask fails on Fedora 34+ (#4120
dc4a680 
)
@tomscut
HDFS-16413. Reconfig dfs usage parameters for datanode (#3863)
ac50657
@zuston @brumi1024
YARN-11106. Fix the test failure due to missing conf of yarn.resource…
6e00a79 
…manager.node-labels.am.default-node-label-expression. Contributed by Junfan Zhang
@szilard-nemeth @brumi1024
YARN-10550. Decouple NM runner logic from SLSRunner. Contributed by S…
ab8c360 
…zilard Nemeth
@Hexiaoqiao
HDFS-16511. Improve lock type for ReplicaMap under fine-grain lock mo…
2bf78e2 
…de. (#4085). Contributed by limingxiang.

Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
@tomscut
HDFS-16507. [SBN read] Avoid purging edit log which is in progress (#…
9a4dddd 
…4082)
@zuston @brumi1024
YARN-11088. Introduce the config to control the AM allocated to non-e…
e044a46 
…xclusive nodes. Contributed by Junfan Zhang
@szilard-nemeth @brumi1024
YARN-11103. SLS cleanup after previously merged SLS refactor jiras. C…
94031b7 
…ontributed by Szilard Nemeth
@xinglin
HADOOP-18169. getDelegationTokens in ViewFs should also fetch the tok…
15a5ea2 
…en from fallback FS (#4094)

HADOOP-18169. getDelegationTokens in ViewFs should also fetch the token from the fallback FS
@9uapaw
YARN-11102. Fix spotbugs error in hadoop-sls module. Contributed by S…
4b1a6bf 
…zilard Nemeth, Andras Gyori.
@tomscut
HDFS-16477. [SPS]: Add metric PendingSPSPaths for getting the number …
34b3275 
…of paths to be processed by SPS (#4009). Contributed by  tomscut.

Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
@GauthamBanasandra
HDFS-16472. Make HDFS setrep tool cross platform (#4130)
4ef1d3e 
* The source files for hdfs_setrep
   uses getopt for parsing the
   command line arguments.
* getopt is available only on Linux
   and thus, isn't cross platform.
* We need to replace getopt
  with boost::program_options
  to make this tool cross platform.
@tomscut
HDFS-16527. Add global timeout rule for TestRouterDistCpProcedure (#4129
966b773 
)

Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
@sodonnel
HDFS-16530. setReplication debug log creates a new string even if deb…
bbfe350 
…ug is disabled (#4142)
@wzhallright
HDFS-16529. Remove unnecessary setObserverRead in TestConsistentReads…
61bbdfd 
…Observer (#4131)

Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
@virajjasani
HDFS-16522. Set Http and Ipc ports for Datanodes in MiniDFSCluster (#…
7c20602 
…4108)

Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
@pjfanning
HADOOP-18178. Upgrade jackson to 2.13.2 and jackson-databind to 2.13.…
4b786c7 
…2.2 (#4111)

Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
@virajjasani
HADOOP-18188. Support touch command for directory (#4135)
f709355 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
@singer-bin
HDFS-16457.Make fs.getspaceused.classname reconfigurable (#4069)
807a428
@tomscut
HDFS-16460. [SPS]: Handle failure retries for moving tasks (#4001)
5412fbf
@pjfanning
HADOOP-18195. Make jackson 1 a runtime scope dependency (#4149)
bfde910 
Contributed by PJ Fanning
@tasanuma
HDFS-16497. EC: Add param comment for liveBusyBlockIndices with HDFS-…
37650ce 
…14768. Contributed by caozhiqiang.
@virajjasani
HADOOP-18191. Log retry count while handling exceptions in RetryInvoc…
b69ede7 
…ationHandler (#4133)
@GauthamBanasandra
HDFS-16473. Make HDFS stat tool cross platform (#4145)
d5e97fe 
* The source files for hdfs_stat
  uses getopt for parsing the
  command line arguments.
* getopt is available only on
  Linux and thus, isn't cross platform.
* We need to replace getopt with
  boost::program_options to make
  this tool cross platform.
@GuoPhilipse
HDFS-16516. Fix Fsshell wrong params (#4090). Contributed by GuoPhili…
5de78ce 
…pse.
@GuoPhilipse GuoPhilipse merged commit 6f29c61 into GuoPhilipse:trunk Apr 11, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.