HBASE-23259: Ability to start minicluster with pre-determined master ports #807
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
@apurtell @wchevreuil FYI. This adds the plumbing needed for the required config generation so that I can get rid of "custom" configuration stuff from the patch for HBASE-18095. |
|
💔 -1 overall
This message was automatically generated. |
|
Fixed the import order checkstyle nit in the latest push. |
virajjasani
reviewed
Nov 8, 2019
hbase-server/src/main/java/org/apache/hadoop/hbase/LocalHBaseCluster.java
Outdated
Show resolved
Hide resolved
hbase-server/src/test/java/org/apache/hadoop/hbase/TestClusterWithFixedMasterPorts.java
Outdated
Show resolved
Hide resolved
|
🎊 +1 overall
This message was automatically generated. |
bharathv
force-pushed
the
HBASE-23259
branch
from
7ffa90c
to
a9d7824
Compare
ndimiduk
reviewed
Nov 12, 2019
| * @param noRegionServers Number of region servers. | ||
| * @param masterClass Impl of master class | ||
| * @param regionServerClass Impl of RS class | ||
| * @param masterPorts Array of ports, 2 per master (RPC/INFO) to use. |
There was a problem hiding this comment.
What's the use-case for specifying the explicit ports to use? This design has a code-smell to me -- a test should grab whatever ports are available on the system and provide an interface for handing that information off to clients, not the other way around.
Why just offer the master ports as random? If this is something we want to support in tests, seems like we need a LocalHBaseClusterBuilder that lets me build up a configuration, adding port pairs for master instances and region server instances (and possible zookeeper, data node, name node, &c).
There was a problem hiding this comment.
I think I over thought about this. I've undone all of these changes and made the patch quite simple. LMK what you think.
To give you some context, the reason I did this change is to have the master end points (comma separated host:port values) populated in the region server configs so that they can use the newly created master based registry to create the internal connections. The issue though is that since the masters were using random ports chosen at runtime, we cannot pre-create a config object before spinning up the cluster.
But after spending some more time on it, it looks like we can actually do that easily. The next patch just waits for the masters to be created -> figures out the ports -> updates the conf -> spins up the region servers with that conf.
| (Class<? extends HRegionServer>)conf.getClass(HConstants.REGION_SERVER_IMPL, | ||
| regionServerClass); | ||
| // Every port except the master ports are random. | ||
| conf.set(HConstants.MASTER_INFO_PORT, "0"); |
There was a problem hiding this comment.
You say the caller should specify pairs of ports for RPC/INFO, yet the INFO port is always random. Shouldn't this conf.set be inside the for-loop?
There was a problem hiding this comment.
Re-did the patch to be much simpler. I agree this is confusing.
| @@ -1049,6 +1049,21 @@ public MiniHBaseCluster startMiniCluster(int numSlaves) throws Exception { | |||
| return startMiniCluster(option); | |||
| } | |||
|
|
|||
| /** | |||
| * Start up a minicluster of hbase, dfs and zookeeper clusters with given slave node number and | |||
| * a set of master ports. All other options will use default values, defined in | |||
There was a problem hiding this comment.
"master ports" are not an argument here... I'm missing something?
There was a problem hiding this comment.
Re-did the patch, not valid anymore.
| public MiniHBaseCluster startMiniClusterWithMasterPorts(int numSlaves) throws Exception { | ||
| StartMiniClusterOption option = StartMiniClusterOption.builder() | ||
| .numRegionServers(numSlaves).numDataNodes(numSlaves) | ||
| .masterPorts(randomFreePorts(1)).build(); |
There was a problem hiding this comment.
In the test environment, why is it that we don't use random ports always?
There was a problem hiding this comment.
Re-did the patch to be much simpler. Not valid anymore.
| import org.junit.experimental.categories.Category; | ||
|
|
||
| @Category(SmallTests.class) | ||
| public class TestClusterWithFixedMasterPorts { |
There was a problem hiding this comment.
Yikes! A test for the test code :(
|
🎊 +1 overall
This message was automatically generated. |
All the clients need to know the master RPC end points while using master
based registry for creating cluster connections. This patch amends the
test cluster utility to populate these configs in the base configuration
object used to spin up the cluster.
The config key added here ("hbase.master.addrs") is used in the subsequent
patches for HBASE-18095.
bharathv
force-pushed
the
HBASE-23259
branch
from
a9d7824
to
24aea56
Compare
|
🎊 +1 overall
This message was automatically generated. |
apurtell
reviewed
Nov 16, 2019
| @@ -171,6 +171,11 @@ | |||
| /** Configuration key for master web API port */ | |||
| public static final String MASTER_INFO_PORT = "hbase.master.info.port"; | |||
|
|
|||
| /** Configuration key for the list of master host:ports **/ | |||
| public static final String MASTER_ADDRS_KEY = "hbase.master.addrs"; | |||
There was a problem hiding this comment.
Make sure parsing can handle both and : comma separated values. The other configuration setting like this is the zk quorum, which accepts a comma-separated list of : or . Let's keep things consistent.
There was a problem hiding this comment.
Just to confirm my understanding is correct, you mean the port number is optional right ? Something like "foo.com,bar.com:1234,baz.com:5678,temp.com"
There was a problem hiding this comment.
Looking at our zookeeper connection string parsing code. We depend on the Hadoop Configuration object to handle parsing of the comma-delimited configuration value. We then manually check for ':' characters to split out ports.
If we're making a habit of doing this kind of parsing (lists of socket addresses), it's probably work encapsulating the logic into a single place.
There was a problem hiding this comment.
Yep, I'll consolidate the parsing logic. It will come in the next patch.
There was a problem hiding this comment.
You plan to address this last nit? I'm +1 with this bit handled.
There was a problem hiding this comment.
@ndimiduk The parsing logic will come as a part of PR for HBASE-23305. There is nothing in this patch that consumes the content of "hbase.master.addrs".
hbase-server/src/main/java/org/apache/hadoop/hbase/LocalHBaseCluster.java
Show resolved
Hide resolved
bharathv
commented
Nov 16, 2019
hbase-server/src/main/java/org/apache/hadoop/hbase/LocalHBaseCluster.java
Show resolved
Hide resolved
| @@ -171,6 +171,11 @@ | |||
| /** Configuration key for master web API port */ | |||
| public static final String MASTER_INFO_PORT = "hbase.master.info.port"; | |||
|
|
|||
| /** Configuration key for the list of master host:ports **/ | |||
| public static final String MASTER_ADDRS_KEY = "hbase.master.addrs"; | |||
There was a problem hiding this comment.
Just to confirm my understanding is correct, you mean the port number is optional right ? Something like "foo.com,bar.com:1234,baz.com:5678,temp.com"
bharathv
commented
Nov 18, 2019
| @@ -171,6 +171,11 @@ | |||
| /** Configuration key for master web API port */ | |||
| public static final String MASTER_INFO_PORT = "hbase.master.info.port"; | |||
|
|
|||
| /** Configuration key for the list of master host:ports **/ | |||
| public static final String MASTER_ADDRS_KEY = "hbase.master.addrs"; | |||
There was a problem hiding this comment.
Yep, I'll consolidate the parsing logic. It will come in the next patch.
ndimiduk
approved these changes
Nov 20, 2019
ndimiduk
pushed a commit
to ndimiduk/hbase
that referenced
this pull request
Nov 20, 2019
…apache#807) All the clients need to know the master RPC end points while using master based registry for creating cluster connections. This patch amends the test cluster utility to populate these configs in the base configuration object used to spin up the cluster. The config key added here ("hbase.master.addrs") is used in the subsequent patches for HBASE-18095. Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
|
Patch applies cleanly to Patch has conflicts on
|
|
ya, I can do that. Thanks for taking care of branch-2 |
bharathv
added a commit
to bharathv/hbase
that referenced
this pull request
Nov 20, 2019
…apache#807) All the clients need to know the master RPC end points while using master based registry for creating cluster connections. This patch amends the test cluster utility to populate these configs in the base configuration object used to spin up the cluster. The config key added here ("hbase.master.addrs") is used in the subsequent patches for HBASE-18095. Signed-off-by: Nick Dimiduk <ndimiduk@apache.org> (cherry picked from commit 834ccb4)
bharathv
added a commit
to bharathv/hbase
that referenced
this pull request
Nov 20, 2019
…apache#807) All the clients need to know the master RPC end points while using master based registry for creating cluster connections. This patch amends the test cluster utility to populate these configs in the base configuration object used to spin up the cluster. The config key added here ("hbase.master.addrs") is used in the subsequent patches for HBASE-18095. Signed-off-by: Nick Dimiduk <ndimiduk@apache.org> (cherry picked from commit 834ccb4)
bharathv
added a commit
to bharathv/hbase
that referenced
this pull request
Nov 20, 2019
…apache#807) All the clients need to know the master RPC end points while using master based registry for creating cluster connections. This patch amends the test cluster utility to populate these configs in the base configuration object used to spin up the cluster. The config key added here ("hbase.master.addrs") is used in the subsequent patches for HBASE-18095. Signed-off-by: Nick Dimiduk <ndimiduk@apache.org> (cherry picked from commit 834ccb4)
ndimiduk
added a commit
that referenced
this pull request
Nov 21, 2019
…#807) (#857) All the clients need to know the master RPC end points while using master based registry for creating cluster connections. This patch amends the test cluster utility to populate these configs in the base configuration object used to spin up the cluster. The config key added here ("hbase.master.addrs") is used in the subsequent patches for HBASE-18095. Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
ndimiduk
pushed a commit
that referenced
this pull request
Nov 21, 2019
…#807) (#858) All the clients need to know the master RPC end points while using master based registry for creating cluster connections. This patch amends the test cluster utility to populate these configs in the base configuration object used to spin up the cluster. The config key added here ("hbase.master.addrs") is used in the subsequent patches for HBASE-18095. Signed-off-by: Nick Dimiduk <ndimiduk@apache.org> (cherry picked from commit 834ccb4)
taklwu
added a commit
to taklwu/hbase
that referenced
this pull request
Jan 26, 2020
* HBASE-23136 (#712)
* HBASE-23136 PartionedMobFileCompactor bulkloaded files shouldn't get replicated (addressing buklload replication related issue raised in HBASE-22380)
Signed-off-by: Josh Elser <elserj@apache.org>
* HBASE-23042 Parameters are incorrect in procedures jsp (#728)
* HBASE-23172 HBase Canary region success count metrics reflect column family successes, not region successes
* HBASE-23012 Add HBase 1.3.6 to the downloads page (#738)
* HBASE-22679 : Revamping CellUtil (#735)
* HBASE-22679 : Revamping CellUtil
* checkstyle fix
* incorporating review
* minor indentation
* HBASE-22460 : Reopen regions with very high Store Ref Counts (#600)
Signed-off-by Anoop Sam John <anoopsamjohn@apache.org>
* HBASE-15519 Add per-user metrics with lossy counting
Introducing property hbase.regionserver.user.metrics.enabled(Default:true)
to disable user metrics in case it accounts for any performance issues
Close #661
Signed-off-by: Josh Elser <elserj@apache.org>
* HBASE-23207 Log a region open journal (#751)
Signed-off-by: Abhishek Singh Chouhan <achouhan@apache.org>
* HBASE-23073 Add an optional costFunction to balance regions according to a capacity rule (#677)
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
* HBASE-23012 Addendum - Update 1.3.6 Release date in the downloads page
* HBASE-23203 NPE in RSGroup info (#747)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
* HBASE-23194 Remove unused methods from TokenUtil (#737)
Signed-off-by: Peter Somogyi <psomogyi@apache.org>
* HBASE-22991 add HBase 1.4.11 to the downloads page.
closes #756
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
* HBASE-23181 Blocked WAL archive: "LogRoller: Failed to schedule flush of XXXX, because it is not online on us" (#753)
Signed-off-by: Lijin Bin <binlijin@apache.org>
Signed-off-by: stack <stack@apache.org>
* HBASE-23216 Add 2.2.2 to download page (#758)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
* HBASE-15519 Add per-user metrics with lossy counting (addendum)
* HBASE-23222 MOB compaction supportability improvements
* better logging on MOB compaction process
* HFileCleanerDelegate to optionally halt removal of mob hfiles
* use archiving when removing committed mob file after bulkload ref failure
closes #763
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
Signed-off-by: Balazs Meszaros <meszibalu@apache.org>
* HBASE-23199 Error populating Table-Attribute fields (#741)
Signed-off-by: GuangxuCheng <guangxucheng@gmail.com>
* HBASE-23187 Update parent region state to SPLIT in meta (#732)
* HBASE-23187 Update parent region state to SPLIT in meta
* Revert "HBASE-23194 Remove unused methods from TokenUtil (#737)"
This reverts commit d7b90b319908113bb90ae871cf4a5843bbf6bbaa.
* HBASE-23208 Unit formatting in Master & RS UI
Signed-off-by: binlijin <binlijin@gmail.com>
Signed-off-by: Sean Busbey <busbey@apache.org>
* HBASE-23227 Upgrade jackson-databind to 2.9.10.1
Signed-off-by: Sean Busbey <busbey@apache.org>
* HBASE-20827 Use backoff on CallQueueTooBigException when reporting region state transition
Signed-off-by: Josh Elser <elserj@apache.org>
* HBASE-23191 EOFE log spam (#733)
Convert log message added for 2.2.0 from INFO to DEBUG.
Signed-off-by: stack <stack@apache.org>
* HBASE-22917 Proc-WAL roll fails saying someone else has already created log (#544)
Signed-off-by: stack <stack@apache.org>
* HBASE-23192 CatalogJanitor consistencyCheck does not log problematic row on exception (#734)
Adds logging of row and complaint if consistency check fails during CJ
checking. Adds a few more null checks. Does edit on the 'HBCK Report'
top line.
Signed-off-by: Reid Chan <reidchan@apache.org>
* HBASE-22739 ArrayIndexOutOfBoundsException when balance (#729)
Signed-off-by: stack <stack@apache.org>
* HBASE-23175 Yarn unable to acquire delegation token for HBase Spark jobs
* HBASE-23231 ReplicationSource do not update metrics after refresh (#778)
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
* HBASE-23231 Addendum remove redundant metrics.clear in ReplicationSource
This is only a problem on master which was introduced when we merge sync replication feature branch back
* HBASE-23184 The HeapAllocation in WebUI is not accurate (#730)
Signed-off-by: stack <stack@apache.org>
* HBASE-23221 Polish the WAL interface after HBASE-23181 (#774)
Removes the closeRegion flag added by HBASE-23181 and instead
relies on reading meta WALEdit content. Modified how qualifier is
written when the meta WALEdit is for a RegionEventDescriptor
so the 'type' is added to the qualifer so can figure type
w/o having to deserialize protobuf value content: e.g.
HBASE::REGION_EVENT::REGION_CLOSE
Added doc on WALEdit and tried to formalize the 'meta' WALEdit
type and how it works. Needs complete redo in part as suggested
by HBASE-8457. Meantime, some doc and cleanup.
Also changed the LogRoller constructor to remove redundant param.
Because of constructor change, need to change also
TestFailedAppendAndSync, TestWALLockup, TestAsyncFSWAL &
WALPerformanceEvaluation.java
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Lijin Bin <binlijin@apache.org>
* Revert "HBASE-22917 Proc-WAL roll fails saying someone else has already created log (#544)"
This reverts commit 538a4c51ff8464f57e26cc22f9c559c1a30b5864.
* HBASE-23238 Additional test and checks for null references on ScannerCallableWithReplicas (#780)
Signed-off-by: Sean Busbey <busbey@apache.org>
(cherry picked from commit 577db5d7e50c56b4773c9ce92b807aae80bf5706 - Test only, removed changes from no more exisiting ScannerCallableWithReplicas class)
* HBASE-23244 NPEs running Canary (#784)
Signed-off-by: Viraj Jasani <virajjasani007@gmail.com>
* HBASE-23244 NPEs running Canary (#784)
Addendum to fix findbugs complaint.
* HBASE-23241 TestExecutorService sometimes fail (#782)
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
* HBASE-23247 [hbck2] Schedule SCPs for 'Unknown Servers' (#791)
Signed-off-by: Sean Busbey <busbey@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
* HBASE-23243 [pv2] Filter out SUCCESS procedures; on decent-sized cluster, plethora overwhelms problems (#790)
Signed-off-by: GuangxuCheng <guangxucheng@gmail.com>
Signed-off-by: Sean Busbey <busbey@apache.org>
* HBASE-23212 : Dynamically reload configs for Region Recovery chore (#773)
* HBASE-23212 : Dynamically reload configs for Region Recovery chore
* remove redundant volatile
* HBASE-23250 Log message about CleanerChore delegate initialization should be at INFO
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
* HBASE-21458 Error: Could not find or load main class org.apache.hadoop.hbase.util.GetJavaProperty
Signed-off-by: Sean Busbey <busbey@apache.org>
* HBASE-23085 Network and Data related Actions
Add monkey actions:
- manipulate network packages with tc (reorder, loose,...)
- add CPU load
- fill the disk
- corrupt or delete regionserver data files
Extend HBaseClusterManager to allow sudo calls.
Signed-off-by: Josh Elser <elserj@apache.org>
Signed-off-by: Balazs Meszaros <meszibalu@apache.org>
* HBASE-22980 HRegionPartioner getPartition() method incorrectly partitions the regions of the table. (#590)
Signed-off-by: Guangxu Cheng <guangxucheng@gmail.com>
* HBASE-23230 Enforce member visibility in HRegionServer (#775)
* Clean up a bunch of private variable leakage into other
classes. Reduces visibility as much as possible, providing getters
where access remains necessary or making use of getters that
already exist. There remains an insidious relationship between
`HRegionServer` and `RSRpcServices`.
* Rename `fs` to `dataFs`, `rootDir` as `dataRootDir` so as to
distinguish from the new `walFs`, `walRootDir` (and make it easier
to spot bugs).
* Cleanup or delete a bunch of lack-luster javadoc comments.
* Delete a handful of methods that are unused according to static
analysis.
* Reduces the warning count as reported by IntelliJ from 100 to 7.
Signed-off-by: stack <stack@apache.org>
* HBASE-23272 Fix link in Developer guide to "code review checklist" (#805)
Signed-off-by: stack <stack@apache.org>
* HBASE-22480 Get block from BlockCache once and return this block to BlockCache twice make ref count error.
* HBASE-23262 Cannot load Master UI (#799)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
* HBASE-23263 NPE in Quotas.jsp (#800)
Signed-off-by: Guangxu Cheng <guangxucheng@gmail.com>
* HBASE-23257: Track clusterID in stand by masters (#798)
This patch implements a simple cache that all the masters
can lookup to serve cluster ID to clients. Active HMaster
is still responsible for creating it but all the masters
will read it from fs to serve clients.
RPCs exposing it will come in a separate patch as a part of
HBASE-18095.
Signed-off-by: Andrew Purtell <apurtell@apache.org>
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
Signed-off-by: Guangxu Cheng <guangxucheng@gmail.com>
* HBASE-23236 Upgrade to yetus 0.11.1
Signed-off-by: stack <stack@apache.org>
* HBASE-22888 Share some stuffs with the initial reader when new stream reader created (#581)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
* HBASE-23268 Remove disable/enable ops from doc when altering schema
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
* HBASE-23228 Allow for jdk8 specific modules on branch-1 in precommit/nightly testing (#804)
Signed-off-by: Josh Elser <elserj@apache.org>
* HBASE-18439 Subclasses of o.a.h.h.chaos.actions.Action all use the same logger
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
Signed-off-by: Guangxu Cheng <gxcheng@apache.org>
* HBASE-23251 - Add Column Family and Table Names to HFileContext and use in HFileWriterImpl logging (#796)
Signed-off-by: Andrew Purtell <apurtell@apache.org>
Signed-off-by: Xu Cang <xucang@apache.org>
Signed-off-by: Zheng Hu <openinx@gmail.com>
* HBASE-23245 : MutableHistogram constructor changes and provide HistogramImpl maxExpected as long (#787)
Signed-off-by: Andrew Purtell <apurtell@apache.org>
Signed-off-by: Xu Cang <xucang@apache.org>
Signed-off-by: Guangxu Cheng <gxcheng@apache.org>
* HBASE-23284 Fix Hadoop wiki link in Developer guide to "Distributions and Commercial Support"
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
* HBASE-23290 shell processlist command is broken
* HBASE-23283 Provide clear and consistent logging about the period of enabled chores
Signed-off-by: Sean Busbey <busbey@apache.org>
* HBASE-19450 Add log about average execution time for ScheduledChore
Signed-off-by: Sean Busbey <busbey@apache.org>
* HBASE-23294 ReplicationBarrierCleaner should delete all the barriers for a removed region which does not belong to any serial replication peer (#827)
Signed-off-by: stack <stack@apache.org>
* HBASE-19450 Addendum Limit logging of chore execution time at INFO to once per 5 minutes.
* Ensure MovingAverage related classes are IA.Private
* Move trace logging into MovingAverage class
Signed-off-by: Duo Zhang <zhangduo@apache.org>
* HBASE-23102: Improper Usage of Map putIfAbsent (#828)
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
* HBASE-22969 A new binary component comparator(BinaryComponentComparator) to perform comparison of arbitrary length and position (#829)
Signed-off-by: Balazs Meszaros <meszibalu@apache.org>
* HBASE-23289 Update links to Hadoop wiki in book and code
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
* HBASE-23282 HBCKServerCrashProcedure for 'Unknown Servers'
Have the existing scheduleRecoveries launch a new HBCKSCP
instead of SCP. It gets regions to recover from Master
in-memory context AND from a scan of hbase:meta. This
new HBCKSCP is For processing 'Unknown Servers', servers that
are 'dead' and purged but still have references in
hbase:meta. Rare occurance but needs tooling to address.
Later have catalogjanitor take care of these deviations
between Master in-memory and hbase:meta content (usually
because of overdriven cluster with failed RPCs to hbase:meta,
etc)
Changed expireServers in ServerManager so could pass in
custom reaction to expired server.... This is how we
run our custom HBCKSCP while keeping all other aspects
of expiring services (rather than try replicate it
externally).
* HBASE-23182 The create-release scripts are broken (#736)
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
* HBASE-23318 LoadTestTool doesn't start (#848)
* Package the test jar from hbase-zookeeper into lib/
Signed-off-by: Duo Zhang <zhangduo@apache.org>
* HBASE-23278 Add a table-level compaction progress display on the UI (#816)
Signed-off-by: Guangxu Cheng <gxcheng@apache.org>
* HBASE-23315 Miscellaneous HBCK Report page cleanup
* Add a bit of javadoc around SerialReplicationChecker.
* Miniscule edit to the profiler jsp page and then a bit of doc on how to make it work that might help.
* Add some detail if NPE getting BitSetNode to help w/ debug.
* Change HbckChore to log region names instead of encoded names; helps doing diagnostics; can take region name and query in shell to find out all about the region according to hbase:meta.
* Add some fix-it help inline in the HBCK Report page – how to fix.
* Add counts in procedures page so can see if making progress; move listing of WALs to end of the page.
* HBASE-23308: Review of NullPointerExceptions (#836)
Signed-off-by: stack <stack@apache.org>
* HBASE-22607. TestExportSnapshotNoCluster fails intermittently
* HBASE-23321 [hbck2] fixHoles of fixMeta doesn't update in-memory state
* HBASE-23259: Populate master address end points in cluster/rs configs (#807)
All the clients need to know the master RPC end points while using master
based registry for creating cluster connections. This patch amends the
test cluster utility to populate these configs in the base configuration
object used to spin up the cluster.
The config key added here ("hbase.master.addrs") is used in the subsequent
patches for HBASE-18095.
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
* HBASE-23322 [hbck2] Simplification on HBCKSCP scheduling
Signed-off-by: Lijin Bin <binlijin@apache.org>
* HBASE-23223 Support the offsetLock of bucketCache to use strong ref (#764)
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
* HBASE-23234 Provide .editorconfig based on checkstyle configuration (#846)
This file is generated using IntelliJ, following these steps:
#. Open Preferences > Editor > Code Style
#. Select (config wheel) > Import Schema > CheckStyle Configuration
#. Select `hbase-checkstyle/src/main/resources/hbase/checkstyle.xml`
#. Select (config wheel) > Export > EditorConfig File
Signed-off-by: Sean Busbey <busbey@apache.org>
* HBASE-23237 Prevent Negative values in metrics requestsPerSecond
Closes #834
Signed-off-by: Guangxu Cheng <gxcheng@apache.org>
Signed-off-by: Josh Elser <elserj@apache.org>
* HBASE-23325 [UI]rsgoup average load keep two decimals (#860)
Signed-off-by: Guangxu Cheng <gxcheng@apache.org>
* HBASE-23328 info:regioninfo goes wrong when region replicas enabled (#863)
Signed-off-by: Ramkrishna <ramkrishna.s.vasudevan@intel.com>
Signed-off-by: Guangxu Cheng <guangxucheng@gmail.com>
* HBASE-23329 Remove unused methods from RequestConverter (#870)
Signed-off-by: Sean Busbey <busbey@apache.org>
* HBASE-23307 Add running of ReplicationBarrierCleaner to hbck2 fixMeta invocation (#859)
Signed-off-by: Lijin Bin <binlijin@apache.org>
* HBASE-22969 A new binary component comparator(BinaryComponentComparator) to perform comparison of arbitrary length and position; ADDENDUM (#869)
Signed-off-by: Sean Busbey <busbey@apache.org>
* HBASE-23085 Network and Data related Actions; ADDENDUM (#871)
Fix percentage in String.format
Signed-off-by: Sean Busbey <busbey@apache.org>
* HBASE-23197 'IllegalArgumentException: Wrong FS' on edits replay when… (#740)
Signed-off-by: Josh Elser <elserj@apache.org>
* HBASE-23334 The table-lock node of zk is not needed since HBASE-16786 (#873)
Signed-off-by: Guangxu Cheng <gxcheng@apache.org>
* HBASE-23312 HBase Thrift SPNEGO configs (HBASE-19852) should be backwards compatible
HBase Thrift SPNEGO configs should not be required.
The `hbase.thrift.spnego.keytab.file` and
`hbase.thrift.spnego.principal` configs should fall
back to the `hbase.thrift.keytab.file` and
`hbase.thrift.kerberos.principal` configs. This will
avoid any issues during upgrades.
Signed-off-by: Josh Elser <elserj@apache.org>
Amending-author: Josh Elser <elserj@apache.org>
Closes #850
* HBASE-23293 [REPLICATION] make ship edits timeout configurable (#825)
Signed-off-by: Guangxu Cheng <gxcheng@apache.org>
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
* HBASE-23336 [CLI] Incorrect row(s) count 'clear_deadservers' (#875)
Signed-off-by: Guangxu Cheng <gxcheng@apache.org>
Signed-off-by: Lijin Bin <binlijin@apache.org>
* HBASE-20395 Displaying thrift server type on the thrift page (#811)
Signed-off-by: Lijin Bin <binlijin@apache.org>
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
* HBASE-23117: Bad enum in hbase:meta info:state column can fail loadMeta and stop startup (#867)
* Handling the BAD value in info:state columns in hbase:meta
* Adding a unit test and region encoded name in the LOG
* Adding a null check for region state to complete the test scenario and fixing the nit
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
Signed-off-by: GuangxuCheng <guangxucheng@gmail.com>
Signed-off-by: stack <stack@apache.org>
* HBASE-23313 [hbck2] setRegionState should update Master in-memory sta… (#864)
Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: stack <stack@apache.org>
* HBASE-20395 Addendum Displaying thrift server type on the thrift page
* HBASE-23323 Update downloads page for Apache HBase 1.4.12. (#886)
Signed-off-by: stack <stack@apache.org>
* HBASE-23352: Allow chaos monkeys to access cmd line params, and improve FillDiskCommandAction (#885)
Instead of using the default properties when checking for monkey
properties, now we use the ones already extended with command line
params.
Change FillDiskCommandAction to try to stop the remote process if the
command failed with an exception.
Signed-off-by: stack <stack@apache.org>
* HBASE-23342 : Handle NPE while closing compressingStream (#877)
Signed-off-by Anoop Sam John <anoopsamjohn@apache.org>
* HBASE-23298 Refactor LogRecoveredEditsOutputSink and BoundedLogWriterCreationOutputSink (#832)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
* HBASE-23335 : Improving cost functions array copy in StochasticLoadBalancer (#874)
* HBASE-23337 Release scripts should rely on maven for deploy. (#887)
- switch to nexus-staging-maven-plugin for asf-release
- refactor release-build to use mvn deploy and its output.
- cleaned up some tabs in the root pom
Signed-off-by: stack <stack@apache.org>
* HBASE-23345 Table need to replication unless all of cfs are excluded (#881)
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Guanghao Zhang <zghao@apache.org>
* HBASE-22096 /storeFile.jsp shows CorruptHFileException when the storeFile is a reference file (#888)
Signed-off-by: Lijin Bin <binlijin@apache.org>
* HBASE-23356 When construct StoreScanner throw exceptions it is possible to left some KeyValueScanner not closed. (#891)
Signed-off-by: GuangxuCheng <guangxucheng@gmail.com>
* HBASE-23357 Add 2.1.8 to download page (#892)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
* HBASE-23362: [WalPrettyPrinter] print/filter by table name. (#898)
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
* HBASE-22529 Add sanity check for in-memory compaction policy
Signed-off-by: Toshihiro Suzuki <brfrn169@gmail.com>
* HBASE-23365 Minor change MemStoreFlusher's log (#900)
Signed-off-by: GuangxuCheng <guangxucheng@gmail.com>
Signed-off-by: Xu Cang <xucang@apache.org>
* HBASE-23361 Limit two decimals in total average load (#897)
Signed-off-by: GuangxuCheng <guangxucheng@gmail.com>
Signed-off-by: Xu Cang <xucang@apache.org>
* HBASE-23367 Remove unused constructor from WALPrettyPrinter (#901)
Signed-off-by: Lijin Bin <binlijin@apache.org>
Signed-off-by: Xu Cang <xucang@apache.org>
* HBASE-23364 HRegionServer sometimes does not shut down.
* HBASE-23373 Log `RetriesExhaustedException` context with full time precision (#903)
Signed-off-by: Lijin Bin <binlijin@apache.org>
* HBASE-23309: Adding the flexibility to ChainWalEntryFilter to filter the whole entry if all cells get filtered (#837)
HBASE-23309: Adding the flexibility to ChainWalEntryFilter to filter the whole entry if all cells get filtered
* HBASE-23303 Add security headers to REST server/info page (#843)
Signed-off-by: Toshihiro Suzuki <brfrn169@gmail.com>
Signed-off-by: Sean Busbey <busbey@apache.org>
* HBASE-22280 Separate read/write handler for priority request(especial… (#202)
Signed-off-by: Yu Li <liyu@apache.org>
* HBASE-18382 add transport type info into Thrift UI (#880)
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
Signed-off-by: Viraj Jasani <virajjasani007@gmail.com>
* HBASE-23377 Balancer should skip disabled tables's regions (#908)
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
Signed-off-by: GuangxuCheng <guangxucheng@gmail.com>
Signed-off-by Anoop Sam John <anoopsamjohn@apache.org>
Signed-off-by: Reid Chan <reidchan@apache.org>
* HBASE-23066 Allow cache on write during compactions when prefetching is (#919)
Contributed by : Jacob LeBlanc
* HBASE-23552 Format Javadocs on ITBLL
We have this nice description in the java doc on ITBLL but it's
unformatted and thus illegible. Add some formatting so that it can be
read by humans.
Signed-off-by: Jan Hentschel <janh@apache.org>
Signed-off-by: Josh Elser <elserj@apache.org>
* HBASE-23379 Clean Up FSUtil getRegionLocalityMappingFromFS
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
* HBASE-23556: Minor ChoreService Cleanup (#927)
* HBASE-23553 Snapshot referenced data files are deleted in some case (#922)
* HBASE-23360 [CLI] Fix help command 'set_quota' for removing limits (#896)
* Revert "HBASE-23066 Allow cache on write during compactions when prefetching is (#919)"
TestCacheOnWrite failing all the time.
This reverts commit d561130e82c5956f0dd9fff5c6f6240a686d3d6a.
* HBASE-23554 Encoded regionname to regionname utility (#923)
Adds shell command regioninfo:
hbase(main):001:0> regioninfo '0e6aa5c19ae2b2627649dc7708ce27d0'
{ENCODED => 0e6aa5c19ae2b2627649dc7708ce27d0, NAME => 'TestTable,,1575941375972.0e6aa5c19ae2b2627649dc7708ce27d0.', STARTKEY => '', ENDKEY => '00000000000000000000299441'}
Took 0.4737 seconds
Signed-off-by: Sean Busbey <busbey@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
* HBASE-23554 Encoded regionname to regionname utility (#923); ADDENDUM
* HBASE-23555 TestQuotaThrottle is broken (#924)
* HBASE-23566: Fix package/packet terminology problem in chaos monkeys (#933)
s/package/packet/g
Signed-off-by: Sean Busbey <busbey@apache.org>
* HBASE-23570 Point users to the async-profiler home page if diagrams are coming up blank (#937)
* HBASE-23575 Remove dead code in AsyncRegistry (#929)
Removes a bunch of dead code and fixes some checkstyle nits.
Signed-off-by: Viraj Jasani <virajjasani007@gmail.com>
Signed-off-by: Sean Busbey <busbey@apache.org>
* HBASE-23380 General cleanup of FSUtil (#912)
* Clean up JavaDocs
* Clean up logging and error messages
* Remove superfluous code
* Replace static code with library call
* Do not swallow Interrupted Exceptions
* Use try-with-resources
* User multi-Exception catches to reduce code size
Signed-off-by: Jan Hentschel <janh@apache.org>
Signed-off-by: Sean Busbey <busbey@apache.org>
* HBASE-22920 github pr testing job should use dev-support script (#883)
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Sean Busbey <busbey@apache.org>
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
* HBASE-23582 Unbalanced braces in string representation of table descriptor
Signed-off-by: Lijin Bin <binlijin@apache.org>
Signed-off-by: Jan Hentschel <janh@apache.org>
* HBASE-23066 Allow cache on write during compactions when prefetching … (#935)
* HBASE-23066 Allow cache on write during compactions when prefetching is enabled
* Fix checkstyle issues - recommit
* HBASE-23065 [hbtop] Top-N heavy hitter user and client drill downs
Signed-off-by: Toshihiro Suzuki <brfrn169@gmail.com>
Signed-off-by: Josh Elser <elserj@apache.org>
Signed-off-by: Andrew Purtell <apurtell@apache.org>
* HBASE-23575 Remove dead code in AsyncRegistry (addendum)
Additions to MiniHBaseCluster needed for branch-2.
Signed-off-by: Jan Hentschel <janh@apache.org>
Signed-off-by: Xu Cang <xucang@apache.org>
Signed-off-by: Sean Busbey <busbey@apache.org>
Signed-off-by: Viraj Jasani <virajjasani007@gmail.com>
* HBASE-23239 Reporting on status of backing MOB files from client-facing cells (#785)
* Adds a new MapReduce job that builds a report of health of mob files
* Also builds background information on mob system use
* add a basic mob architecture in the ref guide to explain how mob takes the reference hfile value and finds the actual cell contents
* add a troubleshooting mob section to the ref guide to explain how to do a mob reference scan.
Signed-off-by: Peter Somogyi <psomogyi@apache.org>
* HBASE-23549 Document steps to disable MOB for a column family (#928)
Signed-off-by: Peter Somogyi <psomogyi@apache.org>
Signed-off-by: Josh Elser <elserj@apache.org>
* HBASE-20461 Implement fsync for AsyncFSWAL (#947)
Signed-off-by: stack <stack@apache.org>
* HBASE-23376 NPE happens while replica region is moving (#906)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
* HBASE-23594 Procedure stuck due to region happen to recorded on two servers. (#953)
Signed-off-by: stack <stack@apache.org>
* HBASE-23564 RegionStates may has some expired serverinfo and make regions do not balance. (#930)
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
* HBASE-23572 In 'HBCK Report', distringush between live, dead, and unknown servers
Signed-off-by: Sean Busbey <busbey@apache.org>
* HBASE-23320 Upgrade surefire plugin to 3.0.0-M4
Signed-off-by: Peter Somogyi <psomogyi@apache.org>
* HBASE-23581 Creating table gets stuck when specifying an invalid split policy as METADATA (#942)
Signed-off-by: Lijin Bin <binlijin@apache.org>
Signed-off-by: Anoop Sam John <anoopsamjohn@apacher.org>
Signed-off-by: Xu Cang <xucang@apache.org>
* HBASE-23589: FlushDescriptor contains non-matching family/output combinations (#949)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Lijin Bin <binlijin@apache.org>
* HBASE-23613 ProcedureExecutor check StuckWorkers blocked by DeadServe… (#960)
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
* HBASE-23326 Implement a ProcedureStore which stores procedures in a HRegion (#941)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
Signed-off-by: stack <stack@apache.org>
* HBASE-23374 ExclusiveMemHFileBlock’s allocator should not be hardcoded as ByteBuffAllocator.HEAP
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Jan Hentschel <janh@apache.org>
* HBASE-23238: Remove 'static'ness of cell counter in LimitKVsReturnFilter (addendum) (#963)
Having it as static means the test cannot be parameterized (ran into
this issue in HBASE-23305). That happens because the field is not
reset between parameterized runs.
* HBASE-23286 Improve MTTR: Split WAL to HFile (#820)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
* HBASE-23619 Used built-in formatting for logger in hbase-zookeeper
Signed-off-by: stack <stack@apache.org>
* Adding developer details to pom.xml
* HBASE-23617 Add a stress test tool for region based procedure store (#962)
Signed-off-by: stack <stack@apache.org>
* HBASE-23621 Reduced the number of Checkstyle violations in tests of hbase-common
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
* HBASE-23622 Reduced the number of Checkstyle violations in hbase-common
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
* HBASE-23618 Add a tool to dump procedure info in the WAL file (#969)
Signed-off-by: stack <stack@apache.org>
* HBASE-23618 Addendum add main method
* HBASE-23626 Reduced number of Checkstyle violations in tests in hbase-common
Signed-off-by: Viraj Jasani <vjasani@apache.org>
* HBASE-23615 Use a dedicated thread for executing WorkerMonitor in Pro… (#961)
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: virajjasani <34790606+virajjasani@users.noreply.github.com>
* HBASE-23590 : Update maxStoreFileRefCount to maxCompactedStoreFileRefCount for auto region recovery based on old reader references
Signed-off-by: Anoop Sam John <anoopsamjohn@apache.org>
* HBASE-23625 Reduced number of Checkstyle violations in hbase-common
Signed-off-by: stack <stack@apache.org>
* HBASE-23627 Resolved remaining Checkstyle violations in hbase-thrift
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: stack <stack@apache.org>
* HBASE-23623 Reduced the number of Checkstyle violations in hbase-rest
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
* HBASE-23624 Add a tool to dump the procedure info in HFile (#975)
Signed-off-by: stack <stack@apache.org>
* HBASE-23629: Add to 'Supporting Projects' in site (#976)
* Update supportingprojects.xml
Signed-off-by: Jan Hentschel <janh@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
* HBASE-23588 : Cache index & bloom blocks on write if CacheCompactedBlocksOnWrite is enabled
Signed-off-by: ramkrish86 <ramkrishna@apache.org>
Signed-off-by: chenxu14 <47170471+chenxu14@users.noreply.github.com>
* HBASE-23628: Remove Apache Commons Digest Base64 (#977)
Signed-off-by: stack <stack@apache.org>
* HBASE-23596 HBCKServerCrashProcedure can double assign
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Lijin Bin <binlijin@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Change its behavior so it will only look in hbase:meta
if the call to the super class turns up zero references.
Only then will it search hbase:meta for references to
'Unknown Servers'. Normal operation where we read Master
context is usual and sufficient. The scan of hbase:meta
is only for case where Master state has been corrupted
and we need to clear out 'Unknown Servers'.
* HBASE-23632 DeadServer cleanup (#979)
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
* HBASE-23098 [bulkload] If one of the peers in a cluster is configured with NAMESPACE level, its hfile-refs(zk) will be backlogged (#676)
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
Signed-off-by: stack <stack@apache.org>
* HBASE-23587 The FSYNC_WAL flag does not work on branch-2.x (#974)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
Signed-off-by: stack <stack@apache.org>
* HBASE-23369 Auto-close 'unknown' Regions reported as OPEN on RegionServers
Master force-closes unknown/incorrect Regions OPEN on RS
M hbase-client/src/main/java/org/apache/hadoop/hbase/MetaTableAccessor.java
Added a note and small refactor.
M hbase-server/src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java
Fix an NPE when CJ ran.
M hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
Minor clean up of log message; make it clearer.
M hbase-server/src/main/java/org/apache/hadoop/hbase/master/ServerManager.java
Make it so closeRegionSilentlyAndWait can be used w/o timeout.
M hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/AssignmentManager.java
If a RegionServer Report notes a Region is OPEN and the Master does not
know of said Region, close it (We used to crash out the RegionServer)
M hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/RegionStateNode.java
Minor tweak of toString -- label should be state, not rit (confusing).
M hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/RegionStates.java
Doc.
M hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/TransitRegionStateProcedure.java
Add region name to exception.
M hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/HBCKServerCrashProcedure.java
Be more careful about which Regions we queue up for reassign. This
procedure is run by the operator so could happen at any time. We
will likely be running this when Master has some accounting of
cluster members so check its answers for what Regions were on
server before running.
M hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
Doc and we were misrepresenting the case where a Region as not in RIT
when we got CLOSE -- we were reporting it as though it was already
trying to CLOSE.
* HBASE-23333 Include Call.toShortString() in sendCall exceptions
* HBASE-23369 Auto-close 'unknown' Regions reported as OPEN on RegionServers; ADDENDUM on master
* HBASE-23635 Reduced number of Checkstyle violations in hbase-mapreduce
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: stack <stack@apache.org>
* HBASE-23588 : Cache index & bloom blocks on write (ADDENDUM)
Signed-off-by: Anoop Sam John <anoopsamjohn@apache.org>
* HBASE-23636 Disable table may hang when regionserver stop or abort. (#982)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: virajjasani <34790606+virajjasani@users.noreply.github.com>
* HBASE-23645 Fixed remaining Checkstyle violations in hbase-common tests
Signed-off-by: Viraj Jasani <vjasani@apache.org>
* HBASE-22908 Link To HBase 1.4 Documentation on HBase Site (#993)
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Sean Busbey <busbey@apache.org>
* HBASE-23651 Region balance throttling can be disabled (#991)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Anoop Sam John <anoopsamjohn@apache.org>
* HBASE-23660 hbase:meta's table.jsp ref to wrong rs address (#999)
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
* HBASE-23658 Fix flaky TestSnapshotFromMaster (#998)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
* HBASE-23663 Allow dot and hyphen in Profiler's URL (#1002)
Signed-off-by: Sean Busbey <busbey@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
* HBASE-23655 Fix flaky TestRSGroupsKillRS: should wait the SCP to finish (#996)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
* HBASE-23654 Adding Apache Trafodion and EsgynDB to 'Powered by Apache HBase' page
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Jan Hentschel <janh@apache.org>
* HBASE-23664 Upgrade JUnit to 4.13 (#1004)
Signed-off-by: Sean Busbey <busbey@apache.org>
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
Signed-off-by: Jan Hentschel <janh@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
* HBASE-22285 A normalizer which merges small size regions with adjacent regions (#978)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: stack <stack@apache.org>
* HBASE-23378: Clean Up FSUtil setClusterId (#910)
Signed-off-by: stack <stack@apache.org>
* HBASE-23055 Alter hbase:meta
Make it so hbase:meta can be altered. TableState for hbase:meta
was hardcoded ENABLED. Make it dynamic. State is now kept in
current active Master. It is transient so falls back to default
if Master crashes. Add to registry a getMetaTableState
which reads mirrored state from zookeeper (NOT from Master and
defaults ENABLED if no implementation or error fetching state).
hbase:meta schema will be bootstrapped from the filesystem.
Changes to filesystem schema are atomic so we should be ok if
Master fails mid-edit (TBD). Undoes a bunch of guards that
prevented our being able to edit hbase:meta.
TODO: Tests, more clarity around hbase:meta table state, and undoing
references to hard-coded hbase:meta regioninfo.
M hbase-client/src/main/java/org/apache/hadoop/hbase/MetaTableAccessor.java
Throw illegal access exception if you try to use MetaTableAccessor
getting state of the hbase:meta table.
M hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionImplementation.java
Add fetching of hbase:meta table state from registry. Adds cache of
tablestates w/ a ttl of 1 second (adjustable).
M hbase-client/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
M hbase-client/src/main/java/org/apache/hadoop/hbase/client/RawAsyncHBaseAdmin.java
Add querying registry for hbase:meta table state.
M hbase-client/src/main/java/org/apache/hadoop/hbase/client/ZKAsyncRegistry.java
Add querying of mirrored table state for hbase:meta table.
M hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/ZNodePaths.java
Shutdown access.
M hbase-server/src/main/java/org/apache/hadoop/hbase/TableDescriptors.java
Just cleanup.
M hbase-server/src/main/java/org/apache/hadoop/hbase/master/TableStateManager.java
Add state holder for hbase:meta. Removed unused methods.
M hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/RegionStateStore.java
Shut down access.
M hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/DisableTableProcedure.java
Allow hbase:meta to be disabled.
M hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/EnableTableProcedure.java
Allow hbase:meta to be enabled.
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
* HBASE-23659 BaseLoadBalancer#wouldLowerAvailability should consider region replicas (#1001)
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
* HBASE-23668 Master log start filling with "Flush journal status" messages"
This reverts commit fb9fa04da72379431d13f22a7e5d8e75ae1267be.
i.e. reapplication of patch that was preamaturely applied.
Signed-off-by: Duo Zhang <zhangduo@apache.org>
* Revert "HBASE-23668 Master log start filling with "Flush journal status" messages"
Minor addendum fixing log message.
* HBASE-23662 : Replace HColumnDescriptor(String cf) with ColumnFamilyDescriptor
Signed-off-by: Peter Somogyi <psomogyi@apache.org>
Signed-off-by: Jan Hentschel <janh@apache.org>
* HBASE-23675 Move to Apache parent POM version 22
* HBASE-23165 [hbtop] Some modifications from HBASE-22988 (#987)
Signed-off-by: stack <stack@apache.org>
* Revert "HBASE-23055 Alter hbase:meta"
This reverts commit 9abdb7b5ae4de0b6d839f5c5d85e9bb899b5edbd.
* HBASE-23646 Resolved remaining Checkstyle violations in tests of hbase-rest
Signed-off-by: Viraj Jasani <vjasani@apache.org>
* HBASE-23681 Add UT for procedure store region flusher (#1024)
Signed-off-by: stack <stack@apache.org>
* HBASE-23662 : Keep HCD constructor until shell usages are replaced
* HBASE-23679 FileSystem objects leak when cleaned up in cleanupBulkLoad
The cleanupBulkLoad method is only called for the first Region in the
table which was being bulk loaded into. This means that potentially N-1
other RegionServers (where N is the number of RegionServers) will leak
one FileSystem object into the FileSystem cache which will never be
cleaned up. We need to do this clean-up as a part of secureBulkLoadHFiles
otherwise we cannot guarantee that heap usage won't grow unbounded.
Closes #1029
Signed-off-by: Sean Busbey <busbey@apache.org>
* HBASE-23383 [hbck2] `fixHoles` should queue assignment procedures for any regions its fixing (#917)
The current process for an operator, after fixing holes in meta, is to
manually disable and enable the whole table. Let's try to avoid
bringing the whole table offline if we can. Have the master attempt to
queue up assignment procedures for any new regions it creates.
Signed-off-by: stack <stack@apache.org>
* HBASE-23687 DEBUG logging cleanup (#1040)
Signed-off-by: Jan Hentschel <janh@apache.org>
* HBASE-23689: Bookmark for github PR to jira redirection (#1042)
Signed-off-by: stack <stack@apache.org>
* HBASE-23688 Update docs for setting up IntelliJ as a development environment (#1041)
Signed-off-by: stack <stack@apache.org>
* HBASE-23569 : Validate that all default chores of HMaster are scheduled
Signed-off-by: Andrew Purtell <apurtell@apache.org>
* HBASE-23691 Add 2.2.3 to download page (#1045)
Signed-off-by: Jan Hentschel <janh@apache.org>
* fix 500/NPE of region.jsp (#1033)
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
* HBASE-23683 Make HBaseInterClusterReplicationEndpoint more extensible (#1027)
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
Signed-off-by: Josh Elser <elserj@apache.org>
Signed-off-by: binlijin <binlijin@gmail.com>
* HBASE-23665: Split unit tests from TestTableName into a separate test-only class. (#1032)
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
* Revert "fix 500/NPE of region.jsp (#1033)"
Missed the jira number on that commit message. Will re-apply it with the jira number.
This reverts commit d60ce17c1765a445e944738f49953579bdf0bba6.
* HBASE-23677 fix 500/NPE of region.jsp (#1033)
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
* HBASE-23674 Too many rit page Numbers show confusion
* HBASE-23694 After RegionProcedureStore completes migration of WALProcedureStore, still running WALProcedureStore.syncThread keeps trying to delete now inexistent log files. (#1048)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
* HBASE-23652 Move the unsupported procedure type check before migrating to RegionProcedureStore (#1018)
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
* HBASE-23695 Fail gracefully if no category is present
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
Closes #1052
* HBASE-23347 Allow custom authentication methods for RPCs
Decouple the HBase internals such that someone can implement
their own SASL-based authentication mechanism and plug it into
HBase RegionServers/Masters.
Comes with a design doc in dev-support/design-docs and an example in
hbase-examples known as "Shade" which uses a flat-password file
for authenticating users.
Closes #884
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
Signed-off-by: Andrew Purtell <apurtell@apache.org>
Signed-off-by: Reid Chan <reidchan@apache.org>
* HBASE-23653 Expose content of meta table in web ui (#1020)
Adds a display of the content of 'hbase:meta' to the Master's
table.jsp, when that table is selected. Supports basic pagination,
filtering, &c.
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
* HBASE-23569 : Validate that all default chores of HRegionServer are scheduled (ADDENDUM)
Signed-off-by: Andrew Purtell <apurtell@apache.org>
* HBASE-23703 Add HBase 2.2.3 documentation to website (#1059)
Signed-off-by: Jan Hentschel <janh@apache.org>
* HBASE-23701 Try to converge automated checks around Category
We have a compile-time guarantee to either have a category array
of length zero or one because the category annotation is non-repeatable.
Also tries to clean up the checking for unit-test annotations.
Closes #1057
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
* HBASE-23690 Checkstyle plugin complains about our checkstyle.xml format; doc how to resolve mismatched version (#1044)
Signed-off-by: Jan Hentschel <janh@apache.org>
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
* HBASE-23612 Add new profile to make hbase build success on ARM (#959)
* HBASE-23700 Upgrade checkstyle and plugin versions (#1056)
Bump checkstyle version to 8.28, maven-checkstyle-plugin to 3.1.0.
As per HBASE-23242 and the updated checkstyle docs[1], the LineLength
check should be placed under an instance of Checker.
[1] https://checkstyle.sourceforge.io/config_sizes.html#LineLength
Co-authored-by: Bharath Vissapragada <bharathv@apache.org>
Signed-off-by: Jan Hentschel <janh@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
* HBASE-23653 Expose content of meta table in web ui; addendum (#1061)
Fix error prone problem
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: stack <stack@apache.org>
* HBASE-23680 RegionProcedureStore missing cleaning of hfile archive (#1022)
Signed-off-by: stack <stack@apache.org>
* HBASE-23661 Reduced number of Checkstyle violations in hbase-rest
Signed-off-by: Peter Somogyi <psomogyi@apache.org>
Signed-off-by: Sean Busbey <busbey@apache.org>
* HBASE-23686 Revert binary incompatible change in ByteRangeUtils and removed reflections in CommonFSUtils
Signed-off-by: Sean Busbey <busbey@apache.org>
* HBASE-23156 start-hbase.sh failed with ClassNotFoundException when build with hadoop3 (#1067)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
* HBASE-23347 Allow custom authentication methods for RPCs; addendum (#1060)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
* HBASE-23055 Alter hbase:meta (#1043)
Make hbase:meta region schema dynamic.
Patch has been under development a good while and its focus has changed
a few times so its bloated with fixup from older versions.
M hbase-server/src/main/java/org/apache/hadoop/hbase/master/TableStateManager.java
M hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/ZNodePaths.java
Shut down access to internals and removed unused methods.
M hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/EnableTableProcedure.java
Cleanup/refactor section on replica-handling.
M hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSTableDescriptors.java
Get hbase:meta schema from filesystem rather than from hard-coding.
* HBASE-20516 Offheap read-path needs more detail (#1081)
Signed-off-by: Anoop Sam John <anoopsamjohn@apacher.org>
* HBASE-23711 - Add test for MinVersions and KeepDeletedCells TTL (#1079)
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
* HBASE-23709 Unwrap the real user to properly dispatch proxy-user auth'n
REST and Thrift servers started failing because the check in
BuiltinProviderSelector wasn't checking the "real" user for kerberos
credentials. This resulted in the KerberosAuthnProvider not being
invoked when it should have been.
Closes #1080
Signed-off-by: Peter Somogyi <psomogyi@apache.org>
* HBASE-23719 Add 1.5.0 release to Downloads (#1083)
Signed-off-by: Jan Hentschel <janh@apache.org>
* HBASE-23720 [create-release] Update yetus version used from 0.11.0 to 0.11.1
* HBASE-23705 Add CellComparator to HFileContext (#1062)
Codecs don't have access to what CellComparator to use. Backfill.
M hbase-common/src/main/java/org/apache/hadoop/hbase/CellComparator.java
Adds a new compareRows with default implementation that takes a ByteBuffer.
Needed by the index in a block encoder implementation.
M hbase-common/src/main/java/org/apache/hadoop/hbase/CellComparatorImpl.java
Adds implementation for meta of new compareRows method. Adds utility
method for figuring comparator based off tablename.
M hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/AbstractDataBlockEncoder.java
M hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/BufferedDataBlockEncoder.java
M hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/RowIndexCodecV1.java
M hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/RowIndexSeekerV1.java
Comparator is in context. Remove redundant handling.
M hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/DataBlockEncoder.java
Comparator is in context. Remove redundant handling. Clean javadoc.
M hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/HFileBlockDecodingContext.java
Clean javadoc.
M hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/RowIndexEncoderV1.java
Cache context so can use it to get comparator to use later.
M hbase-common/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileContext.java
Cache cellcomparator to use. Javdoc on diff between HFileContext and
HFileInfo.
M hbase-common/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileContextBuilder.java
Add CellComparator
M hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat2.java
M hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java
M hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderImpl.java
M hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterImpl.java
M hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileWriter.java
Remove comparator caching. Get from context instead.
M hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/FixedFileTrailer.java
Skip a reflection if we can.
M hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileInfo.java
Javadoc. Removed unused filed.
Signed-off-by: Anoop Sam John <anoopsamjohn@apacher.org>
Signed-off-by: Ramkrishna <ramkrishna.s.vasudevan@intel.com>
Signed-off-by: Jan Hentschel <janh@apache.org>
* HBASE-23715 MasterFileSystem should not create MasterProcWALs dir on … (#1078)
Signed-off-by: Josh Elser <elserj@apache.org>
* HBASE-23069 periodic dependency bump for Sep 2019 (#1082)
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
* HBASE-21065 Try ROW_INDEX_V1 encoding on meta table (fix bloomfilters… (#1012)
Set encoding and blooms on meta as default. Also shutdown access to the
initial meta schema creating method; get from TableDescriptors if you
need access to schema or edit it as you would any other table if you
want to edit it.
Signed-off-by: Viraj Jasani <vjasani@apache.org>
* HBASE-23069 periodic dependency bump for Sep 2019 (#1082); ADDENDUM
For asciidoctor, s/1.5.8/1.5.8.1/.
* HBASE-23722 Real user might be null in non-proxy-user case
Closes #1085
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
* HBASE-23069 periodic dependency bump for Sep 2019 (#1082); ADDENDUM
Remove staging repo added by mistake.
* HBASE-23069 periodic dependency bump for Sep 2019 (#1082); ADDENDUM
AND.... undo thirdparty testing version update.
* HBASE-23710 - Priority configuration for system coprocessors (#1077)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
* HBASE-23729 [Flakeys] TestRSGroupsBasics#testClearNotProcessedDeadServer fails most of the time
* HBASE-23733 [Flakey Tests] TestSplitTransactionOnCluster
* HBASE-23726 Forward-port HBASE-21345 to branch-2.2, 2.3 & master as well.
HBASE-21345 - [hbck2] Allow version check to proceed even though master is 'initializing'.
Just remove the check state from the getClusterStatus call.
Signed-off-by: Michael Stack <stack@apache.org>
Signed-off-by: Peter Somogyi <psomogyi@apache.org>
Signed-off-by: Sakthi <sakthi@apache.org>
(cherry picked from commit dd8496a5460693c49ec0bf5475ef79e40457e6bd)
* HBASE-23735 [Flakey Tests] TestClusterRestartFailover & TestClusterRestartFailoverSplitWithoutZk
* HBASE-23737 [Flakey Tests] TestFavoredNodeTableImport fails 30% of the time; AMENDMENT
This is actual fix; previous added debug to test.
* HBASE-23737 [Flakey Tests] TestFavoredNodeTableImport fails 30% of the time
Second part of this issue -- changes to the test.
Co-authored-by: Wellington Ramos Chevreuil <wellington.chevreuil@gmail.com>
Co-authored-by: meiyi <myimeiyi@gmail.com>
Co-authored-by: Sakthi <sakthivel.azhaku@gmail.com>
Co-authored-by: Viraj Jasani <vjasani@apache.org>
Co-authored-by: Ankit Singhal <ankitsinghal59@gmail.com>
Co-authored-by: Andrew Purtell <apurtell@apache.org>
Co-authored-by: Pierre Zemb <contact@pierrezemb.fr>
Co-authored-by: Karthik Palanisamy <kpalanisamy@hortonworks.com>
Co-authored-by: Sean Busbey <sean.busbey@gmail.com>
Co-authored-by: Duo Zhang <palomino219@gmail.com>
Co-authored-by: Peter Somogyi <psomogyi@apache.org>
Co-authored-by: binlijin <binlijin@gmail.com>
Co-authored-by: Wei-Chiu Chuang <jojochuang@gmail.com>
Co-authored-by: Pankaj <pankaj.kr@huawei.com>
Co-authored-by: Michael Stack <saintstack@users.noreply.github.com>
Co-authored-by: chenxu14 <47170471+chenxu14@users.noreply.github.com>
Co-authored-by: ravowlga123 <ravowlga@gmail.com>
Co-authored-by: BukrosSzabolcs <szabolcs@cloudera.com>
Co-authored-by: Shardul Singh <shardulsingh22@gmail.com>
Co-authored-by: Nick Dimiduk <ndimiduk@gmail.com>
Co-authored-by: Bharath Vissapragada <bharathv@apache.org>
Co-authored-by: Dice <poleon.kd@gmail.com>
Co-authored-by: Geoffrey Jacoby <gjacoby@apache.org>
Co-authored-by: Mingliang Liu <liuml07@gmail.com>
Co-authored-by: Reid Chan <reidchan@apache.org>
Co-authored-by: belugabehr <12578579+belugabehr@users.noreply.github.com>
Co-authored-by: Udai Bhan Kashyap <ukashyap1@bloomberg.net>
Co-authored-by: Baiqiang Zhao <zbq.dean@gmail.com>
Co-authored-by: bsglz <18031031@qq.com>
Co-authored-by: xuqinya1 <45840773+xuqinya1@users.noreply.github.com>
Co-authored-by: Kevin Risden <risdenk@users.noreply.github.com>
Co-authored-by: Guangxu Cheng <gxcheng@apache.org>
Co-authored-by: Sandeep Pal <50725353+sandeepvinayak@users.noreply.github.com>
Co-authored-by: Guanghao Zhang <zghaobac@gmail.com>
Co-authored-by: XinSun <ddupgs@gmail.com>
Co-authored-by: Toshihiro Suzuki <brfrn169@gmail.com>
Co-authored-by: Junegunn Choi <junegunn.c@gmail.com>
Co-authored-by: lhofhansl <lhofhansl@yahoo.com>
Co-authored-by: Andor Molnár <andor@apache.org>
Co-authored-by: Beata Sudi <bea0113@users.noreply.github.com>
Co-authored-by: ramkrish86 <ram_krish_86@hotmail.com>
Co-authored-by: Jan Hentschel <jan.hentschel@ultratendency.com>
Co-authored-by: Manu Manjunath <manu.m.karnataki@gmail.com>
Co-authored-by: Yiran Wu <yiranpinkroom@gmail.com>
Co-authored-by: Aman Poonia <aman.poonia.29@gmail.com>
Co-authored-by: Josh Elser <josh.elser@gmail.com>
Co-authored-by: WenFeiYi <wenfeiyi666@gmail.com>
Co-authored-by: bzhaoopenstack <bzhaojyathousandy@gmail.com>
infraio
pushed a commit
to infraio/hbase
that referenced
this pull request
Aug 17, 2020
…apache#807) (apache#857) All the clients need to know the master RPC end points while using master based registry for creating cluster connections. This patch amends the test cluster utility to populate these configs in the base configuration object used to spin up the cluster. The config key added here ("hbase.master.addrs") is used in the subsequent patches for HBASE-18095. Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This patch adds the plumbing needed to start the mini cluster with
a pre-determined set of master ports. This is required for HBASE-18095
because the internal client 'Connection's from region servers need
to know the master RPC end points in their configuration even before
the mini cluster starts.
Following patches on HBASE-18095 will use this plumbing for unit tests.