Skip to content
Permalink
Browse files
Formatting
  • Loading branch information
saintstack committed Oct 23, 2018
1 parent 5811ff2 commit eb7b20585d8b5c466d39ea8600761add617b5b9a
Showing 1 changed file with 16 additions and 8 deletions.
@@ -28,7 +28,8 @@ _HBCK2_ to generate the _HBCK2_ jar file, running the below will dump out the _H
$ HBASE_CLASSPATH_PREFIX=./hbase-hbck2-1.0.0-SNAPSHOT.jar ./bin/hbase org.apache.hbase.HBCK2
~~~~

```usage: HBCK2 [OPTIONS] COMMAND <ARGS>
```
usage: HBCK2 [OPTIONS] COMMAND <ARGS>
Options:
-d,--debug run with debug output
@@ -138,7 +139,8 @@ and is not letting go.

_STUCK_ Procedures look like this:

```2018-09-12 15:29:06,558 WARN org.apache.hadoop.hbase.master.assignment.AssignmentManager: STUCK Region-In-Transition rit=OPENING, location=va1001.example.org,22101,1536173230599, table=IntegrationTestBigLinkedList_20180626110336, region=dbdb56242f17610c46ea044f7a42895b
```
2018-09-12 15:29:06,558 WARN org.apache.hadoop.hbase.master.assignment.AssignmentManager: STUCK Region-In-Transition rit=OPENING, location=va1001.example.org,22101,1536173230599, table=IntegrationTestBigLinkedList_20180626110336, region=dbdb56242f17610c46ea044f7a42895b
```


@@ -170,7 +172,8 @@ is needed to alieve the blockage.

Lists of locks and procedures can also be obtained via the hbase shell:

```$ echo "list_locks"| hbase shell &> /tmp/locks.txt
```
$ echo "list_locks"| hbase shell &> /tmp/locks.txt
$ echo "list_procedures"| hbase shell &> /tmp/procedures.txt
```

@@ -181,7 +184,8 @@ It can be run with a table focus or against the whole cluster.

For example, to check cluster assigns:

```$ hbase canary -f false -t 6000000 &>/tmp/canary.log
```
$ hbase canary -f false -t 6000000 &>/tmp/canary.log
```

The _-f false_ tells the Canary to keep going across failed Region
@@ -194,7 +198,8 @@ For example, given a Region that has a start row of _d1dddd0c_
belonging to the table _testtable_, do as follows:


```hbase> scan 'testtable', {STARTROW => 'd1dddd0c', LIMIT => 10}
```
hbase> scan 'testtable', {STARTROW => 'd1dddd0c', LIMIT => 10}
```

For an overview on parsing a Region name into its constituent parts, see
@@ -207,7 +212,8 @@ _ENABLED_ or _ENABLING_ table, read the _hbase:meta_ table _info:state_ column.
For example, to find the state of all Regions in the table
_IntegrationTestBigLinkedList_20180626064758_, do the following:

```$ echo " scan 'hbase:meta', {ROWPREFIXFILTER => 'IntegrationTestBigLinkedList_20180626064758,', COLUMN => 'info:state'}"| hbase shell > /tmp/t.txt
```
$ echo " scan 'hbase:meta', {ROWPREFIXFILTER => 'IntegrationTestBigLinkedList_20180626064758,', COLUMN => 'info:state'}"| hbase shell > /tmp/t.txt
```

...then grep for _OPENING_ or _CLOSING_ Regions.
@@ -273,13 +279,15 @@ current list of outstanding Locks.

This should never happen. If it does, here is what it looks like:

```2018-10-01 22:07:42,792 WARN org.apache.hadoop.hbase.master.HMaster: hbase:meta,,1.1588230740 is NOT online; state={1588230740 state=CLOSING, ts=1538456302300, server=ve1017.example.org,22101,1538449648131}; ServerCrashProcedures=true. Master startup cannot progress, in holding-pattern until region onlined.
```
2018-10-01 22:07:42,792 WARN org.apache.hadoop.hbase.master.HMaster: hbase:meta,,1.1588230740 is NOT online; state={1588230740 state=CLOSING, ts=1538456302300, server=ve1017.example.org,22101,1538449648131}; ServerCrashProcedures=true. Master startup cannot progress, in holding-pattern until region onlined.
```

The Master is unable to continue startup because there is no Procedure to assign
_hbase:meta_ (or _hbase:namespace_). To inject one, use the _HBCK2_ tool:

``` HBASE_CLASSPATH_PREFIX=./hbase-hbck2-1.0.0-SNAPSHOT.jar hbase org.apache.hbase.HBCK2 assigns 1588230740
```
HBASE_CLASSPATH_PREFIX=./hbase-hbck2-1.0.0-SNAPSHOT.jar hbase org.apache.hbase.HBCK2 assigns 1588230740
```

...where 1588230740 is the encoded name of the _hbase:meta_ Region.

0 comments on commit eb7b205

Please sign in to comment.