HAWQ-1473 - document ranger plugin service high availability (closes #…

…120)

book/master_middleman/source/subnavs/apache-hawq-nav.erb

@@ -195,6 +195,9 @@
                   <li>
                      <a href="/docs/userguide/2.2.0.0-incubating/ranger/ranger-auditing.html">Auditing Authorization Events</a>
                   </li>
+                  <li>
+                     <a href="/docs/userguide/2.2.0.0-incubating/ranger/ranger-ha.html">High Availability and HAWQ Ranger</a>
+                  </li>
                </ul>
           </li>
           <li>
@@ -705,6 +708,7 @@
                   <li><a href="/docs/userguide/2.2.0.0-incubating/reference/guc/parameter_definitions.html#hawq_rm_yarn_queue_name">hawq_rm_yarn_queue_name</a></li>
                   <li><a href="/docs/userguide/2.2.0.0-incubating/reference/guc/parameter_definitions.html#hawq_rm_yarn_scheduler_address">hawq_rm_yarn_scheduler_address</a></li>
                   <li><a href="/docs/userguide/2.2.0.0-incubating/reference/guc/parameter_definitions.html#hawq_rps_address_port">hawq_rps_address_port</a></li>
+                  <li><a href="/docs/userguide/2.2.0.0-incubating/reference/guc/parameter_definitions.html#hawq_rps_check_local_interval">hawq_rps_check_local_interval</a></li>
                   <li><a href="/docs/userguide/2.2.0.0-incubating/reference/guc/parameter_definitions.html#hawq_segment_address_port">hawq_segment_address_port</a></li>
                   <li><a href="/docs/userguide/2.2.0.0-incubating/reference/guc/parameter_definitions.html#hawq_segment_directory">hawq_segment_directory</a></li>
                   <li><a href="/docs/userguide/2.2.0.0-incubating/reference/guc/parameter_definitions.html#hawq_segment_temp_directory">hawq_segment_temp_directory</a></li>

markdown/admin/MasterMirroring.html.md.erb

@@ -103,6 +103,8 @@ Upon activation of the standby master, HAWQ reconstructs the state of the master
 	```
 
 	The newly-activated master's status should be **Active**. If you configured a new standby master, its status is **Passive**. When a standby master is not configured, the command displays `-No entries found`, the message indicating that no standby master instance is configured.
+
+6. If you have enabled HAWQ Ranger Authentication in high availability mode in your cluster, you must manually update the Ranger HAWQ service definition to identify the new master node. Refer to [Failover to HAWQ Standby Master](../ranger/ranger-ha.html#rps_ha_cfg_masterfailover) in the HAWQ Ranger documentation for additional information.
 
 6. Query the `gp_segment_configuration` table to verify that segments have registered themselves to the new master:
 

markdown/admin/ambari-admin.html.md.erb

@@ -284,10 +284,12 @@ Activating the HAWQ Standby Master promotes the standby host as the new HAWQ Mas
 5.  Ambari displays the host name of the current HAWQ Master that will be removed from the cluster, as well as the HAWQ Standby Master host that will be activated. The information is provided only for review and cannot be edited on this page. Click **Next** to confirm the operation.
 6. Click **OK** to confirm that you want to perform the procedure, as it is not possible to roll back the operation using Ambari.
 
-   Ambari displays a list of tasks that are performed to activate the standby server and remove the previous HAWQ Master host. Click on any of the tasks to view progress or to view the actual log messages that are generated while performing the task.
+    Ambari displays a list of tasks that are performed to activate the standby server and remove the previous HAWQ Master host. Click on any of the tasks to view progress or to view the actual log messages that are generated while performing the task.
 7. Click **Complete** after the Wizard finishes all tasks.
 
-   **Important:** After the Wizard completes, your HAWQ cluster no longer includes a HAWQ Standby Master host. As a best practice, follow the instructions in [Adding a HAWQ Standby Master](#amb-add-standby) to configure a new one.
+    **Important:** After the Wizard completes, your HAWQ cluster no longer includes a HAWQ Standby Master host. As a best practice, follow the instructions in [Adding a HAWQ Standby Master](#amb-add-standby) to configure a new one.
+
+8. If your cluster employs HAWQ Ranger Authentication in high availability mode, you must manually update the HAWQ service definition in Ranger to identify the new master node. Refer to [Failover to HAWQ Standby Master](../ranger/ranger-ha.html#rps_ha_cfg_masterfailover) in the HAWQ Ranger documentation for additional information.
 
 ## <a id="amb-add-standby"></a>Adding a HAWQ Standby Master
 

markdown/ranger/ranger-ha.html.md.erb

@@ -0,0 +1,45 @@
+---
+title: High Availability and HAWQ Ranger
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+  http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+This topic describes specific HAWQ Ranger failover scenarios and procedures. You must have registered and configured a standby HAWQ Ranger Plug-in Service as described in [Register a Standby Ranger Plug-in Service](ranger-integration-config.html#enableha) for these failover scenarios to occur.
+
+
+## Failover to Standby Ranger Plug-in Service <a id="rps_ha_cfg_rpsfailover"></a>
+
+Should the HAWQ master node fail to communicate with the local Ranger Plug-in Service, the master automatically switches over to the Ranger Plug-in Service registered on the HAWQ master standby node. This operation should be transparent to all HAWQ users.
+
+`hawq state cluster` command output identifies the master service state as `Down` when the Ranger Plug-in Service has failed over to the master standby node:
+
+``` shell
+20170515:01:15:26:045131 hawq_state:master:gpadmin-[INFO]:--   HAWQ master Ranger plugin service state      = Down
+20170515:01:15:27:045131 hawq_state:master:gpadmin-[INFO]:--   HAWQ standby Ranger plugin service state     = Active
+```
+
+After failover to the standby Ranger Plug-in Service, the HAWQ master periodically attempts to re-establish contact with the service on the local node. The [`hawq_rps_check_local_interval`](../reference/guc/parameter_definitions.html#hawq_rps_check_local_interval) server configuration parameter identifies the polling time interval for this contact. When communication is restored with the Ranger Plug-in Service on the local node, the HAWQ master automatically switches back to the local service. This operation is similarly transparent to all HAWQ users.
+
+
+## Failover to HAWQ Standby Master <a id="rps_ha_cfg_masterfailover"></a>
+
+If the HAWQ master node goes down, you will activate the master standby node, at which time the standby becomes the new HAWQ master. When the HAWQ master fails over in this manner, the master Ranger Plug-in Service also fails over to the standby node.
+
+After activating a HAWQ standby master node, you must manually update the Ranger HAWQ service definition with the new HAWQ master node connection information. Update this information via the Ranger Admin UI.

markdown/ranger/ranger-integration-config.html.md.erb

@@ -129,6 +129,63 @@ Once the connection between HAWQ and Ranger is configured, you may choose to set
 5. Click **Save** to save your changes.
 6. Select **Service Actions > Restart All** and confirm that you want to restart the HAWQ cluster.
 
+## <a id="enableha"></a>Step 3: (Optional) Register a Standby Ranger Plug-in Service
+
+The HAWQ Ranger Plug-in Service runs on the HAWQ master node. If this service goes down, all HAWQ database operations will fail. Configure a highly available HAWQ Ranger Plug-in Service to eliminate possible downtime should this situation occur. The standby Ranger Plug-in Service runs on the HAWQ standby master node, utilizing the same port number as that when the service runs on the master node. To enable HAWQ Ranger high availability, you must register the standby Ranger Plug-in Service on the standby master node, and then restart the standby.
+
+Configuring both Ranger Adminstration host and HAWQ Ranger Plug-in Service high availability is advised.  However, Ranger Administration host high availability and HAWQ Ranger Plug-in Service high availability are independent; you can configure HAWQ Ranger Plug-in Service HA without configuring HA for the Ranger Administration host.
+
+
+### Prerequisites <a id="rps_ha_cfg_prereq"></a>
+
+Before you configure HAWQ Ranger authentication in high availability mode, ensure that you have:
+
+- (Optional) Configured the Ranger Admininstration host for high availability.
+
+- Configured a HAWQ standby master node for your HAWQ cluster.
+
+    You must configure a standby master for your HAWQ deployment before enabling HAWQ Ranger high availability mode. If you have not configured your HAWQ standby master, follow the instructions in [Adding a HAWQ Standby Master](../admin/ambari-admin.html#amb-add-standby) (if you manage your HAWQ cluster with Ambari) or [Configuring Master Mirroring](../admin/MasterMirroring.html#standby_master_configure) (for a command-line-managed HAWQ cluster).
+
+- Registered the HAWQ Ranger Plug-in Service on your HAWQ master node.
+
+    The HAWQ Ranger Plug-in Service runs on the HAWQ master node. If you have not yet enabled the Ranger Plug-in Service, refer to [Install Ranger Connectivity to HAWQ](ranger-integration-config.html#jar) for registration instructions. (Optional) If you have configured Ranger Administration host HA, make sure to identify the Ranger Administration host HA proxy when you enable the plug-in.
+
+**Note**: If you configured and registered the master HAWQ Ranger Plug-in Service before you initialized your HAWQ standby master node, you do not need to perform the steps in this section.
+
+
+### Procedure <a id="rps_ha_cfg_standbyrps"></a>
+
+**Note**: If you configured and registered the master HAWQ Ranger Plug-in Service before you initialized your HAWQ standby master node, you do not need to perform the steps in this section.
+
+
+1. Synchronize the HAWQ Ranger Plug-in Service configuration files from the HAWQ master node to the standby master node by either manually copying the files, or by running `enable-ranger-plugin.sh` on the standby master. For example (where `$GPHOME` represents your base HAWQ install directory):
+
+    ``` shell
+	gpadmin@master$ scp $GPHOME/ranger/etc/* gpadmin@standby:$GPHOME/ranger/etc/
+	```
+
+	If you choose to execute `enable-ranger-plugin.sh` on the HAWQ standby master, provide the same arguments you used in your invocation of this command on the HAWQ master node:
+
+	``` shell
+	gpadmin@standby$ $GPHOME/ranger/bin/enable-ranger-plugin.sh -r <ranger_admin_node>:<ranger_port> -u <ranger_user> -p <ranger_password> -h <hawq_master>:<hawq_port> -w <hawq_user> -q <hawq_password>
+	```
+
+2. Restart the HAWQ standby master node. You will perform different procedures depending upon whether you manage your HAWQ cluster from the command line or you use Ambari to manage your cluster.
+
+    If you manage your HAWQ cluster from the command line:
+
+    ``` shell
+    gpadmin@master$ hawq stop standby
+    gpadmin@master$ hawq start standby
+    ```
+
+    If you manage your HAWQ cluster with Ambari:
+
+    1. Follow the instructions in [Removing the HAWQ Standby Master](../admin/ambari-admin.html#amb-remove-standby) to remove the HAWQ standby master.
+    2. Follow the instructions in [Adding a HAWQ Standby Master](../admin/ambari-admin.html#amb-add-standby) to re-add the HAWQ standby master.
+
+
+
 
 ## <a id="rpsadminstate"></a>Displaying the Status of HAWQ/Ranger Integration
 
@@ -147,11 +204,17 @@ Determine the status of HAWQ/Ranger integration in your cluster by:
     ``` shell
     gpadmin@master$ hawq state
     ...
-    20170327:16:35:06:508426 hawq_state:master:gpadmin-[INFO]:--   Current HAWQ acl type                          = ranger
-    20170327:16:35:06:508426 hawq_state:master:gpadmin-[INFO]:--   HAWQ Ranger plugin service state               = Active
+    20170327:16:35:06:508426 hawq_state:master:gpadmin-[INFO]:--   Current HAWQ acl type                         = ranger
+    20170327:16:35:06:508426 hawq_state:master:gpadmin-[INFO]:--   HAWQ master Ranger plugin service state       = Active
     ...
     ```
 
+    If you have registered a standby Ranger Plug-in Service, `hawq state` also displays the status of that standby service:
+
+    ``` shell
+    20170327:16:35:06:508426 hawq_state:master:gpadmin-[INFO]:--   HAWQ standby Ranger plugin service state      = Active
+    ```
+
 ## <a id="customconfig"></a> Custom Configuration
 
 Configuration files for the HAWQ Ranger Plug-in Service are located in the `$GPHOME/ranger/etc` directory. These files include:

markdown/ranger/ranger-overview.html.md.erb

@@ -43,9 +43,5 @@ In this release, HAWQ integration with Ranger has several limitations:
 
 - Neither Kerberos authentication nor SSL encryption is supported between a HAWQ node and the Ranger plug-in service, or between the plug-in service and the Ranger Policy Manager.
 
-- The Ranger plug-in service is not compatible with Highly-Available HAWQ deployments. Should you need to activate the standby master in your HAWQ cluster, you must manually update the HAWQ Ranger service definition with the new master node connection information.
-
-- Ranger User Group policies cannot be used with HAWQ in this release. Only User Policies are currently supported.
-
 - Some authorization checks for superuser-restricted authorization events are handled by HAWQ natively, even when Ranger integration is enabled. See [HAWQ-Native Authorization](../clientaccess/hawq-access-checks.html#alwaysnative).
 

markdown/ranger/ranger-policy-creation.html.md.erb

@@ -85,8 +85,6 @@ You can identify one or more users and/or groups to which a policy provides or d
 | Group | \<group-name\> | The group(s) to which you want to provide or deny access. All groups sync'd from \<ranger-admin-node\> are available in the picklist. |
 | User | \<user-name\> | The user(s) to which you want to provide or deny access. All users sync'd from \<ranger-admin-node\> or explicitly registered via the Ranger Admin UI are available in the picklist.  |
 
-**Note**: Group-based assignment of policies is not yet supported in HAWQ. Assign policies to users only.
-
 #### <a id="conditionperms"></a> Permissions
 
 You can assign users the following permissions for allowing or denying access to specific HAWQ resources:

markdown/reference/cli/admin_utilities/hawqactivate.html.md.erb

@@ -23,7 +23,7 @@ under the License.
 
 Activates a standby master host and makes it the active master for the HAWQ system.
 
-**Note:** If HAWQ was installed using Ambari, do not use `hawq activate` to activate a standby master host. The system catalogs could become unsynchronized if you mix Ambari and command line functions. For Ambari-managed HAWQ clusters, always use the Ambari administration interface to activate a standby master. For more information, see [Manging HAWQ Using Ambari](../../../admin/ambari-admin.html#topic1).
+**Note:** If HAWQ was installed using Ambari, do not use `hawq activate` to activate a standby master host. The system catalogs could become unsynchronized if you mix Ambari and command line functions. For Ambari-managed HAWQ clusters, always use the Ambari administration interface to activate a standby master. For more information, see [Managing HAWQ Using Ambari](../../../admin/ambari-admin.html#topic1).
 
 ## <a id="topic1__section2"></a>Synopsis
 
@@ -89,6 +89,10 @@ Immediate shutdown aborts transactions in progress and kills all `postgres` proc
 <dt>-h, -\\\-help (help)  </dt>
 <dd>Displays the online help.</dd>
 
+## <a id="topic1__section515"></a>Notes
+
+If you have enabled HAWQ Ranger Authentication in high availability mode in your cluster, you must perform an additional configuration procedure after activating a HAWQ standby master node. Refer to [Failover to HAWQ Standby Master](../../../ranger/ranger-ha.html#rps_ha_cfg_masterfailover) in the HAWQ Ranger documentation for additional information.
+
 ## <a id="topic1__section5"></a>Example
 
 Activate the standby master host and make it the active master instance for a HAWQ system (run from backup master host you are activating):

markdown/reference/guc/guc_category-list.html.md.erb

@@ -389,6 +389,7 @@ These parameters control certain aspects of Ranger configuration, including enab
 
 -   [hawq\_acl\_type](parameter_definitions.html#hawq_acl_type)
 -   [hawq\_rps\_address\_port](parameter_definitions.html#hawq_rps_address_port)
+-   [hawq\_rps\_check\_local\_interval](parameter_definitions.html#hawq_rps_check_local_interval)
 
 
 

markdown/reference/guc/parameter_definitions.html.md.erb

@@ -328,6 +328,8 @@ Descriptions of the HAWQ server configuration parameters listed alphabetically.
 
 -   **[hawq\_rps\_address\_port](../../reference/guc/parameter_definitions.html#hawq_rps_address_port)**
 
+-   **[hawq\_rps\_check\_local\_interval](../../reference/guc/parameter_definitions.html#hawq_rps_check_local_interval)**
+
 -   **[hawq\_segment\_address\_port](../../reference/guc/parameter_definitions.html#hawq_segment_address_port)**
 
 -   **[hawq\_segment\_directory](../../reference/guc/parameter_definitions.html#hawq_segment_directory)**
@@ -2147,6 +2149,14 @@ Identifies the port on which the HAWQ Ranger Plug-in Service runs. The `hawq_rps
 |-------------------------------------------------------------------------|---------------------|-------------------------|
 | valid port number | 8432 | master, reload |
 
+## <a name="hawq_rps_check_local_interval"></a>hawq\_rps\_check\_local\_interval
+
+Should the HAWQ master node fail to communicate with the local Ranger Plug-in Service and the Ranger Plug-in Service is active on the standby master node, HAWQ attempts to switch back to the service located on the master node as soon as it becomes available. The HAWQ master periodically attempts to re-establish contact with the service on the local node, using `hawq_rps_check_local_interval` as the polling time interval (in seconds) for this contact.
+
+| Value Range                                                             | Default             | Set Classifications     |
+|-------------------------------------------------------------------------|---------------------|-------------------------|
+| 1-65563 | 300 | master, reload |
+
 
 ## <a name="hawq_segment_address_port"></a>hawq\_segment\_address\_port