Add Apache Ranger Authorization

distribution/bin/check-licenses.py

@@ -224,6 +224,7 @@ def build_compatible_license_names():
     compatible_licenses['Apache License Version 2'] = 'Apache License version 2.0'
     compatible_licenses['Apache License v2.0'] = 'Apache License version 2.0'
     compatible_licenses['Apache License, version 2.0'] = 'Apache License version 2.0'
+    compatible_licenses['Apache 2.0 License'] = 'Apache License version 2.0'
 
     compatible_licenses['Public Domain'] = 'Public Domain'
 
@@ -241,6 +242,7 @@ def build_compatible_license_names():
     compatible_licenses['Revised BSD'] = 'BSD-3-Clause License'
     compatible_licenses['New BSD License'] = 'BSD-3-Clause License'
     compatible_licenses['3-Clause BSD License'] = 'BSD-3-Clause License'
+    compatible_licenses['BSD 3-Clause'] = 'BSD-3-Clause License'
 
     compatible_licenses['ICU License'] = 'ICU License'
 
@@ -256,9 +258,11 @@ def build_compatible_license_names():
     compatible_licenses['The Eclipse Public License, Version 1.0'] = 'Eclipse Public License 1.0'
     compatible_licenses['Eclipse Public License - Version 1.0'] = 'Eclipse Public License 1.0'
     compatible_licenses['Eclipse Public License, Version 1.0'] = 'Eclipse Public License 1.0'
+    compatible_licenses['Eclipse Public License v1.0'] = 'Eclipse Public License 1.0'
 
     compatible_licenses['Eclipse Distribution License 1.0'] = 'Eclipse Distribution License 1.0'
     compatible_licenses['Eclipse Distribution License - v 1.0'] = 'Eclipse Distribution License 1.0'
+    compatible_licenses['Eclipse Distribution License v. 1.0'] = 'Eclipse Distribution License 1.0'
     compatible_licenses['EDL 1.0'] = 'Eclipse Distribution License 1.0'
 
     compatible_licenses['Mozilla Public License Version 2.0'] = 'Mozilla Public License Version 2.0'

distribution/pom.xml

@@ -242,6 +242,8 @@
                                         <argument>org.apache.druid.extensions:druid-basic-security</argument>
                                         <argument>-c</argument>
                                         <argument>org.apache.druid.extensions:druid-pac4j</argument>
+                                        <argument>-c</argument>
+                                        <argument>org.apache.druid.extensions:druid-ranger-security</argument>
                                         <argument>${druid.distribution.pulldeps.opts}</argument>
                                     </arguments>
                                 </configuration>

docs/development/extensions-core/druid-ranger-security.md

@@ -0,0 +1,158 @@
+---
+id: druid-ranger-security
+title: "Apache Ranger Security"
+---
+
+<!--
+  ~ Licensed to the Apache Software Foundation (ASF) under one
+  ~ or more contributor license agreements.  See the NOTICE file
+  ~ distributed with this work for additional information
+  ~ regarding copyright ownership.  The ASF licenses this file
+  ~ to you under the Apache License, Version 2.0 (the
+  ~ "License"); you may not use this file except in compliance
+  ~ with the License.  You may obtain a copy of the License at
+  ~
+  ~   http://www.apache.org/licenses/LICENSE-2.0
+  ~
+  ~ Unless required by applicable law or agreed to in writing,
+  ~ software distributed under the License is distributed on an
+  ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+  ~ KIND, either express or implied.  See the License for the
+  ~ specific language governing permissions and limitations
+  ~ under the License.
+  -->
+
+This Apache Druid extension adds:
+
+- an Authorizer which implements access control for the Druid metastore against Apache Ranger
+
+Make sure to [include](../../development/extensions.md#loading-extensions) `druid-ranger-security` as an extension.
+
+Please see [Authentication and Authorization](../../design/auth.md) for more information on the extension interfaces being implemented.
+
+## Configuration
+
+Support for Apache Ranger authorization consists of three elements: configuration of the extension 
+in Apache Druid, configuring the connection to Apache Ranger and providing the service definition for Druid to Apache Ranger.
+
+### Enabling the extension
+Ensure that you have a valid authentication chain and escalator set in your `common.runtime.properties`. For every 
+authenticator your wish to use the authorizer for set `druid.auth.authenticator.<authenticatorName>.authorizerName` 
+to the name you will give the authorizer, e.g. `ranger`. 
+
+Then add the following and amend to your needs (in case you use multiple authorizers):
+
+```
+druid.auth.authorizers=["ranger"]
+druid.auth.authorizer.ranger.type=ranger
+```
+
+The following is an example that uses `druid-basic-security` for authentication and `druid-ranger-security` for 
+authorization.
+
+```
+druid.auth.authenticatorChain=["basic"]
+druid.auth.authenticator.basic.type=basic
+druid.auth.authenticator.basic.initialAdminPassword=password1
+druid.auth.authenticator.basic.initialInternalClientPassword=password2
+druid.auth.authenticator.basic.credentialsValidator.type=metadata
+druid.auth.authenticator.basic.skipOnFailure=false
+druid.auth.authenticator.basic.enableCacheNotifications=true
+druid.auth.authenticator.basic.authorizerName=ranger
+
+druid.auth.authorizers=["ranger"]
+druid.auth.authorizer.ranger.type=ranger
+
+# Escalator
+druid.escalator.type=basic
+druid.escalator.internalClientUsername=druid_system
+druid.escalator.internalClientPassword=password2
+druid.escalator.authorizerName=ranger
+```
+
+---
+**NOTE**
+
+Contrary to the documentation of `druid-basic-auth` Ranger does not automatically provision a highly privileged
+system system user and you will need to do this yourself. This system user in case of `druid-basic-auth` is named 
+`druid_system` and for the escalator it is configurable as shown above. Make sure to take note of these user names and 
+configure `READ` access to `state:STATE` and to `config:security` in your ranger policies, 
+otherwise system services will not work properly.
+---
+
+#### Properties to configure the extension in Apache Druid
+|Property|Description|Default|required|
+|--------|-----------|-------|--------|
+|`druid.auth.ranger.keytab`|Defines the keytab to be used while authenticating against Apache Ranger to obtain policies and provide auditing|null|No|
+|`druid.auth.ranger.principal`|Defines the principal to be used while authenticating against Apache Ranger to obtain policies and provide auditing|null|No|
+|`druid.auth.ranger.use_ugi`|Determines if groups that the authenticated user belongs to should be obtained from Hadoop's `UserGroupInformation`|null|No|
+
+### Configuring the connection to Apache Ranger
+
+The Apache Ranger authorization extension will read several configuration files. Discussing the
+the contents of those files is beyond the scope of this document. Depending on your needs you will 
+need to create them. The minimum you will need to have is a `ranger-druid-security.xml` file 
+that you will need to put in the classpath (e.g. `_common`). For auditing, the configuration is 
+in `ranger-druid-audit.xml`.
+
+### Adding the service definition for Apache Druid to Apache Ranger
+
+At the time of writing of this document Apache Ranger (2.0) does not include a service and 
+service definition yet. You can add the service definition to Apache Ranger by entering the following
+command:
+
+`curl -u <user>:<password> -d "@ranger-servicedef-druid.json" -X POST -H "Accept: application/json" -H "Content-Type: application/json" http://localhost:6080/service/public/v2/api/servicedef/`
+
+You should get back `json` describing the service definition you just added. You can now go to the web
+interface of Apache Ranger which should now include a widget for "Druid". Click the plus sign an create
+the new service. Ensure your service name is equal to what you configured in `ranger-druid-security.xml`.
+
+#### Configuring Apache Ranger policies
+
+When installing a new Druid service inside Apache Ranger for the first time, Ranger will provision the policies
+to allow the administrative user `read/write` access to all properties and data sources. You might want to limit this.
+Do not forget to add the correct policies for the `druid_system` user and the `internalClientUserName` of the escalator.
+
+---
+**NOTE**
+
+Loading new data sources requires `write` access to the `datasource` prior to the loading itself. So if you
+want to create a datasource `wikipedia` you are required to have an `allow` policy inside Apache Ranger before
+trying to load the spec. 
+---
+
+## Usage
+
+### HTTP methods
+
+For information on what HTTP methods are supported on a particular request endpoint, please refer to the [API documentation](../../operations/api-reference.md).
+
+GET requires READ permission, while POST and DELETE require WRITE permission.
+
+### SQL Permissions
+
+Queries on Druid datasources require DATASOURCE READ permissions for the specified datasource.
+
+Queries on the [INFORMATION_SCHEMA tables](../../querying/sql.html#information-schema) will
+return information about datasources that the caller has DATASOURCE READ access to. Other
+datasources will be omitted.
+
+Queries on the [system schema tables](../../querying/sql.html#system-schema) require the following permissions:
+- `segments`: Segments will be filtered based on DATASOURCE READ permissions.
+- `servers`: The user requires STATE READ permissions.
+- `server_segments`: The user requires STATE READ permissions and segments will be filtered based on DATASOURCE READ permissions.
+- `tasks`: Tasks will be filtered based on DATASOURCE READ permissions.
+
+
+### Debugging
+
+If you face difficulty grasping why access is denied to certain elements and the `audit` section in
+Apache Ranger does not give you any detail, you can enable debug logging for `org.apache.druid.security.ranger`.
+To do so add the following in your `log4j2.xml`:
+
+```xml
+<!-- Set level="debug" to see access requests to Apache Ranger -->
+<Logger name="org.apache.druid.security" level="debug" additivity="false">
+  <Appender-ref ref="Console"/>
+</Logger>
+```

docs/development/extensions.md

@@ -54,6 +54,7 @@ Core extensions are maintained by Druid committers.
 |druid-orc-extensions|Support for data in Apache Orc data format.|[link](../development/extensions-core/orc.md)|
 |druid-parquet-extensions|Support for data in Apache Parquet data format. Requires druid-avro-extensions to be loaded.|[link](../development/extensions-core/parquet.md)|
 |druid-protobuf-extensions| Support for data in Protobuf data format.|[link](../development/extensions-core/protobuf.md)|
+|druid-ranger-security|Support for access control through Apache Ranger.|[link](../development/extensions-core/druid-ranger-security.md)|
 |druid-s3-extensions|Interfacing with data in AWS S3, and using S3 as deep storage.|[link](../development/extensions-core/s3.md)|
 |druid-ec2-extensions|Interfacing with AWS EC2 for autoscaling middle managers|UNDOCUMENTED|
 |druid-stats|Statistics related module including variance and standard deviation.|[link](../development/extensions-core/stats.md)|