Skip to content

Commit 8f29b4f

Browse files
yaooqinnulysses-you
yaooqinn
authored and
ulysses-you
committed
[KYUUBI #2395] [DOC] Add Documentation for Spark AuthZ Extension
### _Why are the changes needed?_ ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #2395 from yaooqinn/doc2. Closes #2395 109440b [Kent Yao] [DOC] Add Documentation for Spark AuthZ Extension 852e7fd [Kent Yao] [DOC] Add Documentation for Spark AuthZ Extension dfeef88 [Kent Yao] [DOC] Add Documentation for Spark AuthZ Extension Authored-by: Kent Yao <yao@apache.org> Signed-off-by: ulysses-you <ulyssesyou@apache.org>
1 parent a7674d2 commit 8f29b4f

File tree

8 files changed

+347
-53
lines changed

8 files changed

+347
-53
lines changed

‎docs/security/authorization.md

Lines changed: 0 additions & 49 deletions
This file was deleted.

‎docs/security/authorization/index.rst

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
.. Licensed to the Apache Software Foundation (ASF) under one or more
2+
contributor license agreements. See the NOTICE file distributed with
3+
this work for additional information regarding copyright ownership.
4+
The ASF licenses this file to You under the Apache License, Version 2.0
5+
(the "License"); you may not use this file except in compliance with
6+
the License. You may obtain a copy of the License at
7+
8+
.. http://www.apache.org/licenses/LICENSE-2.0
9+
10+
.. Unless required by applicable law or agreed to in writing, software
11+
distributed under the License is distributed on an "AS IS" BASIS,
12+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
See the License for the specific language governing permissions and
14+
limitations under the License.
15+
16+
.. image:: https://svn.apache.org/repos/asf/comdev/project-logos/originals/kyuubi-1.svg
17+
:align: center
18+
:width: 25%
19+
20+
Kyuubi Authorization Guide
21+
==========================
22+
23+
.. toctree::
24+
:maxdepth: 2
25+
26+
Spark AuthZ Plugin <spark/index>

‎docs/security/authorization/spark/build.md

Lines changed: 104 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,104 @@
1+
<!--
2+
- Licensed to the Apache Software Foundation (ASF) under one or more
3+
- contributor license agreements. See the NOTICE file distributed with
4+
- this work for additional information regarding copyright ownership.
5+
- The ASF licenses this file to You under the Apache License, Version 2.0
6+
- (the "License"); you may not use this file except in compliance with
7+
- the License. You may obtain a copy of the License at
8+
-
9+
- http://www.apache.org/licenses/LICENSE-2.0
10+
-
11+
- Unless required by applicable law or agreed to in writing, software
12+
- distributed under the License is distributed on an "AS IS" BASIS,
13+
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
- See the License for the specific language governing permissions and
15+
- limitations under the License.
16+
-->
17+
18+
19+
20+
# Building Kyuubi Spark AuthZ Plugin
21+
22+
<img src="https://svn.apache.org/repos/asf/comdev/project-logos/originals/kyuubi-1.svg" alt="Kyuubi logo" width="50%" align="right" />
23+
24+
## Build with Apache Maven
25+
26+
Kyuubi Spark AuthZ Plugin is built using [Apache Maven](http://maven.apache.org).
27+
To build it, `cd` to the root direct of kyuubi project and run:
28+
29+
```shell
30+
build/mvn clean package -pl :kyuubi-spark-authz_2.12 -DskipTests
31+
```
32+
33+
After a while, if everything goes well, you will get the plugin finally in two parts:
34+
35+
- The main plugin jar, which is under `./extensions/spark/kyuubi-spark-authz/target/kyuubi-spark-authz_${scala.binary.version}-${project.version}.jar`
36+
- The least transitive dependencies needed, which are under `./extensions/spark/kyuubi-spark-authz/target/scala-${scala.binary.version}/jars`
37+
38+
### Build against Different Apache Spark Versions
39+
40+
The maven option `spark.version` is used for specifying Spark version to compile with and generate corresponding transitive dependencies.
41+
By default, it is always built with the latest `spark.version` defined in kyuubi project main pom file.
42+
Sometimes, it may be incompatible with other Spark distributions, then you may need to build the plugin on your own targeting the Spark version you use.
43+
44+
For example,
45+
46+
```shell
47+
build/mvn clean package -pl :kyuubi-spark-authz_2.12 -DskipTests -Dspark.version=3.0.2
48+
```
49+
50+
The available `spark.version`s are shown in the following table.
51+
52+
| Spark Version | Supported | Remark |
53+
|:-----------------:|:-----------:|:--------------------------------------------------------------------------------------------------------------------------------:|
54+
| master | √ | - |
55+
| 3.3.x | √ | - |
56+
| 3.2.x | √ | - |
57+
| 3.1.x | √ | - |
58+
| 3.0.x | √ | - |
59+
| 2.4.x | √ | - |
60+
| 2.3.x and earlier | × | [PR 2367](https://github.com/apache/incubator-kyuubi/pull/2367) is used to track how we work with older releases with scala 2.11 |
61+
62+
Currently, Spark released with Scala 2.12 are supported.
63+
64+
### Build against Different Apache Ranger Versions
65+
66+
The maven option `ranger.version` is used for specifying Ranger version to compile with and generate corresponding transitive dependencies.
67+
By default, it is always built with the latest `ranger.version` defined in kyuubi project main pom file.
68+
Sometimes, it may be incompatible with other Ranger Admins, then you may need to build the plugin on your own targeting the Ranger Admin version you connect with.
69+
70+
```shell
71+
build/mvn clean package -pl :kyuubi-spark-authz_2.12 -DskipTests -Dranger.version=0.7.0
72+
```
73+
74+
The available `ranger.version`s are shown in the following table.
75+
76+
| Ranger Version | Supported | Remark |
77+
|:--------------:|:-----------:|:------:|
78+
| 2.2.x | √ | - |
79+
| 2.1.x | √ | - |
80+
| 2.0.x | √ | - |
81+
| 1.2.x | √ | - |
82+
| 1.1.x | √ | - |
83+
| 1.0.x | √ | - |
84+
| 0.7.x | √ | - |
85+
| 0.6.x | √ | - |
86+
87+
Currently, all ranger releases are supported.
88+
89+
## Test with ScalaTest Maven plugin
90+
If you omit `-DskipTests` option in the command above, you will also get all unit tests run.
91+
92+
```shell
93+
build/mvn clean package -pl :kyuubi-spark-authz_2.12
94+
```
95+
96+
If any bug occurs and you want to debug the plugin yourself, you can configure `-DdebugForkedProcess=true` and `-DdebuggerPort=5005`(optional).
97+
98+
```shell
99+
build/mvn clean package -pl :kyuubi-spark-authz_2.12 -DdebugForkedProcess=true
100+
```
101+
102+
The tests will suspend at startup and wait for a remote debugger to attach to the configured port.
103+
104+
We will appreciate if you can share the bug or the fix to the Kyuubi community.

‎docs/security/authorization/spark/index.rst

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
.. Licensed to the Apache Software Foundation (ASF) under one or more
2+
contributor license agreements. See the NOTICE file distributed with
3+
this work for additional information regarding copyright ownership.
4+
The ASF licenses this file to You under the Apache License, Version 2.0
5+
(the "License"); you may not use this file except in compliance with
6+
the License. You may obtain a copy of the License at
7+
8+
.. http://www.apache.org/licenses/LICENSE-2.0
9+
10+
.. Unless required by applicable law or agreed to in writing, software
11+
distributed under the License is distributed on an "AS IS" BASIS,
12+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
See the License for the specific language governing permissions and
14+
limitations under the License.
15+
16+
.. image:: https://svn.apache.org/repos/asf/comdev/project-logos/originals/kyuubi-1.svg
17+
:align: center
18+
:width: 25%
19+
20+
Kyuubi Spark AuthZ Plugin
21+
=========================
22+
23+
.. toctree::
24+
:maxdepth: 2
25+
26+
Overview <overview>
27+
Building <build>
28+
Installing <install>

‎docs/security/authorization/spark/install.md

Lines changed: 126 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,126 @@
1+
<!--
2+
- Licensed to the Apache Software Foundation (ASF) under one or more
3+
- contributor license agreements. See the NOTICE file distributed with
4+
- this work for additional information regarding copyright ownership.
5+
- The ASF licenses this file to You under the Apache License, Version 2.0
6+
- (the "License"); you may not use this file except in compliance with
7+
- the License. You may obtain a copy of the License at
8+
-
9+
- http://www.apache.org/licenses/LICENSE-2.0
10+
-
11+
- Unless required by applicable law or agreed to in writing, software
12+
- distributed under the License is distributed on an "AS IS" BASIS,
13+
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
- See the License for the specific language governing permissions and
15+
- limitations under the License.
16+
-->
17+
18+
<img src="https://svn.apache.org/repos/asf/comdev/project-logos/originals/kyuubi-1.svg" alt="Kyuubi logo" width="25%" align="center" />
19+
20+
21+
# Installing and Configuring Kyuubi Spark AuthZ Plugin
22+
23+
## Pre-install
24+
25+
- [Apache Ranger](https://ranger.apache.org/)
26+
27+
This plugin works as a ranger rest client with Apache Ranger admin server to do privilege check.
28+
Thus, a ranger server need to be installed ahead and available to use.
29+
30+
- Building(optional)
31+
32+
If your ranger admin or spark distribution is not compatible with the official pre-built [artifact](https://mvnrepository.com/artifact/org.apache.kyuubi/kyuubi-spark-authz) in maven central.
33+
You need to [build](build.md) the plugin targeting the spark/ranger you are using by yourself.
34+
35+
## Install
36+
37+
With the `kyuubi-spark-authz_*.jar` and its transitive dependencies available for spark runtime classpath, such as
38+
- Copied to `$SPARK_HOME/jars`, or
39+
- Specified to `spark.jars` configuration
40+
41+
## Configure
42+
43+
### Settings for Connecting Ranger Admin
44+
45+
#### ranger-spark-security.xml
46+
- Create `ranger-spark-security.xml` in `$SPARK_HOME/conf` and add the following configurations
47+
for pointing to the right Ranger admin server.
48+
49+
```xml
50+
<configuration>
51+
<property>
52+
<name>ranger.plugin.spark.policy.rest.url</name>
53+
<value>ranger admin address like http://ranger-admin.org:6080</value>
54+
</property>
55+
56+
<property>
57+
<name>ranger.plugin.spark.service.name</name>
58+
<value>a ranger hive service name</value>
59+
</property>
60+
61+
<property>
62+
<name>ranger.plugin.spark.policy.cache.dir</name>
63+
<value>./a ranger hive service name/policycache</value>
64+
</property>
65+
66+
<property>
67+
<name>ranger.plugin.spark.policy.pollIntervalMs</name>
68+
<value>5000</value>
69+
</property>
70+
71+
<property>
72+
<name>ranger.plugin.spark.policy.source.impl</name>
73+
<value>org.apache.ranger.admin.client.RangerAdminRESTClient</value>
74+
</property>
75+
76+
</configuration>
77+
```
78+
79+
#### ranger-spark-audit.xml
80+
81+
Create `ranger-spark-audit.xml` in `$SPARK_HOME/conf` and add the following configurations
82+
to enable/disable auditing.
83+
84+
```xml
85+
<configuration>
86+
87+
<property>
88+
<name>xasecure.audit.is.enabled</name>
89+
<value>true</value>
90+
</property>
91+
92+
<property>
93+
<name>xasecure.audit.destination.db</name>
94+
<value>false</value>
95+
</property>
96+
97+
<property>
98+
<name>xasecure.audit.destination.db.jdbc.driver</name>
99+
<value>com.mysql.jdbc.Driver</value>
100+
</property>
101+
102+
<property>
103+
<name>xasecure.audit.destination.db.jdbc.url</name>
104+
<value>jdbc:mysql://10.171.161.78/ranger</value>
105+
</property>
106+
107+
<property>
108+
<name>xasecure.audit.destination.db.password</name>
109+
<value>rangeradmin</value>
110+
</property>
111+
112+
<property>
113+
<name>xasecure.audit.destination.db.user</name>
114+
<value>rangeradmin</value>
115+
</property>
116+
117+
</configuration>
118+
```
119+
120+
### Settings for Spark Session Extensions
121+
122+
Add `org.apache.kyuubi.plugin.spark.authz.ranger.RangerSparkExtension` to the spark configuration `spark.sql.extensions`.
123+
124+
```properties
125+
spark.sql.extensions=org.apache.kyuubi.plugin.spark.authz.ranger.RangerSparkExtension
126+
```

‎docs/security/authorization/spark/overview.md

Lines changed: 57 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,57 @@
1+
<!--
2+
- Licensed to the Apache Software Foundation (ASF) under one or more
3+
- contributor license agreements. See the NOTICE file distributed with
4+
- this work for additional information regarding copyright ownership.
5+
- The ASF licenses this file to You under the Apache License, Version 2.0
6+
- (the "License"); you may not use this file except in compliance with
7+
- the License. You may obtain a copy of the License at
8+
-
9+
- http://www.apache.org/licenses/LICENSE-2.0
10+
-
11+
- Unless required by applicable law or agreed to in writing, software
12+
- distributed under the License is distributed on an "AS IS" BASIS,
13+
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
- See the License for the specific language governing permissions and
15+
- limitations under the License.
16+
-->
17+
18+
# Kyuubi AuthZ Plugin For Spark SQL
19+
20+
<img src="https://svn.apache.org/repos/asf/comdev/project-logos/originals/kyuubi-1.svg" alt="Kyuubi logo" width="25%" align="right" />
21+
22+
Security is one of the fundamental features for enterprise adoption with Kyuubi.
23+
When deploying Kyuubi against secured clusters,
24+
storage-based authorization is enabled by default, which only provides file-level coarse-grained authorization mode.
25+
When row/column-level fine-grained access control is required,
26+
we can enhance the data access model with the Kyuubi Spark AuthZ plugin.
27+
28+
## Authorization in Kyuubi
29+
30+
### Storage-based Authorization
31+
32+
As Kyuubi supports multi tenancy, a tenant can only visit authorized resources,
33+
including computing resources, data, etc.
34+
Most file systems, such as HDFS, support ACL management based on files and directories.
35+
36+
A so called Storage-based authorization mode is supported by Kyuubi by default.
37+
In this model, all objects, such as databases, tables, partitions, in meta layer are mapping to folders or files in the storage layer,
38+
as well as their permissions.
39+
40+
Storage-based authorization offers users with database, table and partition-level coarse-gained access control.
41+
42+
### SQL-standard authorization with Ranger
43+
44+
A SQL-standard authorization usually offers a row/colum-level fine-grained access control to meet the real-world data security need.
45+
46+
[Apache Ranger](https://ranger.apache.org/) is a framework to enable, monitor and manage comprehensive data security across the Hadoop platform.
47+
This plugin enables Kyuubi with data and metadata control access ability for Spark SQL Engines, including,
48+
49+
- Column-level fine-grained authorization
50+
- Row-level fine-grained authorization, a.k.a. Row-level filtering
51+
- Data masking
52+
53+
## The Plugin Itself
54+
55+
Kyuubi Spark Authz Plugin itself provides general purpose for ACL management for data & metadata while using Spark SQL.
56+
It is not necessary to deploy it with the Kyuubi server and engine, and can be used as an extension for any Spark SQL jobs.
57+
However, the authorization always requires a robust authentication layer and multi tenancy support, so Kyuubi is a perfect match.

0 commit comments

Comments
 (0)