[KYUUBI #3904] [Spark] New Authz Plan Serde Layer

### _Why are the changes needed?_ This PR redesigned the authorization part of the spark authz module with a New Authz Plan Serde Layer. #### Motivation - add a general layer to describe a command, so that we can add a new command or users can add a third-party command easily according to the specification. - get rid of the spark version check. The built-in spark commands frequently vary from version to version, which makes us hard to maintain at compile& runtime phase, and the third-party commands are hard to check by spark versions. #### Data structure ![image](https://user-images.githubusercontent.com/8326978/205860086-c445988f-4867-4a2a-b39e-aa2461559d67.png) Overall, we introduce 2 general basic data structures: - CommandSpec: used to describe a command - `classname` as key for the read-side to get the spec by a particular command - pre-defined operation type - descriptors - Descriptor: used to describe an object, such as table, db, query, - `fieldName`: the object to get - `fieldExtractor`: the method to get the object; use SPI to load - sub-descriptors: such as columns in a table - etc. #### SPI - Extractor: implementations for `fieldExtractor` - key: the name of the extractor for the read-side to get itself - func: converting the field value to specific and general objects #### Code Path - Write code path - automatically generated default json configuration files - custom json configuration files for thrid-party commands - Read code path - Load json as maps - RuleAuthorization -> PrivilegeBuilder.build -> get command desc from maps -> build privileges with the retrieved desc. #### TODOs - Add back the ArcticCommand - Add delta command - Add ways for loading custom json configuration files - Add hudi commands - etc ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #3904 from yaooqinn/na. Closes #3904 efafcba [Kent Yao] sort 7d2b3e4 [Kent Yao] ci 379e933 [Kent Yao] style b52ab41 [Kent Yao] comments 9a24be6 [Kent Yao] [WIP][Extension][Spark] New Authz Plan Serde Layer 8926f04 [Kent Yao] [WIP][Extension][Spark] New Authz Plan Serde Layer b45453a [Kent Yao] [WIP][Extension][Spark] New Authz Plan Serde Layer f56148e [Kent Yao] [WIP][Extension][Spark] New Authz Plan Serde Layer e47749d [Kent Yao] [WIP][Extension][Spark] New Authz Plan Serde Layer c32feef [Kent Yao] [WIP][Extension][Spark] New Authz Plan Serde Layer 49dbb68 [Kent Yao] [WIP][Extension][Spark] New Authz Plan Serde Layer 2bced47 [Kent Yao] Merge branch 'master' into na 453541b [Kent Yao] [WIP][Extension][Spark] New Authz Plan Serde Layer Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Kent Yao <yao@apache.org>
apache · Dec 7, 2022 · 2540f44 · 2540f44
1 parent 749b81d
commit 2540f44
Show file tree

Hide file tree

Showing 41 changed files with 3,958 additions and 1,404 deletions.
diff --git a/extensions/spark/kyuubi-spark-authz/pom.xml b/extensions/spark/kyuubi-spark-authz/pom.xml
@@ -251,6 +251,11 @@
             <scope>provided</scope>
         </dependency>
 
+        <dependency>
+            <groupId>com.fasterxml.jackson.module</groupId>
+            <artifactId>jackson-module-scala_${scala.binary.version}</artifactId>
+        </dependency>
+
         <dependency>
             <groupId>org.apache.kyuubi</groupId>
             <artifactId>kyuubi-common_${scala.binary.version}</artifactId>

diff --git a/...esources/META-INF/services/org.apache.kyuubi.plugin.spark.authz.serde.ActionTypeExtractor b/...esources/META-INF/services/org.apache.kyuubi.plugin.spark.authz.serde.ActionTypeExtractor
@@ -0,0 +1,18 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+org.apache.kyuubi.plugin.spark.authz.serde.OverwriteOrInsertActionTypeExtractor
diff --git a/...in/resources/META-INF/services/org.apache.kyuubi.plugin.spark.authz.serde.ColumnExtractor b/...in/resources/META-INF/services/org.apache.kyuubi.plugin.spark.authz.serde.ColumnExtractor
@@ -0,0 +1,26 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+org.apache.kyuubi.plugin.spark.authz.serde.PartitionColumnExtractor
+org.apache.kyuubi.plugin.spark.authz.serde.PartitionLocsSeqColumnExtractor
+org.apache.kyuubi.plugin.spark.authz.serde.PartitionOptionColumnExtractor
+org.apache.kyuubi.plugin.spark.authz.serde.PartitionSeqColumnExtractor
+org.apache.kyuubi.plugin.spark.authz.serde.StringColumnExtractor
+org.apache.kyuubi.plugin.spark.authz.serde.StringSeqColumnExtractor
+org.apache.kyuubi.plugin.spark.authz.serde.StringSeqLastColumnExtractor
+org.apache.kyuubi.plugin.spark.authz.serde.StringSeqOptionColumnExtractor
+org.apache.kyuubi.plugin.spark.authz.serde.StructFieldSeqColumnExtractor
diff --git a/.../resources/META-INF/services/org.apache.kyuubi.plugin.spark.authz.serde.DatabaseExtractor b/.../resources/META-INF/services/org.apache.kyuubi.plugin.spark.authz.serde.DatabaseExtractor
@@ -0,0 +1,23 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+org.apache.kyuubi.plugin.spark.authz.serde.ResolvedDBObjectNameDatabaseExtractor
+org.apache.kyuubi.plugin.spark.authz.serde.ResolvedNamespaceDatabaseExtractor
+org.apache.kyuubi.plugin.spark.authz.serde.StringDatabaseExtractor
+org.apache.kyuubi.plugin.spark.authz.serde.StringOptionDatabaseExtractor
+org.apache.kyuubi.plugin.spark.authz.serde.StringSeqDatabaseExtractor
+org.apache.kyuubi.plugin.spark.authz.serde.StringSeqOptionDatabaseExtractor
diff --git a/.../resources/META-INF/services/org.apache.kyuubi.plugin.spark.authz.serde.FunctionExtractor b/.../resources/META-INF/services/org.apache.kyuubi.plugin.spark.authz.serde.FunctionExtractor
@@ -0,0 +1,20 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+org.apache.kyuubi.plugin.spark.authz.serde.ExpressionInfoFunctionExtractor
+org.apache.kyuubi.plugin.spark.authz.serde.FunctionIdentifierFunctionExtractor
+org.apache.kyuubi.plugin.spark.authz.serde.StringFunctionExtractor
diff --git a/...ources/META-INF/services/org.apache.kyuubi.plugin.spark.authz.serde.FunctionTypeExtractor b/...ources/META-INF/services/org.apache.kyuubi.plugin.spark.authz.serde.FunctionTypeExtractor
@@ -0,0 +1,20 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+org.apache.kyuubi.plugin.spark.authz.serde.ExpressionInfoFunctionTypeExtractor
+org.apache.kyuubi.plugin.spark.authz.serde.FunctionIdentifierFunctionTypeExtractor
+org.apache.kyuubi.plugin.spark.authz.serde.TempMarkerFunctionTypeExtractor
diff --git a/...ain/resources/META-INF/services/org.apache.kyuubi.plugin.spark.authz.serde.QueryExtractor b/...ain/resources/META-INF/services/org.apache.kyuubi.plugin.spark.authz.serde.QueryExtractor
@@ -0,0 +1,19 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+org.apache.kyuubi.plugin.spark.authz.serde.LogicalPlanOptionQueryExtractor
+org.apache.kyuubi.plugin.spark.authz.serde.LogicalPlanQueryExtractor
diff --git a/...ain/resources/META-INF/services/org.apache.kyuubi.plugin.spark.authz.serde.TableExtractor b/...ain/resources/META-INF/services/org.apache.kyuubi.plugin.spark.authz.serde.TableExtractor
@@ -0,0 +1,23 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+org.apache.kyuubi.plugin.spark.authz.serde.CatalogTableTableExtractor
+org.apache.kyuubi.plugin.spark.authz.serde.DataSourceV2RelationTableExtractor
+org.apache.kyuubi.plugin.spark.authz.serde.IdentifierTableExtractor
+org.apache.kyuubi.plugin.spark.authz.serde.LogicalRelationTableExtractor
+org.apache.kyuubi.plugin.spark.authz.serde.ResolvedTableTableExtractor
+org.apache.kyuubi.plugin.spark.authz.serde.TableIdentifierTableExtractor
diff --git a/...resources/META-INF/services/org.apache.kyuubi.plugin.spark.authz.serde.TableTypeExtractor b/...resources/META-INF/services/org.apache.kyuubi.plugin.spark.authz.serde.TableTypeExtractor
@@ -0,0 +1,19 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+org.apache.kyuubi.plugin.spark.authz.serde.TableIdentifierTableTypeExtractor
+org.apache.kyuubi.plugin.spark.authz.serde.ViewTypeTableTypeExtractor
diff --git a/extensions/spark/kyuubi-spark-authz/src/main/resources/database_command_spec.json b/extensions/spark/kyuubi-spark-authz/src/main/resources/database_command_spec.json
@@ -0,0 +1,129 @@
+[ {
+  "classname" : "org.apache.spark.sql.catalyst.plans.logical.CommentOnNamespace",
+  "databaseDescs" : [ {
+    "fieldName" : "child",
+    "fieldExtractor" : "ResolvedNamespaceDatabaseExtractor",
+    "isInput" : false
+  } ],
+  "opType" : "ALTERDATABASE"
+}, {
+  "classname" : "org.apache.spark.sql.catalyst.plans.logical.CreateNamespace",
+  "databaseDescs" : [ {
+    "fieldName" : "name",
+    "fieldExtractor" : "ResolvedDBObjectNameDatabaseExtractor",
+    "isInput" : false
+  }, {
+    "fieldName" : "namespace",
+    "fieldExtractor" : "StringSeqDatabaseExtractor",
+    "isInput" : false
+  } ],
+  "opType" : "CREATEDATABASE"
+}, {
+  "classname" : "org.apache.spark.sql.catalyst.plans.logical.DescribeNamespace",
+  "databaseDescs" : [ {
+    "fieldName" : "namespace",
+    "fieldExtractor" : "ResolvedNamespaceDatabaseExtractor",
+    "isInput" : true
+  } ],
+  "opType" : "DESCDATABASE"
+}, {
+  "classname" : "org.apache.spark.sql.catalyst.plans.logical.DropNamespace",
+  "databaseDescs" : [ {
+    "fieldName" : "namespace",
+    "fieldExtractor" : "ResolvedNamespaceDatabaseExtractor",
+    "isInput" : false
+  } ],
+  "opType" : "DROPDATABASE"
+}, {
+  "classname" : "org.apache.spark.sql.catalyst.plans.logical.SetCatalogAndNamespace",
+  "databaseDescs" : [ {
+    "fieldName" : "child",
+    "fieldExtractor" : "ResolvedDBObjectNameDatabaseExtractor",
+    "isInput" : true
+  }, {
+    "fieldName" : "namespace",
+    "fieldExtractor" : "StringSeqOptionDatabaseExtractor",
+    "isInput" : true
+  } ],
+  "opType" : "SWITCHDATABASE"
+}, {
+  "classname" : "org.apache.spark.sql.catalyst.plans.logical.SetNamespaceLocation",
+  "databaseDescs" : [ {
+    "fieldName" : "namespace",
+    "fieldExtractor" : "ResolvedNamespaceDatabaseExtractor",
+    "isInput" : false
+  } ],
+  "opType" : "ALTERDATABASE_LOCATION"
+}, {
+  "classname" : "org.apache.spark.sql.catalyst.plans.logical.SetNamespaceProperties",
+  "databaseDescs" : [ {
+    "fieldName" : "namespace",
+    "fieldExtractor" : "ResolvedNamespaceDatabaseExtractor",
+    "isInput" : false
+  } ],
+  "opType" : "ALTERDATABASE"
+}, {
+  "classname" : "org.apache.spark.sql.execution.command.AlterDatabasePropertiesCommand",
+  "databaseDescs" : [ {
+    "fieldName" : "databaseName",
+    "fieldExtractor" : "StringDatabaseExtractor",
+    "isInput" : false
+  } ],
+  "opType" : "ALTERDATABASE"
+}, {
+  "classname" : "org.apache.spark.sql.execution.command.AlterDatabaseSetLocationCommand",
+  "databaseDescs" : [ {
+    "fieldName" : "databaseName",
+    "fieldExtractor" : "StringDatabaseExtractor",
+    "isInput" : false
+  } ],
+  "opType" : "ALTERDATABASE_LOCATION"
+}, {
+  "classname" : "org.apache.spark.sql.execution.command.AnalyzeTablesCommand",
+  "databaseDescs" : [ {
+    "fieldName" : "databaseName",
+    "fieldExtractor" : "StringOptionDatabaseExtractor",
+    "isInput" : true
+  } ],
+  "opType" : "ANALYZE_TABLE"
+}, {
+  "classname" : "org.apache.spark.sql.execution.command.CreateDatabaseCommand",
+  "databaseDescs" : [ {
+    "fieldName" : "databaseName",
+    "fieldExtractor" : "StringDatabaseExtractor",
+    "isInput" : false
+  } ],
+  "opType" : "CREATEDATABASE"
+}, {
+  "classname" : "org.apache.spark.sql.execution.command.DescribeDatabaseCommand",
+  "databaseDescs" : [ {
+    "fieldName" : "databaseName",
+    "fieldExtractor" : "StringDatabaseExtractor",
+    "isInput" : true
+  } ],
+  "opType" : "DESCDATABASE"
+}, {
+  "classname" : "org.apache.spark.sql.execution.command.DropDatabaseCommand",
+  "databaseDescs" : [ {
+    "fieldName" : "databaseName",
+    "fieldExtractor" : "StringDatabaseExtractor",
+    "isInput" : false
+  } ],
+  "opType" : "DROPDATABASE"
+}, {
+  "classname" : "org.apache.spark.sql.execution.command.SetDatabaseCommand",
+  "databaseDescs" : [ {
+    "fieldName" : "databaseName",
+    "fieldExtractor" : "StringDatabaseExtractor",
+    "isInput" : true
+  } ],
+  "opType" : "SWITCHDATABASE"
+}, {
+  "classname" : "org.apache.spark.sql.execution.command.SetNamespaceCommand",
+  "databaseDescs" : [ {
+    "fieldName" : "namespace",
+    "fieldExtractor" : "StringSeqDatabaseExtractor",
+    "isInput" : true
+  } ],
+  "opType" : "SWITCHDATABASE"
+} ]
diff --git a/extensions/spark/kyuubi-spark-authz/src/main/resources/function_command_spec.json b/extensions/spark/kyuubi-spark-authz/src/main/resources/function_command_spec.json
@@ -0,0 +1,75 @@
+[ {
+  "classname" : "org.apache.spark.sql.execution.command.CreateFunctionCommand",
+  "functionDescs" : [ {
+    "fieldName" : "functionName",
+    "fieldExtractor" : "StringFunctionExtractor",
+    "databaseDesc" : {
+      "fieldName" : "databaseName",
+      "fieldExtractor" : "StringOptionDatabaseExtractor",
+      "isInput" : false
+    },
+    "functionTypeDesc" : {
+      "fieldName" : "isTemp",
+      "fieldExtractor" : "TempMarkerFunctionTypeExtractor",
+      "skipTypes" : [ "TEMP" ]
+    },
+    "isInput" : false
+  } ],
+  "opType" : "CREATEFUNCTION"
+}, {
+  "classname" : "org.apache.spark.sql.execution.command.DescribeFunctionCommand",
+  "functionDescs" : [ {
+    "fieldName" : "info",
+    "fieldExtractor" : "ExpressionInfoFunctionExtractor",
+    "databaseDesc" : null,
+    "functionTypeDesc" : {
+      "fieldName" : "info",
+      "fieldExtractor" : "ExpressionInfoFunctionTypeExtractor",
+      "skipTypes" : [ "TEMP", "SYSTEM" ]
+    },
+    "isInput" : true
+  }, {
+    "fieldName" : "functionName",
+    "fieldExtractor" : "FunctionIdentifierFunctionExtractor",
+    "databaseDesc" : null,
+    "functionTypeDesc" : {
+      "fieldName" : "functionName",
+      "fieldExtractor" : "FunctionIdentifierFunctionTypeExtractor",
+      "skipTypes" : [ "TEMP", "SYSTEM" ]
+    },
+    "isInput" : true
+  } ],
+  "opType" : "DESCFUNCTION"
+}, {
+  "classname" : "org.apache.spark.sql.execution.command.DropFunctionCommand",
+  "functionDescs" : [ {
+    "fieldName" : "functionName",
+    "fieldExtractor" : "StringFunctionExtractor",
+    "databaseDesc" : {
+      "fieldName" : "databaseName",
+      "fieldExtractor" : "StringOptionDatabaseExtractor",
+      "isInput" : false
+    },
+    "functionTypeDesc" : {
+      "fieldName" : "isTemp",
+      "fieldExtractor" : "TempMarkerFunctionTypeExtractor",
+      "skipTypes" : [ "TEMP" ]
+    },
+    "isInput" : false
+  } ],
+  "opType" : "DROPFUNCTION"
+}, {
+  "classname" : "org.apache.spark.sql.execution.command.RefreshFunctionCommand",
+  "functionDescs" : [ {
+    "fieldName" : "functionName",
+    "fieldExtractor" : "StringFunctionExtractor",
+    "databaseDesc" : {
+      "fieldName" : "databaseName",
+      "fieldExtractor" : "StringOptionDatabaseExtractor",
+      "isInput" : false
+    },
+    "functionTypeDesc" : null,
+    "isInput" : false
+  } ],
+  "opType" : "RELOADFUNCTION"
+} ]