[Submarine Spark Security] when run sql have expression will return ”UnresolvedAttribute.exprId“ error

[Junyewu]

env:
spark-3.1.1
ranger-2.1
submarine-security-0.6.0

sql code:

spark-sql> create or replace temporary view user_city_temp as 
select id,case when if(city_zip_active >0,city_zip_active,if(city_zip_check >0 ,city_zip_check,city_zip_phone)) in ('0100','0200','0300','0400') then if(city_zip_active >0,city_zip_active,if(city_zip_check >0 ,city_zip_check,city_zip_phone)) else 'other' end as city from user_city;

colum_name           type
city_zip_phone	int	NULL
city_zip_check	int	NULL
city_zip_active	int	NULL

it will return a error

org.apache.spark.sql.catalyst.analysis.UnresolvedException: Invalid call to exprId on unresolved object, tree: 'city_zip_active
	at org.apache.spark.sql.catalyst.analysis.UnresolvedAttribute.exprId(unresolved.scala:157)
	at org.apache.spark.sql.catalyst.optimizer.SubmarineDataMaskingExtension$$anonfun$collectAllAliases$1$$anonfun$applyOrElse$2.$anonfun$applyOrElse$3(SubmarineDataMaskingExtension.scala:191)
	at scala.collection.mutable.HashMap.getOrElseUpdate(HashMap.scala:86)
	at org.apache.spark.sql.catalyst.optimizer.SubmarineDataMaskingExtension$$anonfun$collectAllAliases$1$$anonfun$applyOrElse$2.applyOrElse(SubmarineDataMaskingExtension.scala:191)
	at org.apache.spark.sql.catalyst.optimizer.SubmarineDataMaskingExtension$$anonfun$collectAllAliases$1$$anonfun$applyOrElse$2.applyOrElse(SubmarineDataMaskingExtension.scala:189)
	at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformUp$2(TreeNode.scala:352)

it is ok run in spark-3.1.1 with not spark-security

[Junyewu]

I try to make CreateViewCommand is not doMasking，and it is run ok

org.apache.spark.sql.catalyst.optimizer.scala

  override def apply(plan: LogicalPlan): LogicalPlan = plan match {
    case c: Command => c match {
      case c: CreateDataSourceTableAsSelectCommand => c.copy(query = doMasking(c.query))
      case c: CreateHiveTableAsSelectCommand => c.copy(query = doMasking(c.query))
//      case c: CreateViewCommand => c.copy(child = doMasking(c.child))

[harsha-tenneti]

Do you have any masking policies on any of the columns in the table? If yes, what are the columns and their masking strategy?

[Junyewu]

yes，table have masking
mask column：city_zip_active
masking strategy：redact

[xunliu]

The [Submarine Spark Security] functionality has been moved to the apache/incubator-kyuubi standalone project.