AnalysisException: resolved attribute(s) missing for blockEntityLinkage

[yeikel]

Describe the bug

I'd like to link two dataframes blocking on a field named cd using blockEntityLinkage with the following Schemas :

df1
 |-- name: string (nullable = true)
 |-- address: string (nullable = true)
 |-- cd: string (nullable = true)

df2
 |-- name: string (nullable = true)
 |-- address: string (nullable = true)
 |--cd: string (nullable = true)

 val linkedResults = LuceneRDD.blockEntityLinkage(df1 , df2 ,
    linker,
    Array("cd"),
    Array("cd"),
    500
  )

But it produces the following exception :

Exception in thread "main" org.apache.spark.sql.AnalysisException: resolved attribute(s) cd#351 missing from name#444,address#454,cd#101 in operator !Project [name#444, address#454, cd#101, concat(cd#351) AS __PARTITION_COLUMN__#481];;
!Project [name#444, address#454, cd#101, concat(cd#351) AS __PARTITION_COLUMN__#481]
+- Project [name#444, address#454, cd#101]

A self join in df1 works just fine

I tried renaming the column but it did not work.

I am running Spark 2.1.0 and lucenerdd 0.3.3

Edit : An explanation about the issue can be found here but I don't believe it can be fixed on my end.

Thank you

[yeikel]

For now I can use a regular link with no blocking , but ideally I'd like to know if it possible to use the blocking method.

[zouzias]

Can you try with version 0.3.5 and report back?

[yeikel]

Seems to be fixed in 0.3.5

[yeikel]

Actually it is not..

I am still seeing this in 0.3.5

org.apache.spark.sql.AnalysisException: resolved attribute(s) cd#594 missing from name#738,address#748,mkt_cd#101,id#2 in operator !Project [name#738, address#748, mkt_cd#101, id#2, concat(mkt_cd#594) AS __PARTITION_COLUMN__#778];;

[zouzias]

Can you share the full exception here?

[zouzias]

I pushed a hotfix here: https://github.com/zouzias/spark-lucenerdd/pull/151/files (feedback is welcome) and I plan to release it tonight under 0.3.6-SNAPSHOT.

[yeikel]

Will this be available with Spark 2.1?

I will be testing as soon as it is in maven central

[zouzias]

I reproduced here. I will try to fix now.

[zouzias]

Released a fix under 0.3.6-SNAPSHOT. Can you look again if the exception appears?

Tests are clean on the CI: https://travis-ci.org/zouzias/spark-lucenerdd/jobs/499546155

[yeikel]

I believe it is fixed now. Thank you

[zouzias]

Glad to hear.

[yeikel]

This is very strange.

When I run it in a cluster (reading from a hive table) , I don't see this error anymore. On the other hand, when I run it in my local (reading parquet files) , I see it. I am not sure how to replicate it and the contents of the files are sensitive , so I can't share them here.

I need to test more , but let's leave it closed for now.