Error executing SQL function with SparkLinker on AWS Glue

[roshnaramesh]

Hi,
Apologies in advance for the long post. I'm running a linking job using SparkLinker on AWS Glue and this is my code

conf = SparkConf()

path = similarity_jar_location()
conf.set("spark.jars", path)

sc = SparkContext.getOrCreate(conf=conf)

spark = SparkSession(sc)

s3_checkpoint_path = 's3://s3_bucket/temp/checkpoint/'

spark.sparkContext.setCheckpointDir(s3_checkpoint_path)

df_1 = spark.read.parquet("s3://s3_bucket/temp/df_1.parquet", header=True)
df_2 = spark.read.parquet("s3://s3_bucket/temp/df_2.parquet", header=True)
settings_dict = {
    "link_type": "link_only",
    "blocking_rules_to_generate_predictions": [
        block_on("last_name"),
    ],
    "comparisons": [
        ctl.name_comparison("cust_1_full_name",term_frequency_adjustments=True),
        ctl.name_comparison("cust_2_full_name",term_frequency_adjustments=True),
        ctl.name_comparison("cust_3_full_name",term_frequency_adjustments=True),
        ctl.name_comparison("cust_4_full_name",term_frequency_adjustments=True),
        ctl.name_comparison("full_name",term_frequency_adjustments=True),
    ],
"retain_matching_columns": True,
"retain_intermediate_calculation_columns": True,
"em_convergence": 0.01
}

linker = SparkLinker(
[df_1, df_2],
settings_dict,
input_table_aliases=["df_1", "df_2"]
)
blocking_rule = block_on("last_name")
count = linker.count_num_comparisons_from_blocking_rule(blocking_rule_2)

print(f"Number of comparisons generated by : {count:,.0f}")
if(count>0):
    linker.estimate_u_using_random_sampling(max_pairs=5e5)

When it executes
linker.estimate_u_using_random_sampling(max_pairs=5e5), it gives the following error:

pyspark.sql.utils.AnalysisException: Undefined function: DAMERAU_LEVENSHTEIN. This function is neither a built-in/temporary function, nor a persistent function that is qualified as spark_catalog.default.damerau_levenshtein.; line 64 pos 11

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/tmp/test_splink_job.py", line 103, in <module>
    linker.estimate_u_using_random_sampling(max_pairs=5e5)
  File "/home/spark/.local/lib/python3.10/site-packages/splink/linker.py", line 1493, in estimate_u_using_random_sampling
    estimate_u_values(self, max_pairs, seed)
  File "/home/spark/.local/lib/python3.10/site-packages/splink/estimate_u.py", line 149, in estimate_u_values
    df_params = training_linker._execute_sql_pipeline(sample_dataframe)
  File "/home/spark/.local/lib/python3.10/site-packages/splink/linker.py", line 645, in _execute_sql_pipeline
    raise e
  File "/home/spark/.local/lib/python3.10/site-packages/splink/linker.py", line 639, in _execute_sql_pipeline
    dataframe = self._sql_to_splink_dataframe_checking_cache(
  File "/home/spark/.local/lib/python3.10/site-packages/splink/linker.py", line 905, in _sql_to_splink_dataframe_checking_cache
    splink_dataframe = self._execute_sql_against_backend(
  File "/home/spark/.local/lib/python3.10/site-packages/splink/spark/linker.py", line 400, in _execute_sql_against_backend
    spark_df = self._log_and_run_sql_execution(sql, templated_name, physical_name)
  File "/home/spark/.local/lib/python3.10/site-packages/splink/linker.py", line 719, in _log_and_run_sql_execution
    raise SplinkException(
splink.exceptions.SplinkException: Error executing the following sql for table `__splink__m_u_counts`(__splink__m_u_counts_740580e73):`

Can anyone let me know what I'm doing wrong? The same code runs in my local machine without any errors.
I'm running this job with Spark 3.3 and Glue 4.0. Splink's version is 3.9.2

[RobinL]

I this this discussion is probably the best documentation we have of this:

#1377

In short

path = similarity_jar_location()
conf.set("spark.jars", path)
```
is incompatible with aws glue, and you have to use the `'--extra-jars'` parameter of the glue job to provide jars