apply UDF with more than one argument in spark_apply() function

[sparkcassuser]

i have R function with more than one argument and uses dplyr functions in it.

now, i want to apply this UDF to spark data frame.
The sample code

myfun=function(objdf,x,y,k){
  
  f <- function(x1,y1,x2,y2) { 
    d=(x2-x1) + (y2-y1)
  }   
  search=function(df,x,y,k){
    df1=data.frame(cbind(df,f(x,y,df$xx,df$yy)))
    colnames(df1)=c(colnames(df),"val")
    colnames(df1)
    new_df=df1 %>% arrange(val) %>% head(k)
    return(new_df)
  }
  
  searchwithk <- function(x,y,k) {force(x,y,k);function(df) search(df,x,y,k)}
  
  res <- spark_apply(objdf, function(df) {
    searchwithk(x,y,k)
  })
  #return(res)       
}

#df= spark_dataframe
x=12.12
y=-74.5
k=5
result=myfun(df,x,y,k)
result

it gives me long error / unused parameter in force statement

how to resolve?

[harryprince]

pass all your customized functions in spark_apply function block.

like this:

spark_apply(df,function(df){

search=function(df,x,y,k){
df1=data.frame(cbind(df,f(x,y,df$xx,df$yy)))
colnames(df1)=c(colnames(df),"val")
colnames(df1)
new_df=df1 %>% arrange(val) %>% head(k)
return(new_df)
}

searchwithk <- function(x,y,k) {force(x,y,k);function(df) search(df,x,y,k)}

return(searchwithk(x,y,k))

})

[javierluraschi]

You can also make use of the context parameter in spark_apply() if you need to run with different values of x,y,k. Closing since the question seems to be answered.