Incorrect output tuple number of R-UDF & Performance Bottleneck

[bobbai00]

For the following workflow:

According to the user, the expected # of output tuples of the R UDF should be equal to the # of input tuples.

Also, It takes 5 mins in Python script, but 20 mins using the workflow. According to the user's observation, the R UDF and Sort operator is the bottleneck.

[chenlica]

@jaeyun0503 Please take a look.

[bobbai00]

User has provided the workflow json and the data in the original issue: #3494

[aglinxinyuan]

It seems the number reported by the backend is incorrect. In this case, @jaeyun0503 and I don’t have the necessary context, and we're also not familiar with the R-UDF performance issue. I believe we may not be the right people to handle this. Perhaps @kunwp1 and @Yicong-Huang could take a look?

[mengw15]

I had a discussion with @Yicong-Huang regarding this issue and would like to share our findings and next steps.

Output Tuple Count

First, we were not able to reproduce the issue with the incorrect number of output tuples from the R UDF. In our tests, the number of output tuples matched the number of input tuples as expected.

Performance Bottleneck – Two Considerations

As for the performance bottleneck, we guess there are two possible factors:

1. Transfer Overhead (Python → R)

Yicong suspects that the performance drop could be due to how data is transferred from Python to R. Currently, we may be sending the data as DataFrame format (will check if this is true), which can introduce significant overhead. Instead, we should consider transferring the data as Arrow record batches. Moreover, when the target UDF is R, we should avoid converting the Arrow batch into a DataFrame on the Python side, and instead directly forward the Arrow data to R.

2. Processing Time in R

If the bottleneck lies in R’s processing speed, one potential improvement is to adopt an approach similar to the Python UDF, where we pass source code and let the R process spin up a thread to run that code. This could potentially improve performance.

Next Steps

I plan to conduct a set of tests to better understand the performance gap
including comparing the execution time of R code in Texera versus native R
and measuring the data transfer speed differences between Python UDFs and R UDFs within Texera.

Initially, I’ll focus on investigating data transfer speed to validate the first hypothesis.