feat(flink): implement UDF support for the backend

[deepyaman]

Description of changes

• Add UDF translation rules to the Flink backend (copied from the base SQL backend, because Flink doesn't extend it)
  • Maybe this could be cleaner/not require copy-pasting?
• Add a test for a built-in scalar UDF
• Add a test for a built-in aggregation UDF
• Implement scalar Python UDFs
• Implement scalar pandas UDFs

Issues closed

Resolves #8108

[cpcloud]

Any chance we can hold off on this for the 8.0 release and target the-epic-split? Flink isn't yet ported (working on it now) so there's nothing to code against for the moment.

[deepyaman]

Any chance we can hold off on this for the 8.0 release and target the-epic-split? Flink isn't yet ported (working on it now) so there's nothing to code against for the moment.

Sure! I don't think there's any rush on this.

Maybe I'll just keep this branch for now, implement more UDF stuff (since you mentioned TES doesn't affect UDF too much), and then will make whatever changes and/or rebase on top of the new branch, raising a new PR if need be?

[deepyaman]

UDF tests pass locally:

(ibis-dev) deepyaman@Deepyamans-Air ibis % poetry run pytest --junitxml=junit.xml --cov=ibis --cov-report=xml:coverage.xml -m flink --randomly-dont-reorganize ibis/backends/tests/test_udf.py 
============================================================================================= test session starts ==============================================================================================
platform darwin -- Python 3.10.13, pytest-7.4.3, pluggy-1.3.0
Using --randomly-seed=3454285181
benchmark: 4.0.0 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000)
rootdir: /Users/deepyaman/github/claypotai/ibis
configfile: pyproject.toml
plugins: repeat-0.9.2, hypothesis-6.88.1, snapshot-0.0.0, pytest_httpserver-1.0.8, randomly-3.15.0, cov-4.1.0, clarity-1.0.1, mock-3.12.0, xdist-3.3.1, benchmark-4.0.0
collected 100 items / 95 deselected / 5 selected                                                                                                                                                               

ibis/backends/tests/test_udf.py Exception ignored in: <_io.FileIO name=26 mode='wb' closefd=True>
Traceback (most recent call last):
  File "/opt/miniconda3/envs/ibis-dev/lib/python3.10/site-packages/_pytest/unraisableexception.py", line 59, in __exit__
    del self.unraisable
ResourceWarning: unclosed file <_io.BufferedWriter name=26>
Exception ignored in: <_io.FileIO name=27 mode='rb' closefd=True>
Traceback (most recent call last):
  File "/opt/miniconda3/envs/ibis-dev/lib/python3.10/site-packages/_pytest/unraisableexception.py", line 59, in __exit__
    del self.unraisable
ResourceWarning: unclosed file <_io.BufferedReader name=27>
.xx.x                                                                                                                                                                    [100%]

--------------------------------------------------------------------- generated xml file: /Users/deepyaman/github/claypotai/ibis/junit.xml ---------------------------------------------------------------------

--------- coverage: platform darwin, python 3.10.13-final-0 ----------
Coverage XML written to file coverage.xml

================================================================================= 2 passed, 95 deselected, 3 xfailed in 24.28s =================================================================================

[cpcloud]

🚢 it!

[cpcloud]

@deepyaman Can we wait for #8398 to be merged? You'll be able to delete some boilerplate in this PR once that one's in.

[gforsyth]

#8398 is in now

[deepyaman]

@deepyaman Can we wait for #8398 to be merged? You'll be able to delete some boilerplate in this PR once that one's in.

Sounds good! I'll update this afternoon.

[cpcloud]

You should be able to revert the changes in this file.

[deepyaman]

I think the other changes are still valid, but I've reverted the (overriding) fallback behaviors.

[cpcloud]

Thanks!