Workplan for OmnisciDB UDFs

[pearu]

Background

OmnisciDB UDF support allows users to define and register custom functions (UDFs) to OmnisciDB server so that these can be used withing SQL queries to the server. The UDFs implementation is given as an LLVM IR module string that is linked to the server SQL engine in runtime. OmnisciDB server defines Thrift API for managing the UDFs.

The RBC project provides a Python package rbc that allows users to define and register the UDFs using a Python function definition. The rbc package uses numba for LLVM IR module string generation and registers the UDFs to OmnisciDB server via its Thrift API.

Features

• Retrieve device (CPU, GPU) information from the server required for LLVM IR generation.
• Supported client platforms: Linux, Windows, MacOSX.
• Packaging: conda-forge (https://github.com/conda-forge/rbc-feedstock/), pypi (https://pypi.org/project/rbc-project/)
• Row-wise UDFs - functions taking scalar inputs and returning scalar values are applied to a database table row by row.
• User-defined table functions (UDTF) - functions applied to database table columns and returning non-scalar outputs
  • prototype, uses pointers to columns and data sizes
  • design and implement a user-friendly UI, one array-like argument per table column
• Array UDFs - functions taking array inputs and returning scalar values are applied to database table row by row.
• Variable-length Array UDFs - functions taking array inputs and returning array values, WIP.
• Geo UDFs - functions taking Geo type inputs
• Generalized calls within UDFs to external C/C++ libraries (Stan, XGBoost, CUML, etc).
  • Basic idea: calling a function from an external library from a UDF definition boils down to allowing undefined symbols in the LLVM IR module that are resolved on the server-side. The server must be linked against the external library and provide this information to RBC via device information retrieval Thrift API.

Work-in-progress

• Using elementary math functions within UDFs: PR Call math functions from UDFs heavyai/rbc#24
• Variable length array support:
  • PR Variable lenght arrays support [WIP] heavyai/rbc#19
  • PR https://github.com/omnisci/omniscidb-internal/pull/4133

Work planned

• Maintain and add notebooks demonstrating UDF usages
  • Run notebooks under RBC test-suite to ensure their validity (Evaluate notebooks as a part of testing suite heavyai/rbc#13)
• Release rbc 0.2.1 for omniscidb versions < 5.2 (Release 0.2.1 heavyai/rbc#20). Prepare rbc 0.3 release for omniscdb version 5.2.

Possible breakers/blockers

• Numba is currently undergoing a refactoring process (PR Refactor Numba, part I numba/numba#5197, etc) that will require the corresponding adjustments in RBC when numba 0.49 is released.
• Using Numba to generate LLVM/NVVM IR for different targets numba/numba#4546
• OmnisciDB Thrift API changes (e.g. adding new types) requires syncing rbc accordingly. Solution: include type information to device information retrieval end-points.

[pearu]

RBC defines milestones:

• https://github.com/xnd-project/rbc/milestone/1 - rbc 0.2.1 with omniscidb 5.0 and 5.1 support
• https://github.com/xnd-project/rbc/milestone/2 - rbc 0.3 with omniscidb 5.2 support

[pearu]

• PR Numba RC 0.49 heavyai/rbc#54 - RBC is numba 0.48 and 0.49 compatible

[pearu]

Resolving numba/numba#4546 is on hold as we haven't seen problems with the issue yet.

[pearu]

• PR Retrieve thrift typemap from omniscidb server (requires 5.2 or newer) heavyai/rbc#50 - thrift type map is retrieved from the server

[pearu]

Will close as obsolete:

• Geo UDF - not planned
• Generalized calls within UDFs to external C/C++ libraries - ongoing research
• Run notebooks under RBC test-suite - low priority