Plugable handler for CREATE FUNCTION

[milenkovicm]

Is your feature request related to a problem or challenge?

At the moment datafusion does not support CREATE FUNCTION, it only exposes programatic api for UDF declaration.
Programatic approach does not support declarative UDF, and would not work with SQL only interaction.

I believe that we could expose a pluggable interface like FunctionFactory (part of SessionState) where user can plug their own CREATE FUNCTION handler, and plug their own declarative UDFs, without bringing them to core library.

Describe the solution you'd like

A solution would allow users to declare functions like:

CREATE FUNCTION iris(FLOAT[])
RETURNS FLOAT[]
LANGUAGE TORCH
AS 'models:/iris@champion'

or

CREATE FUNCTION add(BIGINT, BIGINT)
RETURNS BIGINT
LANGUAGE SQL
RETURN $1 + $2

CREATE FUNCTION handler would be configured at the SessionContext configuration phase:

let runtime_config = RuntimeConfig::new();
let runtime_environment = RuntimeEnv::new(runtime_config).unwrap();
let session_config =
    SessionConfig::new().set_str("datafusion.sql_parser.dialect", "PostgreSQL");
let state = SessionState::new_with_config_rt(session_config, Arc::new(runtime_environment))
    // we configure a torch function factory
    // to handle CREATE (and DROP) FUNCTION requests
    .with_function_factory(Arc::new(TorchFunctionFactory {}));

let ctx = SessionContext::new_with_state(state);

in this example we define TorchFunctionFactory,

full example can be found at https://github.com/milenkovicm/torchfusion

Describe alternatives you've considered

No response

Additional context

No response

[alamb]

This would be quite cool (for example, it could potentially be used to support WASM udfs #9326).

[milenkovicm]

yes, that would be third good use case