Support custom functions

[wschella]

See https://www.w3.org/TR/sparql11-query/#extensionFunctions. Could be very useful to pass custom functions that SPARQL doesn't support.

[rubensworks]

@wschella Could you give some (quick) pointers on how we could implement support for custom functions? Someone may work on this in the near future.

[wschella]

Yes.

1. The functions should be passed as an argument to the evaluator, e.g. AsyncEvaluator.ts, down to call to transformAlgebra(algExpr, newCustomFuncs).
2. In transformAlgebra, you should make sure transformOperator handles these unknown operator names, based on the custom functions (which should ofcourse have a name). It can potentially also be used to overwrite behaviour of existing funcs.
3. transformOperator should then return a new expression type: Expression.CustomOperator, similar to Expression.Operator, or Expression.SpecialOperator.
4. The cases should be handled where in evaluation the expression tree, a custom operator is found. On top of my head this should only be relevant in the evaluators, with a similar function as evalSpecialOperator. This new function should be added to the subevaluators like the other methods. Implementing this should be not much more difficult then expr.apply(args). Given that Expression.CustomOperator does the heavy lifting.

Note: It is likely it will be slightly more complicated regarding the type system and sync/async differentation. So you might need Expression.CustomAsyncOperator and Expression.CustomSyncOperator or something, where in a sync evaluator the async operators can not be handled, or can be handled blocking. A good interface regarding the configuration is important.

[rubensworks]

Thanks for the input!

[jitsedesmet]

NOTE to Self: A custom function in Algebra format looks like this:

"input": {
    "type": "filter",
    "input": {
      "type": "bgp",
      "patterns": [...]
    },
    "expression": {
      "type": "expression",
      "expressionType": "named",
      "name": {
        "termType": "NamedNode",
        "value": "http://example.org/functions#even"
      },
      "args": [
        {
          "type": "expression",
          "expressionType": "term",
          "term": {
            "termType": "Variable",
            "value": "id"
          }
        }
      ]
    }
  }

I suspect we should use transformNamed instead of transformOperator.

[wschella]

You're right! I now see I even left a todo there.

sparqlee/lib/Transformation.ts

Line 166 in 18c44c3

// TODO: Support passing functions to override default behaviour;

!

[jitsedesmet]

@wschella
You wrote (I'll edit it a bit ;) ):

transformNamed should then return a new expression type: Expression.CustomOperator, similar to Expression.Operator, or Expression.SpecialOperator.

I was wondering why we can't just use Expression.Named as a starting point? Maybe give another SimpleApplication (given by the user) as an argument to it's constructor. Is there a specific reason you where thinking of creating a new type?
Is what you meant with the "TODO" you mentioned earlier?

[wschella]

The terminology I used is not great, so it could make sense. Also I was a bit confused, my apologies.

The TODO I mentioned there I put there likely because I think the spec allows overriding the functionality of the Named operators (which are functions like toString and such). I forgot that. But this is different case still from adding NEW operators, which this issue was originally meant for: https://www.w3.org/TR/sparql11-query/#extensionFunctions.

We should distinguish between these two cases (and check the spec to see if I'm right).

• I think it would be cleaner to give extensions these a new type, i.e. my arguing CustomOperator. But likely ExtensionOperator is even better to more closely match the spec.
• Using Expression.Named would make sense for the overriding of the default behavior of existing functions (even if only to distinguish from extensions).

Now, since for overriding you likely want to tag the Expression.Named with a source = "default" | "custom", it could just be as clean, and maybe even better, to just make that tag source = "default" | "custom" | "extension".

The question with these matters is always how much their execution differs from each other. On first glance, maybe not at all. But I can also imagine errors or logging should be handled differently. Anyway, I don't have strong opinions (nor the authority to have them), so do as you see fit 😄

[rubensworks]

Thanks for your input @wschella!

Just to make sure we're aligned; the tag source = "default" | "custom" | "extension" would then have the following semantics?

• default: The named expression corresponds to a named expression defined by the SPARQL spec.
• custom: The named expression overrides a named expression, to behave differently from defined by the SPARQL spec. (override may also be a valid type name)
• extension: The named expression introduces a new extension function.

In any case, I guess the injection point (for Comunica) for custom and extension may be the same, and this distinction would only be internally for Sparqlee.

[wschella]

@rubensworks Yeah that sums it up. And override is better indeed.

I do think there is merit in keeping the injection point separate (but equal in config/type). You likely want different errors if something is wrong (e.g. override types need to match the ones from default, while extension has free choice), but more importantly you don't want to allow unintentionally overriding default behavior by choosing an extensions which happens to have the same name or something. I don't know if this is valid. But in config passed to comunica/sparqlee I think explicit difference between { overrideFuncs: ..., extensionFuncs: ...} and { customFuncs: ...} is meaningfull, and adds little (or even reduces) cognitive overhead.

[rubensworks]

I do think there is merit in keeping the injection point separate (but equal in config/type)
You likely want different errors if something is wrong

Good point!
Let's try it this way @jitsedesmet.

[jitsedesmet]

Thank you for your clarification! I think I understand what you're saying. If I am uncertain about anything, I hope I an contact you again 😄

[wschella]

Definitely :)