User functions with string parameters

[stefanritt]

We are very happy with tinyexp++ and would like to use it in a case where we have a user function which needs a std::string parameter to pull some value out of a database. Would you accept a pull request or what would be the proper way to handle this?

[stefanritt]

This is what I got from Codex 5.6 sol:

Note: "odb" is the database we are using.

Can we extend tinyexp++ to support stings as user function arguments?

Yes. In the current TinyExpr++ design, adding string-valued arguments to user functions is quite feasible, but it is more than just adding another function-pointer typedef. The parser currently assumes that every expression node ultimately evaluates to one numeric te_type (double by default).

For example, you probably want to be able to write something like:

double odb_get(std::string_view path);

tep.set_variables_and_functions({
    {"odb", odb_get}
});

double x = tep.evaluate(R"(odb("/Equipment/HV/Voltage"))");

or perhaps mixed arguments:

double odb_get(std::string_view path, double index);

odb("/Equipment/HV/Voltage", 3)

The important distinction is that you do not necessarily need TinyExpr++ to become a general string expression evaluator. If strings are only allowed as arguments to functions, while all functions still return numbers, the extension can remain relatively contained.

TinyExpr++ currently defines user functions explicitly as numerical signatures:

using te_fun0 = te_type (*)();
using te_fun1 = te_type (*)(te_type);
using te_fun2 = te_type (*)(te_type, te_type);
...
using te_fun24 = ...

and stores them in te_variant_type.

So I would not change te_type into something like

std::variant<double, std::string>

throughout the evaluator. That would infect almost every arithmetic operation with type checking and make a small math parser substantially more complicated.

Instead, I would introduce a separate AST node type for string literals.

Conceptually:

class te_expr {
public:
    enum class value_kind {
        numeric,
        string
    };

    value_kind m_kind = value_kind::numeric;

    te_variant_type m_value;
    std::string m_string;

    ...
};

although deriving a te_string_expr from te_expr would fit the current design particularly well, because te_expr is already deliberately polymorphic. The documentation explicitly mentions deriving from te_expr to carry arrays, strings, or other data for custom functions.

I would probably implement it as:

class te_string_expr : public te_expr
{
public:
    explicit te_string_expr(std::string value) :
        te_expr(TE_DEFAULT),
        m_value(std::move(value))
    {}

    std::string m_value;
};

Then the lexer/parser needs one new token: string literal. Today the grammar only has numeric constants, variables, functions, and parenthesized expressions; string literals aren't part of it.

So you'd add handling for:

"hello"
"/Equipment/HV/Voltage"
"some \"quoted\" string"

A useful grammar addition is roughly:

<string> = '"' { character | escape } '"'

<base> =
      <constant>
    | <string>
    | <variable>
    | <function...>
    | "(" <list> ")"

I'd initially support the common escapes:

\\
\"
\n
\r
\t

The next design choice is how to represent function signatures. There are two reasonable approaches.

1. Minimal extension — dedicated string function types

For your likely use case, this is the approach I'd choose.

Add function types such as:

using te_sfun1 = te_type (*)(std::string_view);
using te_sfun2 = te_type (*)(std::string_view, te_type);
using te_sfun3 = te_type (*)(std::string_view, te_type, te_type);

and perhaps the reverse combinations you actually need:

using te_fun1s = te_type (*)(te_type, std::string_view);

Add these to te_variant_type:

using te_variant_type =
    std::variant<
        te_type,
        const te_type*,

        te_fun0,
        te_fun1,
        te_fun2,
        ...

        te_sfun1,
        te_sfun2,
        ...
    >;

The AST function node already contains its child expressions:

std::vector<te_expr*> m_parameters;

At evaluation time, a normal te_fun2 does approximately:

f(eval(param[0]), eval(param[1]))

while a string-aware function would do:

f(
    string_value(param[0]),
    eval(param[1])
)

with something like:

std::string_view string_value(const te_expr* e)
{
    const auto* s = dynamic_cast<const te_string_expr*>(e);
    if (!s)
        throw type_error(...);

    return s->m_value;
}

That makes this possible:

double exists(std::string_view path)
{
    return ...;
}

tep.add_variable_or_function({
    "exists",
    static_cast<te_sfun1>(exists)
});

tep.evaluate(R"(exists("/Equipment/HV"))");

2. More general solution — typed function arguments

If you want eventually to support many signatures such as

f(double, string, double)
f(string, string)
f(double, string)
f(string, double, string)

then defining te_sfun1, te_sfun2, etc. becomes ugly very quickly.

There are 2^N combinations for N parameters.

At that point I'd instead introduce a tiny argument variant:

using te_arg = std::variant<te_type, std::string_view>;

and a new generic user-function API:

using te_user_fun =
    std::function<te_type(std::span<const te_arg>)>;

Then:

double odb(std::span<const te_arg> args)
{
    auto path =
        std::get<std::string_view>(args[0]);

    auto index =
        std::get<double>(args[1]);

    ...
}

The parser expression:

odb("/Equipment/HV/Voltage", 3)

would create children representing:

FUNCTION odb
   STRING "/Equipment/HV/Voltage"
   NUMBER 3

and evaluation would build:

std::array<te_arg, 2> args = {
    std::string_view{...},
    3.0
};

return function(args);

This has a big advantage: all existing numeric function support remains untouched.

TinyExpr++'s current te_fun0 … te_fun24 interface keeps working exactly as today, including the existing variadic support of 1–24 numeric arguments.

You simply add a second category:

te_funX       // legacy numeric
te_user_fun   // typed user function

I think that's the cleanest long-term architecture.

There is one thing I would specifically not allow initially:

"abc" + "def"
x = "hello"
strlen("foo") == strlen("bar")   // fine
"foo" == "bar"                   // not initially

Keep strings as opaque function arguments, not general expression values. That means arithmetic evaluation remains entirely numeric:

numeric expressions -> double
string literal      -> only legal in typed function argument

This makes error handling straightforward too:

sin("foo")

→ parse/type error:

Function SIN expects numeric argument 1

and

odb(123)

→

Function ODB expects string argument 1

I would therefore structure the patch roughly like this:

1. Add recognition of quoted string literals to the lexer.
2. Add te_string_expr : te_expr.
3. Allow string-expression nodes inside function argument lists.
4. Add a te_arg = variant<te_type,string_view>.
5. Add a new user-function type taking span<const te_arg>.
6. Store that new function type in te_variant_type.
7. At function-call evaluation, convert each child node into a te_arg.
8. Leave all existing te_fun0…te_fun24 evaluation completely unchanged.
9. Reject string nodes anywhere except arguments of the new typed functions.
10. Add tests for quoted strings, escapes, mixed parameters, wrong types, empty strings, and commas/parentheses inside strings.

The nice part is that the core recursive-descent arithmetic parser barely changes. The current TinyExpr++ grammar and AST are already well suited to adding another leaf-node type.

For your use case, I would expose an API closer to this:

double my_function(std::string_view name, double x);

rather than forcing callers to unpack std::variants themselves. Internally, however, I would use the generic te_arg representation and add template machinery that automatically wraps ordinary C++ functions. Then users could simply write:

double my_function(std::string_view name, double x)
{
    ...
}

tep.set_variables_and_functions({
    {"myfunc", my_function}
});

and TinyExpr++ would deduce the signature, just as it currently deduces numeric function arity. The library already moved toward compiler-deduced function types rather than explicit TE_FUNCTIONn flags, so that would be consistent with its present API direction.

I estimate this as a modest rather than major modification: probably a few hundred lines including proper signature deduction and tests, with only perhaps 50–100 lines touching the actual parser/evaluator. The main complexity is not parsing "foo"; it's designing the function-binding mechanism so mixed double/string_view signatures stay elegant.

If your concrete target is something like myfunc("name") or odb("/path", index), I can go one step further and map out the exact changes to the current tinyexpr.h and tinyexpr.cpp, function by function, including the C++ types I'd add.

[Blake-Madden]

I'm certainly always happy to check out PRs. If you don't mind, I'll look over this tomorrow and provide some feedback; this definitely looks like an interesting use case.

[Blake-Madden]

The general using te_arg = std::variant<te_type, std::string_view>; approach seems like the best. If you want you could put a PR together for it (along with Catch2 unit tests), or I could work on it next week. I'm hoping it would take me more than a few days if you want to wait, just let me know.