Possible Function enhancements/corrections

[bijington]

To discuss how the existing in-built functions could be improved/or simply corrected 😄.

[graniero]

No prob! I'll be back at my work machine in the next day or so (been taking a few days' break for a change!), and I'll put a PR together.

[bijington]

Awesome thanks! Enjoy the rest of your break 😄

[bijington]

@graniero thank you for the help on this. I can confirm it is in the new release and can even be tested out here:

https://bijington.github.io/expressive-playground/

Note I do need to expand the IgnoreCase options on the playground.

[graniero]

Very happy to help, @bijington ! You have a great library here - being able to lend a hand with some light supporting tasks while you're doing the heavy lifting feels like a good way to say thank you!

Quite willing to lend a hand with other stuff that you're considering too, whenever you're ready to tackle it. I have notifications set, so just @ mention me when you're ready for a second set of eyeballs or pair of typing hands!

[bijington]

@graniero great thank you for the offer and also the very kind words about the library 😄

[graniero]

Behaviour of Random() with integer parameters

This comment just captures a behavioural quirk to which some people might respond "well yeah, of course that's what it does" while other people might respond "wow I didn't expect that", depending on what your go-to random number tool is.

When Random(min_value, max_value) is supplied floating-point parameters, it behaves the way you would expect based on other random-number generators: it returns a random number greater than or equal to min_value and less than max_value. The beauty of this Random() function is that you're not limited to [0, 1): you can give it any values and save yourself the bother of scaling and shifting [0, 1) to the range you want.

When Random(min_value, max_value) is supplied integer parameters, it behaves the same way: returns a random integer in the range [min_value, max_value). Nice and consistent with floating-point parameters. "well yeah, of course that's what it does".

However, if you typically do most of your random number work in Excel (yes, I know all the reasons why you wouldn't want to, but life has a way of happening) and you're used to the RANDBETWEEN(min_value, max_value) function, the behaviour is different than expected: unlike RANDBETWEEN(), you will never get your max_value. Easy to spot with Random(1, 3) but you may not notice it with Random(1, 1000). "wow I didn't expect that"

[bijington]

I am not sure I expected it 😄.

I fully and sadly agree that Excel has created a bit of a standard way of expecting things to work, especially as most of us will probably deal with users expecting to craft expressions in the same way that Excel does. I will need to do some more investigations on what Excel offers and what exactly Expressive does.

I do believe there should be consistency at least so perhaps this quirk will ultimately be considered a bug.

[graniero]

Yeah, it gets tricky because you've got programmers using Expressive to easily handle built-in or generated expressions within their code, which they expect to behave like the tools they're working with (e.g. their programming language); but you've also got programmers using Expressive to produce applications that easily allow users to write/design expressions within the application, which they expect to behave like the tools they're working with (e.g. Excel). In the latter case, the users writing the expressions will not be looking at Expressive's documentation - they won't even know what Expressive is!

The latter case is what I'm personally dealing with: a large operations monitoring system which allows the engineers using the system to set up a library of standard formulas/calculations to be used within or across site projects, and to set up triggering conditions for rules and alarms. For this application, and any other applications like this, users are generally going to expect the functions to work the way they're used to - and welcome to Excel as the de facto standard!

[graniero]

Behaviour of Count() with string vs. number parameters

Again, this comment just captures a behavioural quirk which people should know about. However, unlike with Random() above, this one may warrant some discussion about what would be the most desirable behaviour.

When all of the parameters are numbers, Count() returns the number of parameters.

When all of the parameters are strings, Count() returns the total number of characters across all the strings, i.e. Length(parameter1) + Length(parameter2) +. ....

When the parameters are mixed, Count() returns the number of numeric parameters plus the total number of characters across all the string parameters:

• Count(123) = 1
• Count('123') = 3
• Count(123, 4) = 2
• Count('123', 4) = 4
• Count(123, '4') = 2
• Count('foo', 123) = 4
• Count('foo', '123') = 6

In all cases, the Count() function does not do the same thing as:

• .NET's various Count properties and Count() methods return the number of elements in the collection, or equivalently, the number of parameters in the parameter list.
• Excel's COUNT() function returns a count of the numbers in the parameter list.

I'm not sure if the current behaviour is all that useful. I can't think of a decent use case - maybe other folks can?

The current behaviour with all string parameters is kind of like Length() if you were able to supply an aggregate as a parameter value like you can now do with operators. (by the way, upvote +100 for aggregates in functions, please!!!) Once that's possible, then this behaviour for string parameters would be redundant.

The current behaviour with all numeric parameters is just a count of the number of parameters, basically the same as Count in .NET, for example. The Count() function could be modified so this is the behaviour for mixed parameter types too - at least it would be consistent across types. But right now you already know how many parameters you are supplying (unless the expressions are generated, I suppose). This behaviour won't be all that powerful until you can supply an aggregate as a parameter value. (another upvote +100 for that, please!!!)

[bijington]

My initial thoughts are that this is actually a bug!

A use case that did develop a while back was to allow for aggregate values to be counted. e.g. Count([Prices]) where the Prices variable is in fact an array of numbers. This should give the count of numbers within the variable. I suspect somehow I have gone too far up in the inheritance tree and I am counting IEnumerable which string inherits from.

We should in theory be able to handle aggregates in some functions. Admittedly I may have broken that functionality and there could be other candidates that should support aggregates.

[bijington]

@graniero I have included this in a bug #99 to cover the issue because it is an issue. Good catch!

For what it is worth and again I need to apologies for a lack of documentation on this part but you can use aggregates inside some expressions like Count, in fact I suspect that is part of the cause of this issue.

You can call this:

var aggregates = new [] { 1, 2, 3, 4, 5 };
var values = new Dictionary<string, object> { ["foo"] = aggregates };
var i = new Expressive.Expression("Count([foo])");

Console.WriteLine(i.Evaluate<int>(values));

Also for I have found out to be quite exciting I am able to run really quick tests like this in VS Code using dotnet interactive

[graniero]

@bijington I would agree with comments in #99 that IEnumberable is the likely culprit. Makes good sense to me.

[graniero]

I agree, I think under the current version's structure, the "caret as exponent or bitwise XOR" is most easily handled by an option - especially because it seems to be the only big operator overload issue that's come to light so far. It's also a pretty consistent approach with the rest of the library's semantics and behaviour.

The idea of having function 'bundles' that can be added in one or many at a time is really cool. That adds a ton of extensibility and power in the same way that R and others do. It's a great way to deal with operator overrides, or to deal with the situation where the standard, most commonly used, or most obvious name for a function in one domain is the same name that is used in another domain to do something completely different.

However, it feels like setting up different Contexts to each contain different function bundles is stretching what Context's responsibilities should be. When the Context originally started as basically the full set of functions and operators, and then someone might add or swap out one or two functions or operators, it was OK that the Context was not only the place to manage behavioural options but also the container for the functions, etc. But once you start getting into more involved function schemes etc., then Context as primary function container starts to get in the way of 'mix-and-match' or 'plug-and-play' setups.

I think another container class is needed in between the Context and the functions. The new class would contain a collection of functions, operators, and potentially other things (like constants?), and you would register that class or an instance of that class with your Context instance. Basically, the same as registering & unregistering functions or operators, but done as a cohesive group. You still end up with Context as the go-to spot for available functionality, but you basically start with a set of core built-in functions (managed by one of those container classes) loaded by default when the Context is instantiated, then build up your Context by loading other containers into its collection.

Either that, or if Context remains the container for the function and operator functionality, then you need to add a mechanism to have multiple Contexts active at the same time, either by chaining them together, or by adding another higher-up class to contain multiple Contexts. And in either approach, there is also a new need to check for function name / operator overloads/conflicts between collections, and provide a way to either select/resolve the conflict, or extend the grammar to allow a context as part of the function name....

....starting to sound like a major version development item!

[bijington]

I agree that Context is probably not the correct place to be doing this if in fact it does go ahead. As you said:

that the Context was not only the place to manage behavioural options but also the container for the functions

The idea to use them in the manner that I suggested would muddy the water. It could become very muddy if multiple Contexts are supplied and dealing with things like conflicting behaviours.

You are spot on with it sounding like some major development work! I think the option for now makes perfect sense and then if things do get extended in the manner we have discussed then the exponent behavioural differences could potentially slot in to that.

[graniero]

Yeah, I think when the time comes, ^ as exponent and ^ as bitwise XOR would very easily become part of the bundles, and any ExpressionOptions.CaretAsExponent value can be deprecated then obsoleted.