Extend Filter function with cross-record boolean logic

[cirrusone]

Is your feature request related to a problem? Please describe.
Say I have this filter logic,

Filter = "(Rank=1,Val1<5),(Rank=2,Val1>5)"

Assuming no records have the same Rank, this fails as there cannot be any records that are ranked both 1 and 2. However, sometimes we need cross-record logic such as return 1st ranked item but only if 2nd ranked item meets certain criteria.

In the following code, Example1 and Example2 filters both match a record, but Example3 fails as both cannot be satisfied at same time

using Gridify;

namespace ConsoleApp2
{
    public static class Program
    {
        public static void Main(string[] args)
        {
            Level1[] array = new Level1[3];
            array[0] = new Level1() { Rank = 1, Val1 = 4.3 };
            array[1] = new Level1() { Rank = 2, Val1 = 5.2 };
            array[2] = new Level1() { Rank = 3, Val1 = 7.5 };

            var gridifyMapper = new GridifyMapper<Level1>().GenerateMappings()
                .AddMap("Rank", q => q.Rank)
                .AddMap("Val1", q => q.Val1);

            // Example1: Rank = 1 AND Val1 < 5  - Returns 1 item
            var query1 = new GridifyQuery() { Filter = "(Rank=1,Val1<5)" };
            var expression1 = query1.GetFilteringExpression(gridifyMapper).Compile();
            var filter1 = array.Where(expression1);
            int count1 = filter1.Count();

            // Example2: Rank = 2 AND Val1 > 5  - Returns 1 item
            var query2 = new GridifyQuery() { Filter = "(Rank=2,Val1>5)" };
            var expression2 = query2.GetFilteringExpression(gridifyMapper).Compile();
            var filter2 = array.Where(expression2);
            int count2 = filter2.Count();

            // Example3: (Rank = 1 AND Val1 < 5) AND (Rank = 2 AND Val1 > 5) - Returns 0 records even though both conditions are true
            var query3 = new GridifyQuery() { Filter = "(Rank=1,Val1<5),(Rank=2,Val1>5)" };
            var expression3 = query3.GetFilteringExpression(gridifyMapper).Compile();
            var filter3 = array.Where(expression3);
            int count3 = filter3.Count();
        }
    }

    public class Level1
    {
        public int Rank { get; set; }
        public double Val1 { get; set; }
    }
}

Describe the solution you'd like
Is it possible to extend functionality to include BooleanLogicExpression?

Eg something like this where the query returns if all of the filter expressions are true/false?

var query3 = new GridifyQuery() { Filter = "((Rank=1,Val1<5),(Rank=2,Val1>5))" };
var expression3 = query3.GetBooleanLogicExpression(gridifyMapper).Compile();
bool result = array.Where(expression3);

It similar to chaining as here but trying to apply additional logic outside of a record.

[alirezanet]

I think using the | (or) operator can solve this problem. "(Rank=1,Val1<5)|(Rank=2,Val1>5)"

if not, please provide your expected LINQ query for Example3.

[cirrusone]

It's not strictly OR since both values must be true. The nearest LINQ I can think of at the moment is

bool linqExpression = array.Count(x => (x.Rank == 1 && x.Val1 < 5) || (x.Rank == 2 && x.Val1 > 5)) == 2;

but really I am trying to evaluate if both
(x.Rank == 1 && x.Val1 < 5) == true AND (x.Rank == 2 && x.Val1 > 5) == true

Not sure if it's possible.

That's why I don't think it's a FilterExpression but a new feature such as LogicExpression where certain rules can be appended and determined if true/false.

[alirezanet]

according to your example if we use OR the problem can be solved easily:
e.g

	var query3 = new GridifyQuery() { Filter = "(Rank=1,Val1<5)  |  (Rank=2,Val1>5)" };
	var expression3 = query3.GetFilteringExpression(gridifyMapper).Compile();

	var HasAny = array.Any(expression3);   // get bool result from query
	var Count = array.Count(expression3);  // count the results

the thing is expressions are not only a filtering operation, you can use an expression to do whatever you want.

but really I am trying to evaluate if both
(x.Rank == 1 && x.Val1 < 5) == true AND (x.Rank == 2 && x.Val1 > 5) == true

also, as you said before (there cannot be any records that are ranked both 1 and 2) so we can not use AND between our two conditions here. I am confused what are you asking

[cirrusone]

It doesn't work for filtering as all conditions cannot be met. I was thinking more a boolean condition evaluator, it's close in functionality but outcome is different.

Some of this can be evaluated in code but trying to do it just via the Filter is more desirable. Ability to apply logic to a data set and determine if all of the rules are true/false.

Example1:

Level1[] array = new Level1[3];
            array[0] = new Level1() { Rank = 1, Val1 = 4.3 };  // true
            array[1] = new Level1() { Rank = 2, Val1 = 5.2 };  // true
            array[2] = new Level1() { Rank = 3, Val1 = 7.5 };

Filter = "(Rank=1,Val1<5),(Rank=2,Val1>5)"
Logic = Condition1 AND Condition2 [where condition1 is chained logic (Rank=1,Val1<5)]
Result = true

Example2:

Level1[] array = new Level1[3];
            array[0] = new Level1() { Rank = 1, Val1 = 4.3 };  // true
            array[1] = new Level1() { Rank = 2, Val1 = 4.9 };  // false
            array[2] = new Level1() { Rank = 3, Val1 = 7.5 };

Filter = "(Rank=1,Val1<5),(Rank=2,Val1>5)"
Logic = Condition1 AND Condition2
Result = false

Example3:

Level1[] array = new Level1[3];
            array[0] = new Level1() { Rank = 1, Val1 = 4.3 };   // true
            array[1] = new Level1() { Rank = 2, Val1 = 4.9 };   // false
            array[2] = new Level1() { Rank = 3, Val1 = 7.5 };

Filter = "(Rank=1,Val1<5) | (Rank=2,Val1>5)"
Logic = Condition1 OR Condition2
Result = true

[alirezanet]

So if I understood correctly you don't want to get any results back from your array. you just want to evaluate the whole collection to make sure certain conditions passed or not. the problem is unfortunately, this feature doesn't have any general use-case in most applications that's why I don't think adding this to gridify be a good idea.
another problem is this isn't really a single operation: (this is linq version of what you described)

var x = array.Any(a => a.Rank == 1 && a.Val1 < 5) && 
        array.Any(a => a.Rank == 2 && a.Val1 > 5);

gridify is fully compatible with ORMs and LINQ itself, having this feature breaks this compatibility.

I don't close this issue, maybe later someone or I found a solution to have this in a single Linq query.
thank you for your contribution.

[cirrusone]

You understand correctly. This is a feature suggestion to make a rule builder rather than a filter (ie the filter/rule string may not return any records but still be valid). I believe there is quite an extensive use-case for this as I'm often asked for complex queries which cannot be done in a simple filter, the performance of Gridify makes could make this more achievable.

[alirezanet]

Hi,
I combined a few ideas and now in gridify v2.4.0 we have a newer way to manually create our queries using QueryBuilder class. the good news is QueryBuilder has a few APIs that supports condition evaluation.

var builder = new QueryBuilder<Level1>()
                 .AddCondition("Rank=1,Val1<5")
                 .AddCondition("Rank=2,Val1>5");

var evaluationResult = builder.Evaluate(array);

---

Using a single string

Now because we found a way to evaluate multiple conditions if you want to somehow use a single string you can write your own logic or separator.
e.g using +:

void AddConditionsToBuilder<T>(string conditions, string seprator, IQueryBuilder < T > builder) =>
       conditions.Split(seprator).Select(c => builder.AddCondition(c));

var conditions = "(Rank=1,Val1<5)+(Rank=2,Val1>5)";
var builder = new QueryBuilder<Level1>();
AddConditionsToBuilder(conditions,"+",builder);
var result = builder.Evaluate(array);

---

Using builder to get Filtering expression

I suggest instead of using the mapper and gridifyQuery from now on use the new QueryBuilder

var compiledExpression = builder.BuildFilteringExpression().Compile();

[cirrusone]

This is brilliant! Thank you so much, this library is excellent.

Only have one question... :)

Is it possible to compile the QueryBuilder so you get the same performance as the FilteringExpression?

[alirezanet]

Thank you,

Is it possible to compile the QueryBuilder so you get the same performance as the FilteringExpression?

You still can compile any expression that returns from QueryBuilder but you can not compile the builder.

The output of these methods can be compiled since both returns an Expression.

• BuildFilteringExpression()

Also, feel free to give your feedback.
I've created a discussion about these new features:

#40
#1

[alirezanet]

Forget my last comment,
In v2.4.3 we have three special build methods for the compiling builder.

• BuildCompiledEvaluator
• BuildCompiled
• BuildWithPagingCompiled

---

[cirrusone]

I hope you don't mind my testing and feedback, you are doing great work here.

On the BuildCompiledEvaluator, I'm seeing quite a large difference in performance between doing the actual filter/eval after the expression has been compiled.

Example1:

var filterQuery = new GridifyQuery() { Filter = "SomeFilter" };
var compiledFilterExpression = filterQuery.GetFilteringExpression(filterMapper).Compile();

// --> Measure performance from here

var filter1 = arrayToFilter.Where(compiledFilterExpression);
var match = filter1.Any();

// --> Measure performance to here

Example 2:

var builder = new QueryBuilder<TestClass>()
            .AddCondition("SomeFilter");

var compiledEvaluator = builder.BuildCompiledEvaluator();


// --> Measure performance from here

var evaluate = compiledEvaluator(arrayToFilter);

// --> Measure performance to here

Example1: circa 50µs
Example2: circa 8000µs

I can't provide any data right now but may get some time next week.

[alirezanet]

I hope you don't mind my testing and feedback, you are doing great work here.

No, I love feedbacks :). just try to open another issue if it is a different topic or use discussions.

---

in the first example, you are filtering the data in the second one you evaluating conditions, these two are completely different operations. you can compare BuildCompiled with GetFilteringExpression these two are very much the same although BuildCompiled has extra ordering and paging checks. (eg. Benchmark)

---

update:
also, you were right I found a problem in CompiledEvaluator that will be fixed in the next version