v1.16.0

What's Changed

v1.16.0 is a bug fix release. It has no new settings and no API removals. Some fixes change behavior that code can depend on, for example an exception type or the text of an expression. Each fix with such a change has a Behavior change line. These are not considered breaking due to the nature of them being foundational bugs vs changes in actual intended behavior.

Every example below comes from the test suite or from a run on this release.

Exceptions

All errors that come from a bad filter or sort are now a QueryKitException. An API that maps QueryKitException to 400 no longer returns 500 for these inputs.

A value that does not convert throws ParsingException (#197)

A value that does not convert to the property type threw FormatException, OverflowException, ArgumentException, or InvalidOperationException. Now it throws ParsingException, and the original exception is the inner exception. The message names the value, the target type, and the query name.

Rating == "abc"
  before: FormatException
  after:  ParsingException: The value 'abc' is not a valid Int32 for the filter property 'Rating'.

BirthMonth ^^ ["Bogus"]
  before: ArgumentException: Requested value 'Bogus' was not found.
  after:  ParsingException: The value 'Bogus' is not a valid BirthMonthEnum for the filter property 'BirthMonth'.

BirthMonth ^^ ["January"]
  before and after: filters

Behavior change: catch ParsingException or QueryKitException instead of the System exceptions.

An unknown property in arithmetic is unknown (#151)

An unknown property in arithmetic threw ArgumentException, also when AllowUnknownProperties was true. Now it gets the same result as in every other clause.

(Nope + 1) > 3
  before: ArgumentException: Property 'Nope' not found on type 'TestingPerson'
  after:  UnknownFilterPropertyException: The filter property 'Nope' was not recognized.

(Nope + 1) > 3 || Age > 100   (AllowUnknownProperties, IgnoredClauseBehavior.Remove)
  before: ArgumentException
  after:  x => (x.Age > 100)

Behavior change: catch UnknownFilterPropertyException or QueryKitException instead of ArgumentException. With AllowUnknownProperties, the clause no longer throws.

A bad sort direction throws QueryKitParsingException (#182)

sort: Age sideways
  before: ArgumentException: Invalid direction: sideways. Allowed values are 'asc' and 'desc'.
  after:  QueryKitParsingException with the same message

Behavior change: catch QueryKitParsingException or QueryKitException instead of ArgumentException.

An unknown logical operator throws QueryKitParsingException (#177)

Only code that calls LogicalOperator.GetByOperatorString directly sees this change. The filter parser never sends an unknown operator.

LogicalOperator.GetByOperatorString("xor")
  before: System.Exception: Operator xor is not supported
  after:  QueryKitParsingException: Operator xor is not supported

The has operator names itself in its error (#175)

Title ^$ "winner"   (Title is not a collection)
  before: QueryKitParsingException: DoesNotHaveType is only supported for collections
  after:  QueryKitParsingException: HasType is only supported for collections

Security: PreventFilter, PreventSort, and MaxPropertyDepth

PreventFilter and PreventSort apply to every property path (#192)

PreventFilter applied only to a property on the left side of a comparison, with its name in the exact case. Arithmetic, the right side of a comparison, a property list, derived properties, and custom operations skipped the check. A caller could learn the value of a hidden field one comparison at a time, for example with (Salary + 0) > 50000. PreventSort also needed the exact case.

Now each property reference goes through the check. A prevented clause follows IgnoredClauseBehavior, and a prevented sort is skipped.

(Age + 0) > 30 && Title == "a"   (Age has PreventFilter)
  before: ((x.Age + 0) > 30) AndAlso (x.Title == "a")
  after:  x => ((True == True) AndAlso (x.Title == "a"))

FirstName == LastName   (LastName has PreventFilter)
  before: x => (x.FirstName == x.LastName)
  after:  x => (True == True)

(lastname, FirstName) @=* "s"   (LastName has PreventFilter)
  before: LastName contains "s" OrElse FirstName contains "s"
  after:  x => ((x.FirstName != null) AndAlso x.FirstName.ToLower().Contains("s".ToLower()))

sort: age desc   (Age has HasQueryName("years").PreventSort())
  before: sorts by Age
  after:  the sort is skipped

Behavior change: a filter or sort that reaches a prevented property through one of these paths no longer filters or sorts by that property.

A fully prevented property is ignored by its query name (#194)

A filter on the query name of a property with both PreventFilter and PreventSort threw InvalidOperationException. The member name of the same property already gave an ignored clause. Now the query name gives the same result.

Title has HasQueryName("name").PreventFilter().PreventSort()

name == "x" && Age > 100
  before: InvalidOperationException: 'Title' is not allowed for filtering or sorting.
  after:  x => ((True == True) AndAlso (x.Age > 100))

name == "x" || Age > 100   (IgnoredClauseBehavior.Remove)
  before: InvalidOperationException
  after:  x => (x.Age > 100)

MaxPropertyDepth applies to property paths in arithmetic (#178)

A property path in arithmetic skipped the depth check, so a caller could reach deeper navigation properties than the limit permits.

On Ingredient, MaxPropertyDepth = 0

(Recipe.Rating + 0) > 1
  before: filters, and the query joins Recipe
  after:  QueryKitPropertyDepthExceededException: The property path 'Recipe.Rating' has a depth of 1, which exceeds the maximum allowed depth of 0.

Recipe.Rating == 1
  before and after: QueryKitPropertyDepthExceededException

HasMaxDepth applies only to its own property (#180)

HasMaxDepth on Address also applied to every property whose name starts with Address, for example AddressBackup. Such a path skipped the global MaxPropertyDepth.

MaxPropertyDepth = 0, Address has HasMaxDepth(1)

AddressBackup.State == "x"
  before: filters (uses the limit of Address)
  after:  QueryKitPropertyDepthExceededException

sort: AddressBackup.State
  before: sorts
  after:  QueryKitPropertyDepthExceededException

Address.State == "x"
  before and after: filters

Query names and property paths

Query names resolve in the grammar (#193)

QueryKit no longer replaces query names in the filter text before the parse. The grammar reads a query name as a property. As a result, a query name works in a property list and in arithmetic, and the text of a quoted value does not change. Query names match with the invariant culture.

FirstName has query name "first", Rating has query name "stars"

Title == "first == x"
  before: x => (x.Title == "FirstName == x")
  after:  x => (x.Title == "first == x")

(first, LastName) == "Paul"
  before: UnknownFilterPropertyException: first
  after:  x => ((x.FirstName == "Paul") OrElse (x.LastName == "Paul"))

(stars + 0) > 3
  before: ArgumentException: Property 'stars' not found on type 'TestingPerson'
  after:  x => ((x.Rating + Convert(0, Nullable`1)) > Convert(3, Nullable`1))

FIRST == "x"   (tr-TR culture)
  before: UnknownFilterPropertyException: The filter property 'FIRST' was not recognized.
  after:  x => (x.FirstName == "x")

Behavior change: a query name inside a quoted value or after a dot is not replaced anymore.

A property path works on the right side of a comparison (#196)

The right side of a comparison read only one name, so the README examples with a path on the right side threw. Now the right side reads a dotted path and resolves it like the left side. The path obeys MaxPropertyDepth and PreventFilter.

Title == Author.Name
  before: ParsingException
  after:  x => (x.Title == x.Author.Name)

Title == Author.Nmae
  before: ParsingException
  after:  UnknownFilterPropertyException: The filter property 'Nmae' was not recognized.

Title == "foo.bar"
  before and after: x => (x.Title == "foo.bar")

Behavior change: an unquoted dotted value that is not a property, for example Title == foo.bar, throws UnknownFilterPropertyException instead of ParsingException. Quote the text to compare with a literal value.

A child collection member resolves in any case (#179)

The first member after a collection needed the exact case, and an unknown member threw NullReferenceException.

ingredients.name == "flour"
  before: NullReferenceException
  after:  x => x.Ingredients.Select(y => y.Name).Any(z => (z == "flour"))

Ingredients.Nope == "x"
  before: NullReferenceException
  after:  UnknownFilterPropertyException: The filter property 'Nope' was not recognized.

Behavior change: an unknown member after a collection follows AllowUnknownProperties.

HasConversion is found by the property path (#198)

Every HasConversion lookup now uses the property path, so a query name no longer changes how a converted property filters. A converted number, enum, or Guid is read by its own type.

Email has HasConversion to string

Email == null
  before: compares with new EmailAddress("null"), 0 rows
  after:  compares with null, rows with a null Email

Email.Value == "a@x.com"   (Email has a query name)
  before: x.Email.Value == "a@x.com"
  after:  x.Email == new EmailAddress("a@x.com"), the same as without a query name

Number == 2   (Number is an int with HasConversion<string>() and no query name)
  before: ParsingException, inner InvalidOperationException: The binary operator Equal is not defined for the types 'System.Int32' and 'System.String'.
  after:  compares the int, 1 row

Behavior change: a null literal on a converted reference type matches null rows. A converted Guid gives a Guid constant in the expression text. The rows for a Guid do not change.

Alias replacement keeps a query name after a dot (#127)

Only code that calls QueryKitPropertyMappings.ReplaceAliasesWithPropertyPaths directly sees this change. The segment after a dot belongs to another type, so the old replacement gave a wrong path.

Title has HasQueryName("name")

ReplaceAliasesWithPropertyPaths("Author.Name == \"x\" && name == \"y\"")
  before: Author.Title == "x" && Title == "y"
  after:  Author.Name == "x" && Title == "y"

Values and literals

A comma inside a quoted list item stays in the item (#142)

The list text was split on every comma, so a filter could not match a value that holds a comma. A custom operation still gets the list text as before.

Title ^^ ["Warm, with syrup", "a\b"]
  before: 3 items: Warm | with syrup | a\b - the title "Warm, with syrup" does not match
  after:  2 items: Warm, with syrup | a\b - the title "Warm, with syrup" matches

Title ^^ ["a,b,c"]
  before: 3 values
  after:  1 value: a,b,c

Age ^^ [20, 30, 40]
  before and after: 3 values

Behavior change: a client that sends ["a,b,c"] and means three values must send ["a","b","c"].

A quoted "null" is text for a string property (#189)

Title == "null"
  before: x => (x.Title == null)
  after:  x => (x.Title == "null")

Title @= "null"
  before: x.Title.Contains(null)
  after:  x => ((x.Title != null) AndAlso x.Title.Contains("null"))

Title == null
  before and after: x => (x.Title == null)

Age == "null"
  before and after: x => (x.Age == null)

A quoted custom operation value stays a string (#118)

A quoted value for a custom operation became an int, a decimal, a bool, or null. A client could not send a code with leading zeros. A quoted date or Guid still converts, and an unquoted value converts as before.

Custom operation sku_is is x.Sku == (string)value, and Pancakes has Sku 001

sku_is == "001"   (in memory)
  before: InvalidCastException, the operation gets the int 1
  after:  Pancakes

sku_is == "001"   (Postgres)
  before: 0 rows, r."Sku" = 1::text
  after:  Pancakes, r."Sku" = '001'::text

total_stock_above == 20   (unquoted)
  before and after: Beef Stew

Behavior change: a custom operation that casts a quoted value, for example (int)value for "10", throws InvalidCastException. Send the value without quotes to get a number, a boolean, or null.

A number always uses the '.' decimal point (#181)

The number grammar also accepted the decimal separator of the current culture. Under de-DE, Rating > 4,5 parsed, and the value became 45. The API returned wrong rows with no error.

de-DE culture

Price > 4,5
  before: 0 rows, the value is 45
  after:  ParsingException: There was a parsing failure, likely due to an invalid comparison or logical operator. ... Failed at Line 1, Column 11.

Price > 4.5
  before and after: Beef Stew

Behavior change: a number with a decimal comma throws in every culture. Send the number with a '.' decimal point.

A date in a list uses the invariant culture (#190)

A list of DateOnly, TimeOnly, or DateTimeOffset values used the current culture. A single comparison already used the invariant culture.

Date ^^ ["2024-01-15"]   (ar-SA culture)
  before: FormatException
  after:  matches 2024-01-15

Date ^^ ["2024-01-15"]   (th-TH culture)
  before: 0 rows (read with the Thai Buddhist calendar)
  after:  matches 2024-01-15

Date == "2024-01-15"   (any culture)
  before and after: matches 2024-01-15

A quoted time keeps a short fraction (#183)

A quoted time dropped a fraction with fewer than 3 digits.

Time == "08:30:00.5"
  before: matches 08:30:00
  after:  matches 08:30:00.500

Time == "08:30:00.500"
  before and after: matches 08:30:00.500

Behavior change: Time == "08:30:00.5" matches 08:30:00.500, not 08:30:00. Remove the fraction to match the whole second.

The zone of a date and time comes after the fraction (#165)

The date parser accepted the zone before the fraction, which is not an ISO 8601 format.

SpecificDateTime == 2024-01-15T08:00:00Z.5
  before: parses
  after:  ParsingException

SpecificDate == 2024-01-15T10:00:00+02:00.500
  before: parses
  after:  ParsingException

SpecificDateTime == 2024-01-15T08:00:00.5Z
  before and after: parses

Behavior change: an unquoted date and time value with the zone before the fraction throws ParsingException. Write the zone after the fraction.

A DateTimeOffset value is converted to UTC (#184)

A DateTimeOffset constant kept its offset, and only a parameter went to UTC. Now both paths give the same value. The instant does not change, so the rows do not change.

SpecificDate == 2022-07-01T00:00:03+01:00
  before: new DateTimeOffset(637922304030000000, 01:00:00)
  after:  new DateTimeOffset(637922268030000000, 00:00:00)

Behavior change: the expression text and the Offset of the value change.

A TimeOnly value works on net6.0 (#176, #200)

On net6.0, every filter with a TimeOnly value threw, because the TimeOnly constructor that QueryKit used needs .NET 7. Now the value is a constant when that constructor does not exist. A new test project runs this filter on the .NET 6 runtime in CI.

net6.0

Time == "08:30:00"
  before: ArgumentNullException: Value cannot be null. (Parameter 'constructor')
  after:  x => (x.Time == 08:30)

Operators

Case-sensitive string operators check for null (#185)

The case-sensitive @=, _=, _-=, and their negations had no null check. In memory, a null property threw NullReferenceException. The case-insensitive forms already had the check.

Title values: null, "lamb", "other" (in memory)

Title _= "la"
  before: NullReferenceException
  after:  lamb

Title !_= "la"
  before: NullReferenceException
  after:  null, other

expression for Title _= "lam"
  before: x => x.Title.StartsWith("lam")
  after:  x => ((x.Title != null) AndAlso x.Title.StartsWith("lam"))

Behavior change: the expression text has a null check. On a database, the rows do not change.

String operators reject a null value (#187)

Title @= null built x.Title.Contains(null). In memory, the query threw when it ran. On Postgres, it returned no rows. Now the filter throws when it is parsed.

Directions @= null
  before: ArgumentNullException when the query runs (in memory), 0 rows (Postgres)
  after:  QueryKitParsingException: The '@=' operator does not accept a null value. Use '==' or '!=' to compare with null.

Behavior change: a null value on the right side of @=, _=, _-=, !@=, !_=, or !_-= (and their case-insensitive forms) throws QueryKitParsingException. Use == or != to compare with null.

Does-not-have excludes collections with the value (#188)

Tags !^$ "sweet" matched every collection with any other value. It returned rows that have the value and left out empty collections. On Postgres, EF could not translate it. Now it is the negation of has.

Pancakes [breakfast, sweet], Beef Stew [dinner], Salt Bread [bread, Winner], Plain Water []

Tags !^$ "sweet"   (in memory)
  before: Pancakes, Beef Stew, Salt Bread
  after:  Beef Stew, Salt Bread, Plain Water

Tags !^$ "sweet"   (Postgres)
  before: InvalidOperationException
  after:  Beef Stew, Salt Bread, Plain Water

Behavior change: !^$ returns only collections without the value, and empty collections too.

The operator factories pass usesAll (#186)

The public ComparisonOperator factories took a usesAll argument but did not use it. Filter strings do not change.

Ingredients.Name, value "waffle"

ComparisonOperator.EqualsOperator(usesAll: true)
  before: Any - a recipe with one waffle ingredient matches
  after:  All - only a recipe where every ingredient is waffle matches

Behavior change: code that passed usesAll: true and expected Any must pass usesAll: false.

Operator alias text stays inside a quoted value (#195)

Aliases: and, or, eq, gt

Title eq "salt and pepper or eq gt" and Rating gt 3
  before: x.Title == "salt && pepper || == gt" AndAlso x.Rating > 3
  after:  x => ((x.Title == "salt and pepper or eq gt") AndAlso (x.Rating > 3))

Parser

A long && or || chain cannot overflow the stack (#191)

The parser recursed once for each logical operator. A flat chain of about 212 clauses overflowed a 256 KB stack and stopped the process. Now the parser reads the chain in a loop. Precedence and the expression text do not change.

256 KB stack

Age == 1 || Age == 2 || ... (300 clauses)
  before: stack overflow, the process stops
  after:  parses

Age == 1 && Title == "a" || Age == 2
  before and after: x => (((x.Age == 1) AndAlso (x.Title == "a")) OrElse (x.Age == 2))

EF Core still reads the tree with recursion. It overflowed at about 729 clauses on a 1 MB stack. Use MaxInputLength to limit the size of a filter.

Build

• The integration tests pull the Testcontainers images through mirror.gcr.io, because Docker Hub limited anonymous pulls in CI (#199)
• The Unit Tests workflow also runs QueryKit.Net6Tests on the .NET 6 runtime (#200)

Full Changelog: v1.15.0...v1.16.0