New query parser with numeric expressions and query plugins

[faubi]

This is a rewrite of the code for parsing queries. This allows numeric expressions such as "length + 8 seconds", "playcount * 2", and "(1-rating)*5 days" to be used on both the right and left side of numeric comparisons. Any numerical comparisons that parsed before should still work the same.

This expansion of numeric comparisons addresses issue #1492, and to some extent #198.

This also adds a "@(plugin)" syntax for plugins to extend queries with custom types of searches. This is backwards incompatible with some queries containing '@', however. Three query plugins are included: conditional queries, python expression queries, and including saved searches in queries.

The syntax for the query plugins:

• Conditional queries: @(if: condition query: then query: else query)
• Include saved search: @(saved: some search name)
• Python expressions: @(python: expression). The s variable is the song being matched.

Implementation-wise, this parser parses through each character individually instead of having a separate lexer, which allows the numerical expressions to be much more easily parsed.

The interface of the quodlibet.query.Query class is identical to before, although the internal implementation has changed somewhat.

The following test cases have been added to cover the additions:

tests/test_query.py: TQuery_is_valid.test_extension
tests/test_query.py: TQuery_is_valid.test_numexpr
tests/test_query.py: TQuery.test_numexpr
tests/test_query.py: TQuery.test_numexpr_date
tests/test_query__match.py: TQueryMatch
tests/plugin/test_query.py: TQueryPlugins

The test case TNumericOp in tests/test_query__match.py was removed because it contained only tests pertaining to the previous parser implementation.

I've tried to be backwards compatible with the old parser wherever possible. As far as I can tell, the only exception to this is queries containing '@' (which is now a special character).

This does keep backwards compatibility with date expressions such as '2007-07-19' in numeric comparisons by interpreting them as dates if the comparison contains a 'date' tag and as subtraction otherwise.

[pschwede]

Exciting! Is it also possible to have two variables in one @-expression? E.g.: @(~#playcount / ~#skipcount)

[faubi]

Numeric comparisons with more than one tag work, like #(playcount > skipcount) or #(length < 6 minutes * rating).

Extensions using @() can only have one plugin and argument each, but like other queries can still be combined using &() and |().

[pschwede]

Okay thanks for clearing up. So, I could filter out #(0.5 * playcount > skipcount) which is new in your queries and pretty useful!

However, these queries (conditions, exactly) won't work for column headers in browsers, right?

[lazka]

Sorry for the delay..

On the pull request:

• That amount of change seems a bit risky for not asking first or opening an
  issue... but ok..
• This pull requests really implements two separate things so it should be split
  in two pull requests.
• Both features seem like nice additions.

On the implementation/features:

Why isn't it possible to implement the numeric expressions on top of the
existing parser or as an additional parser?

[faubi]

I figured this pull request would be fairly unexpected, but after writing it primarily for my own use I decided to make a pull request in case it's worth including in the main project.

As for the implementation, the lexer parsed all / as the beginning of a regular expression, preventing it from also being used as the division operator in different context. Making it lexerless also greatly simplified the numerical expression parsing by preventing having to parse series of tokens like playcount *, (, length - 2 minutes, ) as a single expression.

If the loss of backwards compatibility with @ needs to be avoided, I think an option can be added to the preferences menu to enable both the loading of query plugins and the @() rule in the parser.

[lazka]

As for the implementation, the lexer parsed all / as the beginning of a regular expression, preventing it from also being used as the division operator in different context. Making it lexerless also greatly simplified the numerical expression parsing by preventing having to parse series of tokens like playcount *, (, length - 2 minutes, ) as a single expression.

I see, thanks. I would hav epreferred if the numexp parsing was separate, but the new code looks readable/documented enough..

If the loss of backwards compatibility with @ needs to be avoided, I think an option can be added to the preferences menu to enable both the loading of query plugins and the @() rule in the parser.

I don't think that's a big problem..

One problem I see is that spaces in tags no longer work and escaping them doesn't help. e.g. my tag=foono longer works.

[faubi]

Oh, the tag parsing must have been overly restrictive. Tags containing spaces should now work properly.

[pschwede]

#(rating >= (playcount+1) / (skipcount+1)) is reeeally useful! 👍

[spxxxk]

Can you provide examples for the @() syntax ? I can't figure it out on my own ...

[faubi]

With the python expression query plugin:

• @(python: s('genre') in s('title')) find songs where the title contains the genre name. That's a bit contrived, but basically any python expression with the tags works.

With the saved search plugin:

• &(@(saved: favorites), genre=rock) finds songs in the saved search titled 'Favorites' whose genre is rock.
• |(@(saved: mysearch), @(saved: another)) finds songs in either the saved search 'mysearch' or the search 'another'

With the conditional plugin:

• @(if: #(rating > 0.7), genre=|(classical, electronic), genre=classical) finds classical music regardless of rating but electronic music only if it's rated highly.

None of these work unless the corresponding plugin is first enabled in the plugins menu.

[spxxxk]

Thanks , enabled in the plugins was the missing bit...
Nice additions. I will definitely have a use for them.

Note : @-queries appear valid (turn green) even if plugins aren't enabled. Not sure what can be done about it.

[spxxxk]

By any chance, does the @(python: expression) syntax support sorting by tag ? For example, sorting by last played time ?

[lazka]

By any chance, does the @(python: expression) syntax support sorting by tag ? For example, sorting by last played time ?

No, only filtering.