Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

New query parser with numeric expressions and query plugins #1799

Merged
merged 54 commits into from Feb 1, 2016

Conversation

@faubi
Copy link

@faubi faubi commented Jan 27, 2016

This is a rewrite of the code for parsing queries. This allows numeric expressions such as "length + 8 seconds", "playcount * 2", and "(1-rating)*5 days" to be used on both the right and left side of numeric comparisons. Any numerical comparisons that parsed before should still work the same.

This expansion of numeric comparisons addresses issue #1492, and to some extent #198.

This also adds a "@(plugin)" syntax for plugins to extend queries with custom types of searches. This is backwards incompatible with some queries containing '@', however. Three query plugins are included: conditional queries, python expression queries, and including saved searches in queries.

The syntax for the query plugins:

  • Conditional queries: @(if: condition query: then query: else query)
  • Include saved search: @(saved: some search name)
  • Python expressions: @(python: expression). The s variable is the song being matched.

Implementation-wise, this parser parses through each character individually instead of having a separate lexer, which allows the numerical expressions to be much more easily parsed.

The interface of the quodlibet.query.Query class is identical to before, although the internal implementation has changed somewhat.

The following test cases have been added to cover the additions:

tests/test_query.py: TQuery_is_valid.test_extension
tests/test_query.py: TQuery_is_valid.test_numexpr
tests/test_query.py: TQuery.test_numexpr
tests/test_query.py: TQuery.test_numexpr_date
tests/test_query__match.py: TQueryMatch
tests/plugin/test_query.py: TQueryPlugins

The test case TNumericOp in tests/test_query__match.py was removed because it contained only tests pertaining to the previous parser implementation.

I've tried to be backwards compatible with the old parser wherever possible. As far as I can tell, the only exception to this is queries containing '@' (which is now a special character).

This does keep backwards compatibility with date expressions such as '2007-07-19' in numeric comparisons by interpreting them as dates if the comparison contains a 'date' tag and as subtraction otherwise.

@pschwede
Copy link
Contributor

@pschwede pschwede commented Jan 29, 2016

Exciting! Is it also possible to have two variables in one @-expression? E.g.: @(~#playcount / ~#skipcount)

@faubi
Copy link
Author

@faubi faubi commented Jan 29, 2016

Numeric comparisons with more than one tag work, like #(playcount > skipcount) or #(length < 6 minutes * rating).

Extensions using @() can only have one plugin and argument each, but like other queries can still be combined using &() and |().

@pschwede
Copy link
Contributor

@pschwede pschwede commented Jan 29, 2016

Okay thanks for clearing up. So, I could filter out #(0.5 * playcount > skipcount) which is new in your queries and pretty useful!

However, these queries (conditions, exactly) won't work for column headers in browsers, right?

@lazka
Copy link
Member

@lazka lazka commented Jan 31, 2016

Sorry for the delay..

On the pull request:

  • That amount of change seems a bit risky for not asking first or opening an
    issue... but ok..
  • This pull requests really implements two separate things so it should be split
    in two pull requests.
  • Both features seem like nice additions.

On the implementation/features:

Why isn't it possible to implement the numeric expressions on top of the
existing parser or as an additional parser?

@faubi
Copy link
Author

@faubi faubi commented Feb 1, 2016

I figured this pull request would be fairly unexpected, but after writing it primarily for my own use I decided to make a pull request in case it's worth including in the main project.

As for the implementation, the lexer parsed all / as the beginning of a regular expression, preventing it from also being used as the division operator in different context. Making it lexerless also greatly simplified the numerical expression parsing by preventing having to parse series of tokens like playcount *, (, length - 2 minutes, ) as a single expression.

If the loss of backwards compatibility with @ needs to be avoided, I think an option can be added to the preferences menu to enable both the loading of query plugins and the @() rule in the parser.

@lazka
Copy link
Member

@lazka lazka commented Feb 1, 2016

As for the implementation, the lexer parsed all / as the beginning of a regular expression, preventing it from also being used as the division operator in different context. Making it lexerless also greatly simplified the numerical expression parsing by preventing having to parse series of tokens like playcount *, (, length - 2 minutes, ) as a single expression.

I see, thanks. I would hav epreferred if the numexp parsing was separate, but the new code looks readable/documented enough..

If the loss of backwards compatibility with @ needs to be avoided, I think an option can be added to the preferences menu to enable both the loading of query plugins and the @() rule in the parser.

I don't think that's a big problem..

One problem I see is that spaces in tags no longer work and escaping them doesn't help. e.g. my tag=foono longer works.

@faubi
Copy link
Author

@faubi faubi commented Feb 1, 2016

Oh, the tag parsing must have been overly restrictive. Tags containing spaces should now work properly.

lazka added a commit that referenced this pull request Feb 1, 2016
New query parser with numeric expressions and query plugins
@lazka lazka merged commit 39630b4 into quodlibet:master Feb 1, 2016
1 check passed
1 check passed
continuous-integration/travis-ci/pr The Travis CI build passed
Details
@pschwede
Copy link
Contributor

@pschwede pschwede commented Feb 11, 2016

#(rating >= (playcount+1) / (skipcount+1)) is reeeally useful! 👍

@spxxxk
Copy link

@spxxxk spxxxk commented Feb 12, 2016

Can you provide examples for the @() syntax ? I can't figure it out on my own ...

@faubi
Copy link
Author

@faubi faubi commented Feb 12, 2016

With the python expression query plugin:

  • @(python: s('genre') in s('title')) find songs where the title contains the genre name. That's a bit contrived, but basically any python expression with the tags works.

With the saved search plugin:

  • &(@(saved: favorites), genre=rock) finds songs in the saved search titled 'Favorites' whose genre is rock.
  • |(@(saved: mysearch), @(saved: another)) finds songs in either the saved search 'mysearch' or the search 'another'

With the conditional plugin:

  • @(if: #(rating > 0.7), genre=|(classical, electronic), genre=classical) finds classical music regardless of rating but electronic music only if it's rated highly.

None of these work unless the corresponding plugin is first enabled in the plugins menu.

@spxxxk
Copy link

@spxxxk spxxxk commented Feb 12, 2016

Thanks , enabled in the plugins was the missing bit...
Nice additions. I will definitely have a use for them.

Note : @-queries appear valid (turn green) even if plugins aren't enabled. Not sure what can be done about it.

@spxxxk
Copy link

@spxxxk spxxxk commented Apr 21, 2016

By any chance, does the @(python: expression) syntax support sorting by tag ? For example, sorting by last played time ?

@lazka
Copy link
Member

@lazka lazka commented May 1, 2016

By any chance, does the @(python: expression) syntax support sorting by tag ? For example, sorting by last played time ?

No, only filtering.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

4 participants