New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

New query parser with numeric expressions and query plugins #1799

Merged
merged 54 commits into from Feb 1, 2016

Conversation

Projects
None yet
4 participants
@faubi

faubi commented Jan 27, 2016

This is a rewrite of the code for parsing queries. This allows numeric expressions such as "length + 8 seconds", "playcount * 2", and "(1-rating)*5 days" to be used on both the right and left side of numeric comparisons. Any numerical comparisons that parsed before should still work the same.

This expansion of numeric comparisons addresses issue #1492, and to some extent #198.

This also adds a "@(plugin)" syntax for plugins to extend queries with custom types of searches. This is backwards incompatible with some queries containing '@', however. Three query plugins are included: conditional queries, python expression queries, and including saved searches in queries.

The syntax for the query plugins:

  • Conditional queries: @(if: condition query: then query: else query)
  • Include saved search: @(saved: some search name)
  • Python expressions: @(python: expression). The s variable is the song being matched.

Implementation-wise, this parser parses through each character individually instead of having a separate lexer, which allows the numerical expressions to be much more easily parsed.

The interface of the quodlibet.query.Query class is identical to before, although the internal implementation has changed somewhat.

The following test cases have been added to cover the additions:

tests/test_query.py: TQuery_is_valid.test_extension
tests/test_query.py: TQuery_is_valid.test_numexpr
tests/test_query.py: TQuery.test_numexpr
tests/test_query.py: TQuery.test_numexpr_date
tests/test_query__match.py: TQueryMatch
tests/plugin/test_query.py: TQueryPlugins

The test case TNumericOp in tests/test_query__match.py was removed because it contained only tests pertaining to the previous parser implementation.

I've tried to be backwards compatible with the old parser wherever possible. As far as I can tell, the only exception to this is queries containing '@' (which is now a special character).

This does keep backwards compatibility with date expressions such as '2007-07-19' in numeric comparisons by interpreting them as dates if the comparison contains a 'date' tag and as subtraction otherwise.

@pschwede

This comment has been minimized.

Contributor

pschwede commented Jan 29, 2016

Exciting! Is it also possible to have two variables in one @-expression? E.g.: @(~#playcount / ~#skipcount)

@faubi

This comment has been minimized.

faubi commented Jan 29, 2016

Numeric comparisons with more than one tag work, like #(playcount > skipcount) or #(length < 6 minutes * rating).

Extensions using @() can only have one plugin and argument each, but like other queries can still be combined using &() and |().

@pschwede

This comment has been minimized.

Contributor

pschwede commented Jan 29, 2016

Okay thanks for clearing up. So, I could filter out #(0.5 * playcount > skipcount) which is new in your queries and pretty useful!

However, these queries (conditions, exactly) won't work for column headers in browsers, right?

@lazka

This comment has been minimized.

Member

lazka commented Jan 31, 2016

Sorry for the delay..

On the pull request:

  • That amount of change seems a bit risky for not asking first or opening an
    issue... but ok..
  • This pull requests really implements two separate things so it should be split
    in two pull requests.
  • Both features seem like nice additions.

On the implementation/features:

Why isn't it possible to implement the numeric expressions on top of the
existing parser or as an additional parser?

@faubi

This comment has been minimized.

faubi commented Feb 1, 2016

I figured this pull request would be fairly unexpected, but after writing it primarily for my own use I decided to make a pull request in case it's worth including in the main project.

As for the implementation, the lexer parsed all / as the beginning of a regular expression, preventing it from also being used as the division operator in different context. Making it lexerless also greatly simplified the numerical expression parsing by preventing having to parse series of tokens like playcount *, (, length - 2 minutes, ) as a single expression.

If the loss of backwards compatibility with @ needs to be avoided, I think an option can be added to the preferences menu to enable both the loading of query plugins and the @() rule in the parser.

@lazka

This comment has been minimized.

Member

lazka commented Feb 1, 2016

As for the implementation, the lexer parsed all / as the beginning of a regular expression, preventing it from also being used as the division operator in different context. Making it lexerless also greatly simplified the numerical expression parsing by preventing having to parse series of tokens like playcount *, (, length - 2 minutes, ) as a single expression.

I see, thanks. I would hav epreferred if the numexp parsing was separate, but the new code looks readable/documented enough..

If the loss of backwards compatibility with @ needs to be avoided, I think an option can be added to the preferences menu to enable both the loading of query plugins and the @() rule in the parser.

I don't think that's a big problem..

One problem I see is that spaces in tags no longer work and escaping them doesn't help. e.g. my tag=foono longer works.

@faubi

This comment has been minimized.

faubi commented Feb 1, 2016

Oh, the tag parsing must have been overly restrictive. Tags containing spaces should now work properly.

lazka added a commit that referenced this pull request Feb 1, 2016

Merge pull request #1799 from faubiguy/newparser
New query parser with numeric expressions and query plugins

@lazka lazka merged commit 39630b4 into quodlibet:master Feb 1, 2016

1 check passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details
@pschwede

This comment has been minimized.

Contributor

pschwede commented Feb 11, 2016

#(rating >= (playcount+1) / (skipcount+1)) is reeeally useful! 👍

@spxxxk

This comment has been minimized.

spxxxk commented Feb 12, 2016

Can you provide examples for the @() syntax ? I can't figure it out on my own ...

@faubi

This comment has been minimized.

faubi commented Feb 12, 2016

With the python expression query plugin:

  • @(python: s('genre') in s('title')) find songs where the title contains the genre name. That's a bit contrived, but basically any python expression with the tags works.

With the saved search plugin:

  • &(@(saved: favorites), genre=rock) finds songs in the saved search titled 'Favorites' whose genre is rock.
  • |(@(saved: mysearch), @(saved: another)) finds songs in either the saved search 'mysearch' or the search 'another'

With the conditional plugin:

  • @(if: #(rating > 0.7), genre=|(classical, electronic), genre=classical) finds classical music regardless of rating but electronic music only if it's rated highly.

None of these work unless the corresponding plugin is first enabled in the plugins menu.

@spxxxk

This comment has been minimized.

spxxxk commented Feb 12, 2016

Thanks , enabled in the plugins was the missing bit...
Nice additions. I will definitely have a use for them.

Note : @-queries appear valid (turn green) even if plugins aren't enabled. Not sure what can be done about it.

@spxxxk

This comment has been minimized.

spxxxk commented Apr 21, 2016

By any chance, does the @(python: expression) syntax support sorting by tag ? For example, sorting by last played time ?

@lazka

This comment has been minimized.

Member

lazka commented May 1, 2016

By any chance, does the @(python: expression) syntax support sorting by tag ? For example, sorting by last played time ?

No, only filtering.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment