ENH: An argument for .query() that returns all rows

[astrowonk]

Is your feature request related to a problem?

Pandas query strings are very useful, however when using a variable to pass in the query, e.g.

df.query(my_query)

There is no null or empty my_query that will return all rows. An empty string returns a ValueError. Any non-string argument returns an error.

Describe the solution you'd like

An empty string or None or some keyword argument (even with an empty expr to .query() should return all rows.

Additional context

Without the ability to pass something to .query that returns all rows you end up with something like this, when _query may or may not set. It'd be nice to be able to avoid this conditional by having something to pass to query that returns all rows.

if _query:
   dfs = [f(x).query(_query) for x in the_list]
else:
   dfs = [f(x) for x in list]

[ukarroum]

Can you give an example of use cases, when you think having this feature can be useful ? (As oposed to just using df instead of df.query(None?) )

As mentionned here : https://stackoverflow.com/a/53861261/14517936 , if you really have to use df.query you can always do :

ALL = slice(None)
df.query('@ALL', engine='python')

or :

df.query('tuple()')

if your df is not a multiindexed.

[astrowonk]

I have a variable _query that is sometimes None, sometimes various different user defined queries. Right now I'm doing something like the conditional in my original post, so I check the _query and don't call .query(_query) if it's None.

I'm sorry I did not find that stack overflow. I read the documentation for query which did not mention any way to return all rows. It seems to me that adding a supported and documented way to return all rows would be useful to avoid conditionals, creating new functions, or other workarounds. Both tuple() and the slice(None) idea appear to work, but this feels like something that should be supported/documented.

I think I could submit a PR on this easily enough if it would be welcome. (either checking for None or an empty string.)

[rhshadrach]

I don't see the reason to avoid branching:

dfs = [f(x) for x in list]
if _query:
   dfs = [x.query(_query) for x in dfs]

In general, I don't think one should invent no-op parameters to try to avoid branching. However, no selection does seem to be a natural result of df.query("") and I have built query strings like the following:

qry_str = ""
if a is not None:
   qry_str += f"a == {a} and "
if b is not None:
   qry_str += f"b == {b} and "
if len(qry_str) > 0:
    qry_str = qry_str[:-4]
df = df.query(qry_str)

for which the no-op of the empty string would eliminate an edge case.

[astrowonk]

However, no selection does seem to be a natural result of df.query("") and I have built query strings like the following:

Exactly, I don't think we need to invent anything since the natural result of an empty string seems to be return all rows.

[astrowonk]

I don't see the reason to avoid branching:

dfs = [f(x) for x in list]

if _query:

   dfs = [x.query(_query) for x in dfs]

This seems like memory inefficient as it's building a list of every complete dataframe first then reducing it to a list of smaller dfs in the conditional. If empty string query returned all rows you'd not only save an if but use less peak memory, right? f(x).query(_query)

[rhshadrach]

What do you think of:

def maybe_query(df, query):
    result = df.query(query) if query else df
    return result

dfs = [f(x).pipe(maybe_query, query) for x in list]

In any case, my opposition is only to going out of ones way to create a no-op. For any method, df.method is the user asking pandas to do something with the DataFrame so it doesn't make sense to me to add a special dont_do_anything object to the logic so users can do df.method(dont_do_anything).

In this case, I think the empty string is a natural no-op for query, similar in spirit to if all([]).

[hchau630]

any updates on this? Seems like it is quite a simple feature to implement that would yield a big QOL improvement.

[wany-oh]

A more natural behavior when an empty string is given would be to return an empty frame rather than full rows.

[hchau630]

@wany-oh Do you have a use case in mind in which an empty query string should return an empty frame? As mentioned by @rhshadrach, a common use case (for me, and likely for many others) is to build up query strings in a similar manner as such

query = []
if a is not None:
   query.append(f"a == {a}")
if b is not None:
   query.append(f"b == {b}")
df = df.query(' and '.join(query))

in which case an empty query string naturally means 'select all'. Personally, I've never encountered a scenario in which I want an empty query string to return an empty dataframe.