Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: SQL interface "off-by-one' indexing error with GROUP BY clauses that use position ordinals #15584

Merged
merged 1 commit into from Apr 11, 2024

Conversation

alexander-beedie
Copy link
Collaborator

@alexander-beedie alexander-beedie commented Apr 10, 2024

Closes #14704.

We already had all the machinery in place for handling "GROUP BY" clauses1 that use ordinal position values, but were defeated by a classic "off by one" error (as SQL is 1-indexed). Fixed it and added the missing test coverage. Will look at adding support for ordinal position values in "ORDER BY" clauses separately.

(Having ordinal position values in both clauses is supported by standard PostgreSQL syntax).

Example

import polars as pl

df = pl.DataFrame({
    "a": ["xx", "yy", "xx", "yy", "xx", "zz"],
    "b": [1, 1, 1, 2, 2, 2],
    "c": [99, 99, 66, 66, 66, 66],
})

with pl.SQLContext(frame=df) as ctx:
    df = ctx.execute(
        """
        SELECT NULL::date as dt, c, SUM(b) AS total_b
        FROM frame
        GROUP BY 2, 1
        """
    ).collect()
    
    # shape: (2, 3)
    # ┌──────┬─────┬─────────┐
    # │ dt   ┆ c   ┆ total_b │
    # │ ---  ┆ --- ┆ ---     │
    # │ date ┆ i64 ┆ i64     │
    # ╞══════╪═════╪═════════╡
    # │ null ┆ 66  ┆ 7       │
    # │ null ┆ 99  ┆ 2       │
    # └──────┴─────┴─────────┘

Footnotes

  1. https://www.postgresql.org/docs/current/sql-select.html#SQL-GROUPBY

@github-actions github-actions bot added fix Bug fix python Related to Python Polars rust Related to Rust Polars labels Apr 10, 2024
@alexander-beedie alexander-beedie changed the title fix: SQL engine 'off-by-one' indexing error with "GROUP BY" clauses that use position ordinals fix: SQL engine "off-by-one' indexing error with GROUP BY clauses that use position ordinals Apr 10, 2024
@alexander-beedie alexander-beedie added the A-sql Area: Polars SQL functionality label Apr 10, 2024
Copy link

codecov bot commented Apr 10, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 81.11%. Comparing base (835d198) to head (8c9a081).

Additional details and impacted files
@@           Coverage Diff           @@
##             main   #15584   +/-   ##
=======================================
  Coverage   81.11%   81.11%           
=======================================
  Files        1367     1367           
  Lines      174934   174934           
  Branches     2530     2530           
=======================================
+ Hits       141890   141902   +12     
+ Misses      32569    32557   -12     
  Partials      475      475           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copy link

codspeed-hq bot commented Apr 10, 2024

CodSpeed Performance Report

Merging #15584 will not alter performance

Comparing alexander-beedie:fix-sql-ordinal-group-by (8c9a081) with main (835d198)

Summary

✅ 22 untouched benchmarks

@alexander-beedie alexander-beedie changed the title fix: SQL engine "off-by-one' indexing error with GROUP BY clauses that use position ordinals fix: SQL interface "off-by-one' indexing error with GROUP BY clauses that use position ordinals Apr 11, 2024
Copy link
Collaborator

@reswqa reswqa left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Classic "off by one" error 😆

@stinodego stinodego merged commit c8e26ca into pola-rs:main Apr 11, 2024
31 checks passed
@alexander-beedie alexander-beedie deleted the fix-sql-ordinal-group-by branch April 11, 2024 13:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-sql Area: Polars SQL functionality fix Bug fix python Related to Python Polars rust Related to Rust Polars
Projects
None yet
Development

Successfully merging this pull request may close these issues.

unexpected SQL result with numbered entities for GROUP BY
3 participants