Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SPARK-38236][SQL][3.2][3.1] Check if table location is absolute by "new Path(locationUri).isAbsolute" in create/alter table #35591

Closed
wants to merge 5 commits into from

Commits on Feb 21, 2022

  1. [SPARK-38236][SQL] Treat table location as absolute when the first le…

    …tter of its path is slash in create/alter table
    
    After apache#28527, we change to create table under the database location when the table location is relative. However the criteria to determine if a table location is relative/absolute is `URI.isAbsolute`, which basically checks if the table location URI has a scheme defined. So table URIs like `/table/path` are treated as relative and the scheme and authority of the database location URI are used to create the table. For example, when the database location URI is `s3a://bucket/db`, the table will be created at `s3a://bucket/table/path`, while it should be created under the file system defined in `SessionCatalog.hadoopConf` instead.
    
    This change fixes that by treating table location as absolute when the first letter of its path is slash.
    
    This also applies to alter table.
    
    This is to fix the behavior described above.
    
    Yes. When users try to create/alter a table with a location that starts with a slash but without a scheme defined, the table will be created under/altered to the file system defined in `SessionCatalog.hadoopConf`, instead of the one defined in the database location URI.
    
    Updated unit tests.
    
    Closes apache#35462 from bozhang2820/spark-31709.
    
    Authored-by: Bo Zhang <bo.zhang@databricks.com>
    Signed-off-by: Wenchen Fan <wenchen@databricks.com>
    bozhang2820 committed Feb 21, 2022
    Configuration menu
    Copy the full SHA
    89d4cc1 View commit details
    Browse the repository at this point in the history
  2. Fix tests

    bozhang2820 committed Feb 21, 2022
    Configuration menu
    Copy the full SHA
    13e48fe View commit details
    Browse the repository at this point in the history

Commits on Feb 22, 2022

  1. Configuration menu
    Copy the full SHA
    004d0df View commit details
    Browse the repository at this point in the history
  2. Trigger tests again

    bozhang2820 committed Feb 22, 2022
    Configuration menu
    Copy the full SHA
    0b94f18 View commit details
    Browse the repository at this point in the history

Commits on Feb 23, 2022

  1. Configuration menu
    Copy the full SHA
    e23eeb4 View commit details
    Browse the repository at this point in the history