Skip to content

Commit

Permalink
[SPARK-30150][SQL] ADD FILE, ADD JAR, LIST FILE & LIST JAR Command do…
Browse files Browse the repository at this point in the history
… not accept quoted path

### What changes were proposed in this pull request?
`add file "abc.txt"` and `add file 'abc.txt'` are not supported.
For these two spark sql gives `FileNotFoundException`.
Only `add file abc.txt` is supported currently.

After these changes path can be given as quoted text for ADD FILE, ADD JAR, LIST FILE, LIST JAR commands in spark-sql

### Why are the changes needed?

In many of the spark-sql commands (like create table ,etc )we write path in quoted format only.  To maintain this consistency we should support quoted format with this command as well.

### Does this PR introduce any user-facing change?
Yes. Now users can write path with quotes.

### How was this patch tested?
Manually tested.

Closes #26779 from iRakson/SPARK-30150.

Authored-by: root1 <raksonrakesh@gmail.com>
Signed-off-by: Wenchen Fan <wenchen@databricks.com>
  • Loading branch information
iRakson authored and cloud-fan committed Dec 12, 2019
1 parent 3741a36 commit 2936507
Show file tree
Hide file tree
Showing 3 changed files with 25 additions and 2 deletions.
Expand Up @@ -222,7 +222,7 @@ statement
multipartIdentifier partitionSpec? #loadData
| TRUNCATE TABLE multipartIdentifier partitionSpec? #truncateTable
| MSCK REPAIR TABLE multipartIdentifier #repairTable
| op=(ADD | LIST) identifier .*? #manageResource
| op=(ADD | LIST) identifier (STRING | .*?) #manageResource
| SET ROLE .*? #failNativeCommand
| SET .*? #setConfiguration
| RESET #resetConfiguration
Expand Down
Expand Up @@ -308,9 +308,14 @@ class SparkSqlAstBuilder(conf: SQLConf) extends AstBuilder(conf) {
* ADD (FILE[S] <filepath ...> | JAR[S] <jarpath ...>)
* LIST (FILE[S] [filepath ...] | JAR[S] [jarpath ...])
* }}}
*
* Note that filepath/jarpath can be given as follows;
* - /path/to/fileOrJar
* - "/path/to/fileOrJar"
* - '/path/to/fileOrJar'
*/
override def visitManageResource(ctx: ManageResourceContext): LogicalPlan = withOrigin(ctx) {
val mayebePaths = remainder(ctx.identifier).trim
val mayebePaths = if (ctx.STRING != null) string(ctx.STRING) else remainder(ctx.identifier).trim
ctx.op.getType match {
case SqlBaseParser.ADD =>
ctx.identifier.getText.toLowerCase(Locale.ROOT) match {
Expand Down
Expand Up @@ -233,4 +233,22 @@ class SparkSqlParserSuite extends AnalysisTest {
parser.parsePlan("ALTER SCHEMA foo SET DBPROPERTIES ('x' = 'y')"))
assertEqual("DESC DATABASE foo", parser.parsePlan("DESC SCHEMA foo"))
}

test("manage resources") {
assertEqual("ADD FILE abc.txt", AddFileCommand("abc.txt"))
assertEqual("ADD FILE 'abc.txt'", AddFileCommand("abc.txt"))
assertEqual("ADD FILE \"/path/to/abc.txt\"", AddFileCommand("/path/to/abc.txt"))
assertEqual("LIST FILE abc.txt", ListFilesCommand(Array("abc.txt")))
assertEqual("LIST FILE '/path//abc.txt'", ListFilesCommand(Array("/path//abc.txt")))
assertEqual("LIST FILE \"/path2/abc.txt\"", ListFilesCommand(Array("/path2/abc.txt")))
assertEqual("ADD JAR /path2/_2/abc.jar", AddJarCommand("/path2/_2/abc.jar"))
assertEqual("ADD JAR '/test/path_2/jar/abc.jar'", AddJarCommand("/test/path_2/jar/abc.jar"))
assertEqual("ADD JAR \"abc.jar\"", AddJarCommand("abc.jar"))
assertEqual("LIST JAR /path-with-dash/abc.jar",
ListJarsCommand(Array("/path-with-dash/abc.jar")))
assertEqual("LIST JAR 'abc.jar'", ListJarsCommand(Array("abc.jar")))
assertEqual("LIST JAR \"abc.jar\"", ListJarsCommand(Array("abc.jar")))
assertEqual("ADD FILE /path with space/abc.txt", AddFileCommand("/path with space/abc.txt"))
assertEqual("ADD JAR /path with space/abc.jar", AddJarCommand("/path with space/abc.jar"))
}
}

0 comments on commit 2936507

Please sign in to comment.