-
Notifications
You must be signed in to change notification settings - Fork 28k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-16335][SQL] Structured streaming should fail if source directory does not exist #14002
Conversation
LGTM |
Test build #61570 has finished for PR 14002 at commit
|
Test build #3155 has finished for PR 14002 at commit
|
Test build #61583 has finished for PR 14002 at commit
|
Test build #3158 has finished for PR 14002 at commit
|
Test build #3157 has finished for PR 14002 at commit
|
Test build #61604 has finished for PR 14002 at commit
|
Test build #61605 has finished for PR 14002 at commit
|
Merging in master/2.0. |
…ry does not exist ## What changes were proposed in this pull request? In structured streaming, Spark does not report errors when the specified directory does not exist. This is a behavior different from the batch mode. This patch changes the behavior to fail if the directory does not exist (when the path is not a glob pattern). ## How was this patch tested? Updated unit tests to reflect the new behavior. Author: Reynold Xin <rxin@databricks.com> Closes #14002 from rxin/SPARK-16335. (cherry picked from commit d601894) Signed-off-by: Reynold Xin <rxin@databricks.com>
What changes were proposed in this pull request?
In structured streaming, Spark does not report errors when the specified directory does not exist. This is a behavior different from the batch mode. This patch changes the behavior to fail if the directory does not exist (when the path is not a glob pattern).
How was this patch tested?
Updated unit tests to reflect the new behavior.