Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

1011 Running sql using sparkconnect should not print full stack trace #1012

Open
wants to merge 9 commits into
base: master
Choose a base branch
from
Open

1011 Running sql using sparkconnect should not print full stack trace #1012

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
bf00376
Prune sparkConnect SQL stack trace on AnalysisException
b1ackout May 9, 2024
1143d25
Add AnalysisException on MissingPackageError check
b1ackout May 9, 2024
d7e0e8f
Lint
b1ackout May 10, 2024
11bedfa
Revert changes
b1ackout May 23, 2024
7578d8c
Fix typo
b1ackout May 23, 2024
7102f97
Add AnalysisException to is_non_sqlalchemy_error
b1ackout May 23, 2024
48c1623
Lint
b1ackout May 23, 2024
d951b42
Fix typo
b1ackout May 23, 2024
e95d599
Update is_non_sqlalchemy_error to support pysparks AnalysisException
b1ackout May 27, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
4 changes: 2 additions & 2 deletions src/sql/run/sparkdataframe.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,9 +9,9 @@


def handle_spark_dataframe(dataframe, should_cache=False):
"""Execute a ResultSet sqlaproxy using pysark module."""
"""Execute a ResultSet sqlaproxy using pyspark module."""
edublancas marked this conversation as resolved.
Show resolved Hide resolved
if not DataFrame and not CDataFrame:
raise exceptions.MissingPackageError("pysark not installed")
raise exceptions.MissingPackageError("pyspark not installed")

return SparkResultProxy(dataframe, dataframe.columns, should_cache)

Expand Down
17 changes: 13 additions & 4 deletions src/sql/util.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,12 @@
from sqlglot.errors import ParseError
from sqlalchemy.exc import SQLAlchemyError
from ploomber_core.dependencies import requires

try:
from pyspark.sql.utils import AnalysisException
except ModuleNotFoundError:
AnalysisException = None

import ast
from os.path import isfile
import re
Expand Down Expand Up @@ -556,11 +562,14 @@ def is_non_sqlalchemy_error(error):
"pyodbc.ProgrammingError",
# Clickhouse errors
"DB::Exception:",
# Pyspark
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed these as they are included in AnalysisException

"UNRESOLVED_ROUTINE",
"PARSE_SYNTAX_ERROR",
]
return any(msg in str(error) for msg in specific_db_errors)
is_pyspark_analysis_exception = (
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If AnalysisException is imported then checks if the error is of instance of pyspark's Analysis Exception and handles it accordingly

isinstance(error, AnalysisException) if AnalysisException else False
)
return (
any(msg in str(error) for msg in specific_db_errors)
or is_pyspark_analysis_exception
)


def if_substring_exists(string, substrings):
Expand Down