Skip to content

[SPARK-35812][PYTHON] Throw ValueError if version and timestamp are used together in to_delta #33023

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
Yikun wants to merge 2 commits into from
Closed

[SPARK-35812][PYTHON] Throw ValueError if version and timestamp are used together in to_delta #33023

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
4352eea
Throw an error if version and timestamp are used together in DataFram…
Yikun Jun 22, 2021
118b5b7
Append doc
Yikun Jun 23, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 7 additions & 2 deletions python/pyspark/pandas/namespace.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -507,11 +507,14 @@ def read_delta(
Path to the Delta Lake table.
version : string, optional
Specifies the table version (based on Delta's internal transaction version) to read from,
using Delta's time travel feature. This sets Delta's 'versionAsOf' option.
using Delta's time travel feature. This sets Delta's 'versionAsOf' option. Note that
this paramter and `timestamp` paramter cannot be used together, otherwise it will raise a
`ValueError`.
timestamp : string, optional
Specifies the table version (based on timestamp) to read from,
using Delta's time travel feature. This must be a valid date or timestamp string in Spark,
and sets Delta's 'timestampAsOf' option.
and sets Delta's 'timestampAsOf' option. Note that this paramter and `version` paramter
cannot be used together, otherwise it will raise a `ValueError`.
index_col : str or list of str, optional, default: None
Index column of table in Spark.
options
Expand Down Expand Up @@ -562,6 +565,8 @@ def read_delta(
3 13
4 14
"""
if version is not None and timestamp is not None:
raise ValueError("version and timestamp cannot be used together.")
Copy link
Contributor

@cloud-fan cloud-fan Jun 23, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how about the document?

Copy link
Member Author

@Yikun Yikun Jun 23, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added the doc for version and timestamp

if "options" in options and isinstance(options.get("options"), dict) and len(options) == 1:
options = options.get("options") # type: ignore

Expand Down
9 changes: 8 additions & 1 deletion python/pyspark/pandas/tests/test_namespace.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@
import pandas as pd

from pyspark import pandas as ps
from pyspark.pandas.namespace import _get_index_map
from pyspark.pandas.namespace import _get_index_map, read_delta
from pyspark.pandas.utils import spark_column_equals
from pyspark.testing.pandasutils import PandasOnSparkTestCase
from pyspark.testing.sqlutils import SQLTestUtils
Expand Down Expand Up @@ -329,6 +329,13 @@ def check(actual, expected):

self.assertRaises(KeyError, lambda: _get_index_map(sdf, ["year", "hour"]))

def test_read_delta_with_wrong_input(self):
self.assertRaisesRegex(
ValueError,
"version and timestamp cannot be used together",
lambda: read_delta("fake_path", version="0", timestamp="2021-06-22"),
)


if __name__ == "__main__":
import unittest
Expand Down