Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ARROW-16547: [Python] to_pandas fails with FixedOffset timezones when timestamp_as_object is used #14448

Merged
merged 11 commits into from Oct 27, 2022
Merged

ARROW-16547: [Python] to_pandas fails with FixedOffset timezones when timestamp_as_object is used #14448

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
7e05650
First try of replacing .fromutc() method in arrow_to_pandas.cc
AlenkaF Oct 18, 2022
464377e
Make corrections - thank you Miles
AlenkaF Oct 18, 2022
c64b8c6
Use OwnedRef to avoid a list of Py_DECREF
AlenkaF Oct 19, 2022
439089e
Add python code to the comment
AlenkaF Oct 19, 2022
d5097e8
Replace PyObject_CallOneArg with PyObject_CallObject to see if that f…
AlenkaF Oct 19, 2022
dd4b09a
Add a test to test_pandas.py
AlenkaF Oct 19, 2022
64c6c6c
Change PyObject_CallObject to PyObject_CallMethod
AlenkaF Oct 19, 2022
2ae9bfc
Apply suggestion from code review- Joris
AlenkaF Oct 19, 2022
f94d5bf
Change the check in the test
AlenkaF Oct 20, 2022
bc8410e
Re-apply suggestion and correct the test
AlenkaF Oct 20, 2022
c145440
Update python/pyarrow/tests/test_pandas.py
AlenkaF Oct 26, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
14 changes: 13 additions & 1 deletion python/pyarrow/src/arrow/python/arrow_to_pandas.cc
View file
Open in desktop
Expand Up @@ -1074,11 +1074,23 @@ struct ObjectWriterVisitor {
auto ConvertTimezoneAware = [&](typename Type::c_type value, PyObject** out) {
PyObject* naive_datetime;
RETURN_NOT_OK(ConvertTimezoneNaive(value, &naive_datetime));

// convert the timezone naive datetime object to timezone aware
AlenkaF marked this conversation as resolved.
Show resolved Hide resolved
*out = PyObject_CallMethod(tzinfo.obj(), "fromutc", "O", naive_datetime);
// two step conversion of the datetime mimics Python's code:
// dt.replace(tzinfo=datetime.timezone.utc).astimezone(tzinfo)
// first step: replacing timezone with timezone.utc (replace method)
OwnedRef args(PyTuple_New(0));
OwnedRef keywords(PyDict_New());
PyDict_SetItemString(keywords.obj(), "tzinfo", PyDateTime_TimeZone_UTC);
OwnedRef naive_datetime_replace(PyObject_GetAttrString(naive_datetime, "replace"));
OwnedRef datetime_utc(PyObject_Call(naive_datetime_replace.obj(), args.obj(), keywords.obj()));
// second step: adjust the datetime to tzinfo timezone (astimezone method)
*out = PyObject_CallMethod(datetime_utc.obj(), "astimezone", "O", tzinfo.obj());

// the timezone naive object is no longer required
Py_DECREF(naive_datetime);
RETURN_IF_PYERROR();

return Status::OK();
};

Expand Down
12 changes: 12 additions & 0 deletions python/pyarrow/tests/test_pandas.py
View file
Open in desktop
Expand Up @@ -4472,6 +4472,18 @@ def test_timestamp_as_object_non_nanosecond(resolution, tz, dt):
assert result[0] == expected


def test_timestamp_as_object_fixed_offset():
# ARROW-16547 to_pandas with timestamp_as_object=True and FixedOffset
pytz = pytest.importorskip("pytz")
import datetime
timezone = pytz.FixedOffset(120)
dt = timezone.localize(datetime.datetime(2022, 5, 12, 16, 57))

table = pa.table({"timestamp_col": pa.array([dt])})
result = table.to_pandas(timestamp_as_object=True)
assert pa.table(result) == table


def test_threaded_pandas_import():
invoke_script("pandas_threaded_import.py")

Expand Down