Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Concat Series[datetimetz] & Series[category] loses timezone #23816

Closed
TomAugspurger opened this issue Nov 20, 2018 · 6 comments

Comments

Projects
None yet
4 participants
@TomAugspurger
Copy link
Contributor

commented Nov 20, 2018

In [8]: a = pd.Series(pd.date_range('20170101', periods=4, tz='US/Pacific'))

In [9]: b = pd.Series(['a', 'b'], dtype='category')

In [10]: pd.concat([a, b], ignore_index=True)[0]
Out[10]: Timestamp('2017-01-01 08:00:00')

Out[10] should have a tz like the following

In [11]: pd.concat([a, b.astype(object)], ignore_index=True)[0]
Out[11]: Timestamp('2017-01-01 00:00:00-0800', tz='US/Pacific', freq='D')
@evangelineliu

This comment has been minimized.

Copy link
Contributor

commented Nov 20, 2018

Hi,

Can my team and I contribute to this issue?

Thanks.

@TomAugspurger

This comment has been minimized.

Copy link
Contributor Author

commented Nov 20, 2018

Sure, you can give it a shot.

I'm not sure where the best place is for the fix yet. Somewhere in core/dtypes/concat.py, but it'll be a bit tricky to find the best time to convert to objects.

@TomAugspurger

This comment has been minimized.

Copy link
Contributor Author

commented Nov 30, 2018

That's the offset from the timezone ("US/Pacific").

Looking more closely, I think the issue is at

def _concat_asobject(to_concat):
.

We call np.asarray(x) on non-categorical, like our datetimes. I wonder if we could rewrite that as

to_concat = [np.asarray(x.astype(object)) for x in to_concat]

We know we need object dtype anyway, so there's no avoiding that. We need the asarray(...) for things like SparseArray, since SparseArray.astype(object) is still sparse.

Maybe give that change a shot and see if the tests still pass.

@evangelineliu

This comment has been minimized.

Copy link
Contributor

commented Nov 30, 2018

Okay I'll take a look at the suggested change. Thanks!

@evangelineliu

This comment has been minimized.

Copy link
Contributor

commented Dec 1, 2018

This should be closed

@jreback

This comment has been minimized.

Copy link
Contributor

commented Dec 1, 2018

@evangelineliu things are closed with a merged pull request

@gfyoung gfyoung added the Bug label Dec 2, 2018

@jreback jreback modified the milestones: Contributions Welcome, 0.24.0 Dec 4, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.