BUG: Timestamp == date match stdlib

[jbrockmendel]

• closes #xxxx
• tests added / passed
• passes black pandas
• passes git diff upstream/master -u -- "*.py" | flake8 --diff
• whatsnew entry

ATM we have one reasonable behavior Timestamp("2020-09-04") == date(2020, 9, 4) and two un-reasonable behaviors: ``Timestamp("2020-09-04").tz_localize("US/Pacific") == date(2020, 9, 4), Timestamp.now() == Timestamp.now().date()`. Since the stdlib datetime doesnt consider `datetime(2020, 9, 4) == date(2020, 9, 4)`, this follows the stdlib and considers them never equal.

I'm still getting one test failure locally.

cc @mroeschke

[mroeschke]

Seems reasonable. In general do you think we should reel back datetime.date support in ops with datetime64 dtypes?

[jbrockmendel]

In general do you think we should reel back datetime.date support in ops with datetime64 dtypes?

Yes. I think indexing should behave like comparisons, in which case date objects would not be allowed. If we were to allow date objects in indexing, I think it would make more sense to treat date(2020, 9, 5) like "2020-09-05" than like datetime(2020, 9, 5), i.e. like a partial-slice. But i think thats more hassle than its worth.

[jorisvandenbossche]

Shouldn't we rather see this as an API change than a bug fix?
I would say yes. I can imagine people relying on this to filter to all timestamps on a given date.

[jbrockmendel]

I can imagine people relying on this to filter to all timestamps on a given date.

That's conceivable, but I think its much more likely that people are assuming that it casts to datetime-at-midnight and so are silently getting incorrect results.

[jorisvandenbossche]

its much more likely that people are assuming that it casts to datetime-at-midnight and so are silently getting incorrect results.

Well, and they will then also start to see a warning to make them aware of this.

---

Now, it's not only about the "timestamp getting normalized to midnight when compared to date" behaviour, because as I understand it with this PR also Timestamps as midnight will no longer compare equal? I think we should first deprecate that.

[jbrockmendel]

Updated to deprecate.

[jreback]

we are exposing this to the user?

[jreback]

any idea why this happens?

[jbrockmendel]

when it tries to make a MultiIndex out of the A and B columns, it goes through Categorical, which passes convert_dates=True to maybe_infer_to_datetimelike,

[jreback]

this should not be warnign at the top level

[jbrockmendel]

we could finagle it so that we dont pass convert_dates=True to maybe_infer_to_datetimelike in this case, but that gets pretty complicated, especially compared to just warning the user that date objects are finicky

[jbrockmendel]

Looks like a while back you were in favor of telling people not to pass dates to the MultiIndex constructors.

[jorisvandenbossche]

There are still some warnings raised by the tests. For example in the github actions build I see:

pandas/tests/indexes/datetimes/test_indexing.py::TestGetIndexer::test_get_indexer_out_of_bounds_date[target0-positions0]
pandas/tests/indexes/datetimes/test_indexing.py::TestGetIndexer::test_get_indexer_out_of_bounds_date[target1-positions1]
pandas/tests/indexes/datetimes/test_indexing.py::TestGetIndexer::test_get_indexer_out_of_bounds_date[target2-positions2]
  /home/runner/work/pandas/pandas/pandas/core/indexes/base.py:3297: FutureWarning: Comparison of Timestamp with datetime.date is deprecated in order to match the standard library behavior.  In a future version these will be considered non-comparable.Use ts == pd.Timestamp(dt) instead.
    indexer = self._engine.get_indexer(target._get_engine_target())

[jreback]

cc @jorisvandenbossche

[jbrockmendel]

There are still some warnings raised by the tests. For example in the github actions build I see:

Huh, why isnt that failing the npdev build?

[jbrockmendel]

Updated test_get_indexer_out_of_bounds_date to catch the warning. Weird thing is that locally every-other-run it fails saying it the expected warning was not emitted.

[jreback]

hmm https://dev.azure.com/pandas-dev/pandas/_build/results?buildId=51340&view=logs&j=f39ddbca-85f6-5283-3bfe-05832bcf5208&t=60a5c9af-5075-529a-e870-f4d886d8e7ce

now builds are failing because of warnings

[jbrockmendel]

now builds are failing because of warnings

harumph, yah that looks like the same thing where that was happening every-other-run locally

[jreback]

https://github.com/pandas-dev/pandas/pull/36131/checks?check_run_id=1631224554 looks legit

[jbrockmendel]

suppressed the warning, green

[jorisvandenbossche]

Why can the warning be suppressed in the tests? It's not a valid user case?

[jbrockmendel]

Why can the warning be suppressed in the tests? It's not a valid user case?

its flaky and i haven't figured out why

[jreback]

thanks @jbrockmendel