Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

expect_column_values_to_be_in_set fails on datetimes #422

Closed
RoyalTS opened this issue Apr 11, 2019 · 2 comments
Closed

expect_column_values_to_be_in_set fails on datetimes #422

RoyalTS opened this issue Apr 11, 2019 · 2 comments

Comments

@RoyalTS
Copy link
Contributor

RoyalTS commented Apr 11, 2019

This seems like it should validate correctly but doesn't:

import pandas as pd
import great_expectations as ge
 
df = pd.DataFrame({'a': pd.to_datetime(['2018-01-01'])})
df_ge = ge.from_pandas(df)
df_ge.expect_column_values_to_be_in_set('a', pd.to_datetime(['2018-01-01']))
@jcampbell
Copy link
Member

The issue here is that GE is converting your values set to string prior to saving the expectation (to ensure the expectation configuration objects are fully json serializable), so the resulting object fails the comparison to your datetime object in the dataframe.

Because datetimes are such an important case, for several other expectations (e.g. expect_column_values_to_be_between), we specifically allow you to set a flag parse_strings_as_datetimes to enable this type of comparison; that should clearly be available in this case.

@jcampbell
Copy link
Member

The new PR addresses the acute pain of this issue, but it's still counterintuitive to have to specify parse_strings_as_datetimes when none of the types you're passing in are strings--it's accommodating the need for the generic storage format of the expectation configuration.

I think this touches on the need for types #207 and so if @RoyalTS agrees suggest we close this once we merge the PR as an acute fix and follow this issue there.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants