fix: Apply normalization to all dttm columns

[kgabryje]

SUMMARY

If python_date_format is set for a column, normalize the column to a datetime format. When that column is filtered by a timestamp value, denormalize the timestamp to the original format.

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

Before:

After:

TESTING INSTRUCTIONS

1. Create a chart using video_games_sales dataset from sample data
2. Use column year which has python_date_format set to %Y
3. Verify that the chart works correctly
4. Verify that in the "Results" table the original value of year column is a timestamp and formatted value is in a human readable datetime format
5. Verify that adding a time range filter on column year works correctly
6. Add the chart to a dashboard and add a VALUE native filter
7. Select a few values (years), apply the filter and verify that the chart was correctly filtered
8. Repeat those steps using a different dataset and a dttm column which doesn't have python_date_format set

ADDITIONAL INFORMATION

• Has associated issue:
• Required feature flags:
• Changes UI
• Includes DB Migration (follow approval process in SIP-59)
  • Migration is atomic, supports rollback & is backwards-compatible
  • Confirm DB migration upgrade and downgrade tested
  • Runtime estimates and downtime expectations provided
• Introduces new feature or API
• Removes existing feature or API

[villebro]

Two small style comments. Also, could we add a unit test that at least covers the happy path here?

[villebro]

Is there risk that labels will contain duplicated columns? If so, maybe we should put a set(...) inside the tuple(...) to dedupe them?

[kgabryje]

Good point, I don't know if there's such risk but can't hurt to make it bulletproof 👍

[villebro]

could we simplify to

Suggested change

	and kwargs.get("filters")
	and len(kwargs["filters"])
	and kwargs.get("filters", []))

[betodealmeida]

You mentioned during our call that you've added lines 145-146 to make the linter happy. A better way — more explicit — is to do:

            column = cast(BaseColumn, datasource.get_column(filter_col))

This informs the linter that column is always of type BaseColumn.

[kgabryje]

Isn’t it safer to check for None since .get_column can return None if the col name doesn't exist? I don't know if such scenario is possible though

[betodealmeida]

Ah, I thought datasource.get_column(filter_col) was guaranteed to return a truthy value. If that's not the case, then this is fine.

[kgabryje]

@betodealmeida @villebro Tests added!

[betodealmeida]

Thanks!

[betodealmeida]

Suggested change

	if datasource_model_instance and kwargs.get("filters", []):
	if datasource_model_instance and kwargs.get("filters"):

[betodealmeida]

Ah, I thought datasource.get_column(filter_col) was guaranteed to return a truthy value. If that's not the case, then this is fine.

[john-bodley]

Rather than the continue I think one if statement (based on truthiness and not falseness) is likely more readable. i.e.,

if (
    query_filter.get("op") != FilterOperator.TEMPORAL_RANGE
    and (filter_col := query_filter.get("col"))
    and isinstance(filter_col, str)
    and (column := datasource.get_column(filter_col))
 ):
     ...

[john-bodley]

@kgabryje BTW the reason for the comments post merge is at Airbnb we're started seeing a number of issues with filters returning the wrong temporal type, i.e., %Y-%m-%d columns are being retuned as epoch (in milliseconds) which is throwing errors. My current hypothesis is this PR might be related to the issue as it was part of 3.0.1 which we deployed internally a few days ago.

Additionally per the PR description it states,

If python_date_format is set for a column, normalize the column to a datetime format. When that column is filtered by a timestamp value, denormalize the timestamp to the original format.

however I'm struggling to find the logic the normalization (or denormalization) logic, i.e., I only see one transformation which seems to be stringifying epoch.

[john-bodley]

A lot of this logic and the logic defined on lines 156-164 was already defined here. Wouldn't it be better (and potentially less error prone) if we adhered to the DRY principle and reused the same helper function.

[john-bodley]

Why are we returning the filters where in actuality we're mutating the query_filters in place?

[john-bodley]

What is the reason for this change, i.e., why are we including a filter which was flagged for removal if it operation is not a temporal range?

[john-bodley]

If value is Any but not an int there's no guarantee that the return type will be a str. Am I missing something here?

[kgabryje]

Hey @john-bodley thanks for comments! I'm going to work with @michael-s-molina on reproducing the bug that you encountered and I'll address the rest of your comments when I work on the fix

[john-bodley]

If value is coming from the frontend and is a timestamp in milliseconds shouldn't epoch_s be str(value / 1000) and epoch_ms be str(value)?