fix(filter-indicator): show filters handled by jinja as applied

[villebro]

SUMMARY

Currently only legacy charts support identifying filters applied via jinja templates. However, this only supports the filter_values function, and is done using regex on the virtual table query. Therefore, filters applied via the get_filters function don't work on either v1 or legacy charts, as they aren't caught by the regex. In addition, this approach has the unlikely downside of false positives, if the regex would happen to match a parameter in the regular query, as it didn't check for opening/closing curly braces.

This PR replaces the regex-based identification method with a similar approach to how removed filters are currently identified. The applied filters are added to a list during template rendering, which is made available to the code that is using the template processor. This way the collection of applied filters is automated and happens during template evaluation.

The changes proposed here are backwards compatible, and will display filters correctly when the cache is updated (for reviewers remember to force refresh the chart if caching is enabled).

BEFORE

On a dashboard with a virtual dataset referencing the filter_values jinja function, we can see that the ECharts plugin which uses the v1 chart data endpoint is unable to identify the applied filter value correctly due to the column referenced not being available in the dataset. Notice, however, the NVD3 line chart on the right which identifies the filter correctly due to the regex in viz.py:

AFTER

Now applied filters are identified correctly both on legacy and v1 charts:

When applying a filter whose column isn't 1) available in the dataset 2) referenced in the jinja code, the filter is also correctly identified as being incompatible:

TESTING INSTRUCTIONS

1. Enable the ENABLE_TEMPLATE_PROCESSING feature flag.
2. Create a virtual dataset with the following query: select *, '{{ filter_values('gender', 'unset')[0] }}' as series from random_time_series
3. Create a legacy and v1 chart referencing the new virtual dataset
4. Create a dashboard with both charts and add a native filter referencing the "gender" column on the "birth_names" dataset
5. Apply a filter, e.g. "boy"
6. See the indicators work on both the new and old chart

ADDITIONAL INFORMATION

• Has associated issue: closes Jinja Incompatible Filter Warnings #15209
• Required feature flags:
• Changes UI
• Includes DB Migration (follow approval process in SIP-59)
  • Migration is atomic, supports rollback & is backwards-compatible
  • Confirm DB migration upgrade and downgrade tested
  • Runtime estimates and downtime expectations provided
• Introduces new feature or API
• Removes existing feature or API

[codecov]

Codecov Report

Merging #17140 (8296df8) into master (37944e1) will decrease coverage by 0.01%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master   #17140      +/-   ##
==========================================
- Coverage   76.89%   76.87%   -0.02%     
==========================================
  Files        1038     1038              
  Lines       55515    55528      +13     
  Branches     7564     7564              
==========================================
- Hits        42690    42689       -1     
- Misses      12575    12589      +14     
  Partials      250      250              

Flag	Coverage Δ
hive	81.45% <100.00%> (+0.01%)	⬆️
mysql	?
postgres	81.91% <100.00%> (+<0.01%)	⬆️
presto	81.79% <100.00%> (+0.01%)	⬆️
python	82.37% <100.00%> (-0.05%)	⬇️
sqlite	81.59% <100.00%> (+0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
superset/common/query_context.py	90.90% <ø> (ø)
superset/common/query_actions.py	92.75% <100.00%> (+0.10%)	⬆️
superset/common/utils.py	89.28% <100.00%> (+0.39%)	⬆️
superset/connectors/sqla/models.py	87.32% <100.00%> (+0.05%)	⬆️
superset/jinja_context.py	89.84% <100.00%> (+0.21%)	⬆️
superset/models/helpers.py	90.58% <100.00%> (+0.03%)	⬆️
superset/viz.py	57.99% <100.00%> (+0.05%)	⬆️
superset/db_engine_specs/mysql.py	94.04% <0.00%> (-3.58%)	⬇️
superset/reports/commands/log_prune.py	85.71% <0.00%> (-3.58%)	⬇️
superset/commands/importers/v1/utils.py	89.13% <0.00%> (-2.18%)	⬇️
... and 3 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 37944e1...8296df8. Read the comment docs.

[villebro]

/testenv up FEATURE_DASHBOARD_NATIVE_FILTERS=true FEATURE_DASHBOARD_CROSS_FILTERS=true FEATURE_ENABLE_TEMPLATE_PROCESSING=true

[github-actions]

@villebro Ephemeral environment spinning up at http://34.221.208.55:8080. Credentials are admin/admin. Please allow several minutes for bootstrapping and startup.

[villebro]

FYI I added a test dashboard to the eph env

[zhaoyongjie]

LGTM. this approach is better than before.

[github-actions]

Ephemeral environment shutdown and build artifacts deleted.

[xxcmp1977]

I'm using from_dttm and to_dttm in my dataset, standard date filter box.
Still can see 'Incompatible filters' yellow warning.

Should it work for from_dttm and to_dttm as well ?

[igor-lukyanov]

@villebro could you please help to figure out how to deal w/ 'Incompatible filters' message regarding time columns?
I.e., {{ from_dttm }} and {{ to_dttm }}. Seems the fix does not cover them, does it? Thank you.

[alfredomagallon]

Hello, I am having the same issue with the yellow warning showing when I use {{ from_dttm }} and {{ to_dttm }} in the query.
Any workaround?
Thanks!