fix: take form_data reference for metrics for pivot_v2 table reports

[hughhhh]

SUMMARY

When executing pivot v2 table report if the verbose_map doesn't match up with the df columns we are throwing this error for metric/values. To fix this for pivot v2 table we won't be using the verbose map to change the values/metrics names to have better alignment.

Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1516, in full_dispatch_request
rv = self.dispatch_request()
File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1502, in dispatch_request
return self.ensure_sync(self.view_functions[rule.endpoint])(*req.view_args)
File "/usr/local/lib/python3.8/site-packages/flask_appbuilder/security/decorators.py", line 89, in wraps
return f(self, *args, *kwargs)
File "/usr/local/lib/python3.8/site-packages/superset/views/base_api.py", line 113, in wraps
raise ex
File "/usr/local/lib/python3.8/site-packages/superset/views/base_api.py", line 110, in wraps
duration, response = time_function(f, self, args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/superset/utils/core.py", line 1533, in time_function
response = func(*args, *kwargs)
File "/usr/local/lib/python3.8/site-packages/superset/utils/log.py", line 244, in wrapper
value = f(args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/superset/charts/data/api.py", line 160, in get_data
return self._get_data_response(
File "/usr/local/lib/python3.8/site-packages/superset/charts/data/api.py", line 396, in _get_data_response
return self._send_chart_response(result, form_data, datasource)
File "/usr/local/lib/python3.8/site-packages/superset/charts/data/api.py", line 343, in _send_chart_response
result = apply_post_process(result, form_data, datasource)
File "/usr/local/lib/python3.8/site-packages/superset/charts/post_processing.py", line 339, in apply_post_process
processed_df = post_processor(df, form_data, datasource)
File "/usr/local/lib/python3.8/site-packages/superset/charts/post_processing.py", line 234, in pivot_table_v2
return pivot_df(
File "/usr/local/lib/python3.8/site-packages/superset/charts/post_processing.py", line 89, in pivot_df
df = df.pivot_table(
File "/usr/local/lib/python3.8/site-packages/pandas/core/frame.py", line 7951, in pivot_table
return pivot_table(
File "/usr/local/lib/python3.8/site-packages/pandas/core/reshape/pivot.py", line 95, in pivot_table
table = __internal_pivot_table(
File "/usr/local/lib/python3.8/site-packages/pandas/core/reshape/pivot.py", line 141, in __internal_pivot_table
raise KeyError(i)
KeyError: 'COUNT()'

Click here to view exception in pandas

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

TESTING INSTRUCTIONS

ADDITIONAL INFORMATION

• Has associated issue:
• Required feature flags:
• Changes UI
• Includes DB Migration (follow approval process in SIP-59)
  • Migration is atomic, supports rollback & is backwards-compatible
  • Confirm DB migration upgrade and downgrade tested
  • Runtime estimates and downtime expectations provided
• Introduces new feature or API
• Removes existing feature or API

[codecov]

Codecov Report

Merging #21029 (26fa175) into master (6650076) will increase coverage by 0.09%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master   #21029      +/-   ##
==========================================
+ Coverage   66.35%   66.45%   +0.09%     
==========================================
  Files        1767     1767              
  Lines       67356    67553     +197     
  Branches     7147     7147              
==========================================
+ Hits        44694    44892     +198     
+ Misses      20834    20833       -1     
  Partials     1828     1828              

Flag	Coverage Δ
hive	53.35% <0.00%> (+0.19%)	⬆️
javascript	52.11% <ø> (ø)
mysql	81.07% <100.00%> (+0.11%)	⬆️
postgres	81.12% <100.00%> (+0.11%)	⬆️
presto	53.25% <0.00%> (+0.19%)	⬆️
python	81.60% <100.00%> (+0.11%)	⬆️
sqlite	79.73% <100.00%> (+0.12%)	⬆️
unit	50.89% <100.00%> (+0.41%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
superset/charts/post_processing.py	89.70% <100.00%> (+0.15%)	⬆️
superset/common/query_context_processor.py	88.31% <0.00%> (-0.98%)	⬇️
superset/explore/utils.py	100.00% <0.00%> (ø)
superset/dao/base.py	96.82% <0.00%> (+1.42%)	⬆️
superset/config.py	93.19% <0.00%> (+1.72%)	⬆️
superset/explore/permalink/api.py	94.82% <0.00%> (+1.72%)	⬆️
superset/explore/form_data/api.py	97.84% <0.00%> (+2.15%)	⬆️
superset/views/sql_lab.py	63.81% <0.00%> (+3.28%)	⬆️

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

[betodealmeida]

Can you add a test to https://github.com/apache/superset/blob/master/tests/unit_tests/charts/test_post_processing.py? We have some cool unit tests for the post processing. :)

[betodealmeida]

Awesome!

[betodealmeida]

Nit: I would make this assertion more generic, since it might be useful to capture future regressions in other parts of the code. Something like:

assert apply_post_process(result, form_data, datasource=sqla_table) = {
    "queries": [
        {
            "data": { "COUNT(*)": ...,  ... },
        },
    ]
)