Exports fail when certain columns are hidden and a field is remapped

[noahmoss]

Describe the bug
Exports fail under specific conditions. I can confirm that the steps below will repro this, but I'm not sure yet what exact set of conditions is necessary.

Logs

2022-04-28 17:05:46,683 ERROR middleware.catch-exceptions :: Error processing query: null
{:database_id 1,
 :started_at #t "2022-04-28T17:05:46.647614Z[UTC]",
 :json_query
 {:query {:source-table 2, :fields [["field" 9 nil] ["field" 3 nil] ["field" 5 nil]]},
  :type "query",
  :database 1,
  :middleware {:process-viz-settings? true, :skip-results-metadata? true, :format-rows? false},
  :async? true,
  :viz-settings
  {:table.pivot false,
   :metabase.shared.models.visualization-settings/table-columns
   [{:metabase.shared.models.visualization-settings/table-column-name "ID",
     :metabase.shared.models.visualization-settings/table-column-field-ref [:field 9 nil],
     :metabase.shared.models.visualization-settings/table-column-enabled true}
    {:metabase.shared.models.visualization-settings/table-column-name "USER_ID",
     :metabase.shared.models.visualization-settings/table-column-field-ref [:field 3 nil],
     :metabase.shared.models.visualization-settings/table-column-enabled true}
    {:metabase.shared.models.visualization-settings/table-column-name "PRODUCT_ID",
     :metabase.shared.models.visualization-settings/table-column-field-ref [:field 5 nil],
     :metabase.shared.models.visualization-settings/table-column-enabled true}
    {:metabase.shared.models.visualization-settings/table-column-name "SUBTOTAL",
     :metabase.shared.models.visualization-settings/table-column-field-ref [:field 6 nil],
     :metabase.shared.models.visualization-settings/table-column-enabled false}
    {:metabase.shared.models.visualization-settings/table-column-name "TAX",
     :metabase.shared.models.visualization-settings/table-column-field-ref [:field 8 nil],
     :metabase.shared.models.visualization-settings/table-column-enabled false}
    {:metabase.shared.models.visualization-settings/table-column-name "TOTAL",
     :metabase.shared.models.visualization-settings/table-column-field-ref [:field 7 nil],
     :metabase.shared.models.visualization-settings/table-column-enabled false}
    {:metabase.shared.models.visualization-settings/table-column-name "DISCOUNT",
     :metabase.shared.models.visualization-settings/table-column-field-ref [:field 1 nil],
     :metabase.shared.models.visualization-settings/table-column-enabled false}
    {:metabase.shared.models.visualization-settings/table-column-name "CREATED_AT",
     :metabase.shared.models.visualization-settings/table-column-field-ref [:field 4 {:temporal-unit :default}],
     :metabase.shared.models.visualization-settings/table-column-enabled false}
    {:metabase.shared.models.visualization-settings/table-column-name "QUANTITY",
     :metabase.shared.models.visualization-settings/table-column-field-ref [:field 2 nil],
     :metabase.shared.models.visualization-settings/table-column-enabled false}],
   :metabase.shared.models.visualization-settings/column-settings {},
   :table.cell_column "PRODUCT_ID",
   :card.title nil,
   :table.column_formatting [],
   :table.pivot_column "USER_ID"}},
 :native nil,
 :status :failed,
 :class java.lang.Exception,
 :stacktrace
 ["--> query_processor.store$default_field.invokeStatic(store.clj:255)"
  "query_processor.store$default_field.invoke(store.clj:251)"
  "query_processor.store$eval52221$field__52226$fn__52227.invoke(store.clj:264)"
  "query_processor.store$eval52221$field__52226.invoke(store.clj:261)"
  "query_processor.middleware.validate_temporal_bucketing$validate_temporal_bucketing.invokeStatic(validate_temporal_bucketing.clj:35)"
  "query_processor.middleware.validate_temporal_bucketing$validate_temporal_bucketing.invoke(validate_temporal_bucketing.clj:29)"
  "query_processor$preprocess_STAR_$fn__67153.invoke(query_processor.clj:124)"
  "query_processor$preprocess_STAR_.invokeStatic(query_processor.clj:122)"
  "query_processor$preprocess_STAR_.invoke(query_processor.clj:117)"
  "query_processor$fn__67161$combined_pre_process__67162$combined_pre_process_STAR___67163.invoke(query_processor.clj:204)"
  "query_processor.middleware.resolve_database_and_driver$resolve_database_and_driver$fn__65135$fn__65140.invoke(resolve_database_and_driver.clj:35)"
  "driver$do_with_driver.invokeStatic(driver.clj:75)"
  "driver$do_with_driver.invoke(driver.clj:71)"
  "query_processor.middleware.resolve_database_and_driver$resolve_database_and_driver$fn__65135.invoke(resolve_database_and_driver.clj:34)"
  "query_processor.middleware.fetch_source_query$resolve_card_id_source_tables$fn__63725.invoke(fetch_source_query.clj:281)"
  "query_processor.middleware.store$initialize_store$fn__65969$fn__65970.invoke(store.clj:11)"
  "query_processor.store$do_with_store.invokeStatic(store.clj:44)"
  "query_processor.store$do_with_store.invoke(store.clj:38)"
  "query_processor.middleware.store$initialize_store$fn__65969.invoke(store.clj:10)"
  "query_processor.middleware.normalize_query$normalize$fn__63937.invoke(normalize_query.clj:22)"
  "metabase_enterprise.audit_app.query_processor.middleware.handle_audit_queries$handle_internal_queries$fn__66592.invoke(handle_audit_queries.clj:130)"
  "query_processor.middleware.constraints$add_default_userland_constraints$fn__53418.invoke(constraints.clj:52)"
  "query_processor.middleware.process_userland_query$process_userland_query$fn__65008.invoke(process_userland_query.clj:145)"
  "query_processor.middleware.catch_exceptions$catch_exceptions$fn__62875.invoke(catch_exceptions.clj:162)"
  "query_processor.reducible$async_qp$qp_STAR___55028$thunk__55030.invoke(reducible.clj:100)"
  "query_processor.reducible$async_qp$qp_STAR___55028$fn__55032.invoke(reducible.clj:105)"],
 :card_id nil,
 :context :csv-download,
 :error "Error: Field 4 is not present in the Query Processor Store.",
 :row_count 0,
 :running_time 0,
 :preprocessed nil,
 :data {:rows [], :cols []}}

To Reproduce
Steps to reproduce the behavior:

1. In the data model for the Orders table of the Sample DB, remap the User ID field to use the Name field on the foreign key
2. Go to Browse Data -> Sample Database -> Orders
3. Click Visualization -> Table, and hide all columns except for ID, User ID and Product ID
4. Without saving the question, download the results in any format. You should get an invalid file and the above error in the logs

Information about your Metabase Installation:

Current master

Severity
P1?

[noahmoss]

Ran a bisect and discovered #21338 appears to be the PR that introduced this...FYI @WiNloSt

I'll keep looking into it to see if I can narrow down the exact cause.

[noahmoss]

It will repro if you just hide the "Created At" field, it seems

[WiNloSt]

@flamber Maybe you could help see what I did wrong here. I tested the commit 5197df5 (#21338) with the repro steps @noahmoss described, but I could still download the correct CSV, I tested that the current master would produce incorrect CSV though.

I'll take more time to look into this tomorrow morning.

[noahmoss]

@WiNloSt Sorry, I just tested it again, and I think the remapped field is actually important. Not sure why I thought it wasn't. I'll revert the original issue description.