Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Custom Expression case is using wrong field reference when nested query #17512

Closed
flamber opened this issue Aug 19, 2021 · 6 comments · Fixed by #19008
Closed

Custom Expression case is using wrong field reference when nested query #17512

flamber opened this issue Aug 19, 2021 · 6 comments · Fixed by #19008
Assignees
Labels
Priority:P2 Average run of the mill bug Querying/Nested Queries Questions based on other saved questions Querying/Notebook/Custom Expression .Regression Bugs that were previously fixed and/or bugs unintentionally shipped with new features. .Reproduced Issues reproduced in test (usually Cypress) Type:Bug Product defects
Milestone

Comments

@flamber
Copy link
Contributor

flamber commented Aug 19, 2021

Describe the bug
Custom Expression case is using wrong field reference when nested query

Regression since 0.40.0

To Reproduce

  1. Custom question > Sample Dataset > Orders
  2. Custom Expression Distinct(case([Discount] > 0, [Subtotal], [Total])) as "CE", grouped by CreatedAt:Month
  3. Custom Column 1 + 1 as "CC" (to trigger nested query)
    image
  4. Errors with Column "source.SUBTOTAL" not found;, because it's incorrectly referencing the field instead of the expression.
Generated SQL query
SELECT
  "source"."CREATED_AT" AS "CREATED_AT",
  "source"."SUBTOTAL" AS "SUBTOTAL",
  "source"."CC" AS "CC" 
FROM
  (
    SELECT
      "source"."CREATED_AT" AS "CREATED_AT",
      "source"."SUBTOTAL" AS "SUBTOTAL",
      (
        1 + 1
      )
      AS "CC" 
    FROM
      (
        SELECT
          parsedatetime(formatdatetime("PUBLIC"."ORDERS"."CREATED_AT", 'yyyyMM'), 'yyyyMM') AS "CREATED_AT",
          count(distinct 
          CASE
            WHEN
              "PUBLIC"."ORDERS"."DISCOUNT" > 0 
            THEN
              "PUBLIC"."ORDERS"."SUBTOTAL" 
            ELSE
              "PUBLIC"."ORDERS"."TOTAL" 
          END
) AS "CE bad case field" 
        FROM
          "PUBLIC"."ORDERS" 
        GROUP BY
          parsedatetime(formatdatetime("PUBLIC"."ORDERS"."CREATED_AT", 'yyyyMM'), 'yyyyMM') 
        ORDER BY
          parsedatetime(formatdatetime("PUBLIC"."ORDERS"."CREATED_AT", 'yyyyMM'), 'yyyyMM') ASC
      )
      "source"
  )
  "source"
Full stacktrace
2021-08-19 13:22:37,295 ERROR middleware.catch-exceptions :: Error processing query: null
{:database_id 4,
 :started_at #t "2021-08-19T13:22:35.848511+02:00[Europe/Copenhagen]",
 :via
 [{:status :failed,
   :class clojure.lang.ExceptionInfo,
   :error "Error executing query",
   :stacktrace
   ["--> driver.sql_jdbc.execute$execute_reducible_query$fn__80797.invoke(execute.clj:480)"
    "driver.sql_jdbc.execute$execute_reducible_query.invokeStatic(execute.clj:477)"
    "driver.sql_jdbc.execute$execute_reducible_query.invoke(execute.clj:463)"
    "driver.sql_jdbc.execute$execute_reducible_query.invokeStatic(execute.clj:472)"
    "driver.sql_jdbc.execute$execute_reducible_query.invoke(execute.clj:463)"
    "driver.sql_jdbc$fn__82286.invokeStatic(sql_jdbc.clj:54)"
    "driver.sql_jdbc$fn__82286.invoke(sql_jdbc.clj:52)"
    "driver.h2$fn__81038.invokeStatic(h2.clj:83)"
    "driver.h2$fn__81038.invoke(h2.clj:80)"
    "query_processor.context$executef.invokeStatic(context.clj:59)"
    "query_processor.context$executef.invoke(context.clj:48)"
    "query_processor.context.default$default_runf.invokeStatic(default.clj:68)"
    "query_processor.context.default$default_runf.invoke(default.clj:66)"
    "query_processor.context$runf.invokeStatic(context.clj:45)"
    "query_processor.context$runf.invoke(context.clj:39)"
    "query_processor.reducible$pivot.invokeStatic(reducible.clj:34)"
    "query_processor.reducible$pivot.invoke(reducible.clj:31)"
    "query_processor.middleware.mbql_to_native$mbql__GT_native$fn__47894.invoke(mbql_to_native.clj:25)"
    "query_processor.middleware.check_features$check_features$fn__47008.invoke(check_features.clj:39)"
    "query_processor.middleware.limit$limit$fn__47880.invoke(limit.clj:37)"
    "query_processor.middleware.cache$maybe_return_cached_results$fn__46460.invoke(cache.clj:204)"
    "query_processor.middleware.optimize_temporal_filters$optimize_temporal_filters$fn__48140.invoke(optimize_temporal_filters.clj:204)"
    "query_processor.middleware.validate_temporal_bucketing$validate_temporal_bucketing$fn__50072.invoke(validate_temporal_bucketing.clj:50)"
    "query_processor.middleware.auto_parse_filter_values$auto_parse_filter_values$fn__45579.invoke(auto_parse_filter_values.clj:43)"
    "query_processor.middleware.wrap_value_literals$wrap_value_literals$fn__41713.invoke(wrap_value_literals.clj:161)"
    "query_processor.middleware.annotate$add_column_info$fn__41588.invoke(annotate.clj:608)"
    "query_processor.middleware.permissions$check_query_permissions$fn__46880.invoke(permissions.clj:81)"
    "query_processor.middleware.pre_alias_aggregations$pre_alias_aggregations$fn__49001.invoke(pre_alias_aggregations.clj:40)"
    "query_processor.middleware.cumulative_aggregations$handle_cumulative_aggregations$fn__47081.invoke(cumulative_aggregations.clj:60)"
    "query_processor.middleware.resolve_joined_fields$resolve_joined_fields$fn__49298.invoke(resolve_joined_fields.clj:102)"
    "query_processor.middleware.resolve_joins$resolve_joins$fn__49611.invoke(resolve_joins.clj:171)"
    "query_processor.middleware.add_implicit_joins$add_implicit_joins$fn__45155.invoke(add_implicit_joins.clj:190)"
    "query_processor.middleware.large_int_id$convert_id_to_string$fn__47844.invoke(large_int_id.clj:59)"
    "query_processor.middleware.format_rows$format_rows$fn__47825.invoke(format_rows.clj:74)"
    "query_processor.middleware.add_default_temporal_unit$add_default_temporal_unit$fn__44449.invoke(add_default_temporal_unit.clj:23)"
    "query_processor.middleware.desugar$desugar$fn__47147.invoke(desugar.clj:21)"
    "query_processor.middleware.binning$update_binning_strategy$fn__45966.invoke(binning.clj:229)"
    "query_processor.middleware.resolve_fields$resolve_fields$fn__46683.invoke(resolve_fields.clj:34)"
    "query_processor.middleware.add_dimension_projections$add_remapping$fn__44804.invoke(add_dimension_projections.clj:314)"
    "query_processor.middleware.add_implicit_clauses$add_implicit_clauses$fn__45033.invoke(add_implicit_clauses.clj:147)"
    "query_processor.middleware.upgrade_field_literals$upgrade_field_literals$fn__50021.invoke(upgrade_field_literals.clj:40)"
    "query_processor.middleware.add_source_metadata$add_source_metadata_for_source_queries$fn__45318.invoke(add_source_metadata.clj:123)"
    "query_processor.middleware.reconcile_breakout_and_order_by_bucketing$reconcile_breakout_and_order_by_bucketing$fn__49173.invoke(reconcile_breakout_and_order_by_bucketing.clj:100)"
    "query_processor.middleware.auto_bucket_datetimes$auto_bucket_datetimes$fn__45526.invoke(auto_bucket_datetimes.clj:147)"
    "query_processor.middleware.resolve_source_table$resolve_source_tables$fn__46730.invoke(resolve_source_table.clj:45)"
    "query_processor.middleware.parameters$substitute_parameters$fn__48983.invoke(parameters.clj:111)"
    "query_processor.middleware.resolve_referenced$resolve_referenced_card_resources$fn__46782.invoke(resolve_referenced.clj:79)"
    "query_processor.middleware.expand_macros$expand_macros$fn__47531.invoke(expand_macros.clj:184)"
    "query_processor.middleware.add_timezone_info$add_timezone_info$fn__45327.invoke(add_timezone_info.clj:15)"
    "query_processor.middleware.splice_params_in_response$splice_params_in_response$fn__49974.invoke(splice_params_in_response.clj:32)"
    "query_processor.middleware.resolve_database_and_driver$resolve_database_and_driver$fn__49184$fn__49188.invoke(resolve_database_and_driver.clj:31)"
    "driver$do_with_driver.invokeStatic(driver.clj:60)"
    "driver$do_with_driver.invoke(driver.clj:56)"
    "query_processor.middleware.resolve_database_and_driver$resolve_database_and_driver$fn__49184.invoke(resolve_database_and_driver.clj:25)"
    "query_processor.middleware.fetch_source_query$resolve_card_id_source_tables$fn__47771.invoke(fetch_source_query.clj:274)"
    "query_processor.middleware.store$initialize_store$fn__49983$fn__49984.invoke(store.clj:11)"
    "query_processor.store$do_with_store.invokeStatic(store.clj:44)"
    "query_processor.store$do_with_store.invoke(store.clj:38)"
    "query_processor.middleware.store$initialize_store$fn__49983.invoke(store.clj:10)"
    "query_processor.middleware.validate$validate_query$fn__50028.invoke(validate.clj:10)"
    "query_processor.middleware.normalize_query$normalize$fn__47907.invoke(normalize_query.clj:22)"
    "query_processor.middleware.add_rows_truncated$add_rows_truncated$fn__45173.invoke(add_rows_truncated.clj:35)"
    "query_processor.middleware.results_metadata$record_and_return_metadata_BANG_$fn__49959.invoke(results_metadata.clj:147)"
    "query_processor.middleware.constraints$add_default_userland_constraints$fn__47024.invoke(constraints.clj:42)"
    "query_processor.middleware.process_userland_query$process_userland_query$fn__49070.invoke(process_userland_query.clj:134)"
    "query_processor.middleware.catch_exceptions$catch_exceptions$fn__46964.invoke(catch_exceptions.clj:173)"
    "query_processor.reducible$async_qp$qp_STAR___38252$thunk__38253.invoke(reducible.clj:103)"
    "query_processor.reducible$async_qp$qp_STAR___38252.invoke(reducible.clj:109)"
    "query_processor.reducible$sync_qp$qp_STAR___38261$fn__38264.invoke(reducible.clj:135)"
    "query_processor.reducible$sync_qp$qp_STAR___38261.invoke(reducible.clj:134)"
    "query_processor$process_userland_query.invokeStatic(query_processor.clj:241)"
    "query_processor$process_userland_query.doInvoke(query_processor.clj:237)"
    "query_processor$fn__50118$process_query_and_save_execution_BANG___50127$fn__50130.invoke(query_processor.clj:253)"
    "query_processor$fn__50118$process_query_and_save_execution_BANG___50127.invoke(query_processor.clj:245)"
    "query_processor$fn__50162$process_query_and_save_with_max_results_constraints_BANG___50171$fn__50174.invoke(query_processor.clj:265)"
    "query_processor$fn__50162$process_query_and_save_with_max_results_constraints_BANG___50171.invoke(query_processor.clj:258)"
    "api.dataset$run_query_async$fn__56409.invoke(dataset.clj:56)"
    "query_processor.streaming$streaming_response_STAR_$fn__56388$fn__56389.invoke(streaming.clj:72)"
    "query_processor.streaming$streaming_response_STAR_$fn__56388.invoke(streaming.clj:71)"
    "async.streaming_response$do_f_STAR_.invokeStatic(streaming_response.clj:65)"
    "async.streaming_response$do_f_STAR_.invoke(streaming_response.clj:63)"
    "async.streaming_response$do_f_async$fn__16088.invoke(streaming_response.clj:84)"],
   :error_type :invalid-query,
   :ex-data
   {:sql
    "-- Metabase:: userID: 1 queryType: MBQL queryHash: a7bb65e395450d8a448b9dc885670906a5ff4e41c59acc322372c4106c39d069\nSELECT \"source\".\"CREATED_AT\" AS \"CREATED_AT\", \"source\".\"SUBTOTAL\" AS \"SUBTOTAL\", \"source\".\"CC\" AS \"CC\" FROM (SELECT \"source\".\"CREATED_AT\" AS \"CREATED_AT\", \"source\".\"SUBTOTAL\" AS \"SUBTOTAL\", (1 + 1) AS \"CC\" FROM (SELECT parsedatetime(formatdatetime(\"PUBLIC\".\"ORDERS\".\"CREATED_AT\", 'yyyyMM'), 'yyyyMM') AS \"CREATED_AT\", count(distinct CASE WHEN \"PUBLIC\".\"ORDERS\".\"DISCOUNT\" > 0 THEN \"PUBLIC\".\"ORDERS\".\"SUBTOTAL\" ELSE \"PUBLIC\".\"ORDERS\".\"TOTAL\" END) AS \"CE bad case field\" FROM \"PUBLIC\".\"ORDERS\" GROUP BY parsedatetime(formatdatetime(\"PUBLIC\".\"ORDERS\".\"CREATED_AT\", 'yyyyMM'), 'yyyyMM') ORDER BY parsedatetime(formatdatetime(\"PUBLIC\".\"ORDERS\".\"CREATED_AT\", 'yyyyMM'), 'yyyyMM') ASC) \"source\") \"source\" LIMIT 2000",
    :params nil,
    :type :invalid-query}}],
 :state "42S22",
 :error_type :invalid-query,
 :json_query
 {:database 4,
  :query
  {:source-query
   {:source-table 11,
    :aggregation
    [["aggregation-options"
      ["distinct" ["case" [[[">" ["field" 81 nil] 0] ["field" 84 nil]]] {:default ["field" 88 nil]}]]
      {:name "CE bad case field", :display-name "CE bad case field"}]],
    :breakout [["field" 86 {:temporal-unit "month"}]]},
   :expressions {:CC ["+" 1 1]}},
  :type "query",
  :parameters [],
  :middleware {:js-int-to-string? true, :add-default-userland-constraints? true}},
 :native
 {:query
  "SELECT \"source\".\"CREATED_AT\" AS \"CREATED_AT\", \"source\".\"SUBTOTAL\" AS \"SUBTOTAL\", \"source\".\"CC\" AS \"CC\" FROM (SELECT \"source\".\"CREATED_AT\" AS \"CREATED_AT\", \"source\".\"SUBTOTAL\" AS \"SUBTOTAL\", (1 + 1) AS \"CC\" FROM (SELECT parsedatetime(formatdatetime(\"PUBLIC\".\"ORDERS\".\"CREATED_AT\", 'yyyyMM'), 'yyyyMM') AS \"CREATED_AT\", count(distinct CASE WHEN \"PUBLIC\".\"ORDERS\".\"DISCOUNT\" > 0 THEN \"PUBLIC\".\"ORDERS\".\"SUBTOTAL\" ELSE \"PUBLIC\".\"ORDERS\".\"TOTAL\" END) AS \"CE bad case field\" FROM \"PUBLIC\".\"ORDERS\" GROUP BY parsedatetime(formatdatetime(\"PUBLIC\".\"ORDERS\".\"CREATED_AT\", 'yyyyMM'), 'yyyyMM') ORDER BY parsedatetime(formatdatetime(\"PUBLIC\".\"ORDERS\".\"CREATED_AT\", 'yyyyMM'), 'yyyyMM') ASC) \"source\") \"source\" LIMIT 2000",
  :params nil},
 :status :failed,
 :class org.h2.jdbc.JdbcSQLException,
 :stacktrace
 ["org.h2.message.DbException.getJdbcSQLException(DbException.java:357)"
  "org.h2.message.DbException.get(DbException.java:179)"
  "org.h2.message.DbException.get(DbException.java:155)"
  "org.h2.expression.ExpressionColumn.optimize(ExpressionColumn.java:150)"
  "org.h2.expression.Alias.optimize(Alias.java:51)"
  "org.h2.command.dml.Select.prepare(Select.java:858)"
  "org.h2.command.Parser.prepare(Parser.java:263)"
  "org.h2.engine.Session.prepare(Session.java:576)"
  "org.h2.table.TableView.compileViewQuery(TableView.java:120)"
  "org.h2.table.TableView.initColumnsAndTables(TableView.java:172)"
  "org.h2.table.TableView.init(TableView.java:113)"
  "org.h2.table.TableView.<init>(TableView.java:70)"
  "org.h2.table.TableView.createTempView(TableView.java:559)"
  "org.h2.command.Parser.readTableFilter(Parser.java:1367)"
  "org.h2.command.Parser.parseSelectSimpleFromPart(Parser.java:2138)"
  "org.h2.command.Parser.parseSelectSimple(Parser.java:2287)"
  "org.h2.command.Parser.parseSelectSub(Parser.java:2133)"
  "org.h2.command.Parser.parseSelectUnion(Parser.java:1946)"
  "org.h2.command.Parser.parseSelect(Parser.java:1919)"
  "org.h2.command.Parser.parsePrepared(Parser.java:463)"
  "org.h2.command.Parser.parse(Parser.java:335)"
  "org.h2.command.Parser.parse(Parser.java:307)"
  "org.h2.command.Parser.prepareCommand(Parser.java:278)"
  "org.h2.engine.Session.prepareLocal(Session.java:611)"
  "org.h2.engine.Session.prepareCommand(Session.java:549)"
  "org.h2.jdbc.JdbcConnection.prepareCommand(JdbcConnection.java:1247)"
  "org.h2.jdbc.JdbcStatement.executeInternal(JdbcStatement.java:217)"
  "org.h2.jdbc.JdbcStatement.execute(JdbcStatement.java:205)"
  "com.mchange.v2.c3p0.impl.NewProxyStatement.execute(NewProxyStatement.java:75)"
  "--> driver.sql_jdbc.execute$fn__80717.invokeStatic(execute.clj:344)"
  "driver.sql_jdbc.execute$fn__80717.invoke(execute.clj:342)"
  "driver.sql_jdbc.execute$execute_statement_or_prepared_statement_BANG_.invokeStatic(execute.clj:352)"
  "driver.sql_jdbc.execute$execute_statement_or_prepared_statement_BANG_.invoke(execute.clj:349)"
  "driver.sql_jdbc.execute$execute_reducible_query$fn__80797.invoke(execute.clj:478)"
  "driver.sql_jdbc.execute$execute_reducible_query.invokeStatic(execute.clj:477)"
  "driver.sql_jdbc.execute$execute_reducible_query.invoke(execute.clj:463)"
  "driver.sql_jdbc.execute$execute_reducible_query.invokeStatic(execute.clj:472)"
  "driver.sql_jdbc.execute$execute_reducible_query.invoke(execute.clj:463)"
  "driver.sql_jdbc$fn__82286.invokeStatic(sql_jdbc.clj:54)"
  "driver.sql_jdbc$fn__82286.invoke(sql_jdbc.clj:52)"
  "driver.h2$fn__81038.invokeStatic(h2.clj:83)"
  "driver.h2$fn__81038.invoke(h2.clj:80)"
  "query_processor.context$executef.invokeStatic(context.clj:59)"
  "query_processor.context$executef.invoke(context.clj:48)"
  "query_processor.context.default$default_runf.invokeStatic(default.clj:68)"
  "query_processor.context.default$default_runf.invoke(default.clj:66)"
  "query_processor.context$runf.invokeStatic(context.clj:45)"
  "query_processor.context$runf.invoke(context.clj:39)"
  "query_processor.reducible$pivot.invokeStatic(reducible.clj:34)"
  "query_processor.reducible$pivot.invoke(reducible.clj:31)"
  "query_processor.middleware.mbql_to_native$mbql__GT_native$fn__47894.invoke(mbql_to_native.clj:25)"
  "query_processor.middleware.check_features$check_features$fn__47008.invoke(check_features.clj:39)"
  "query_processor.middleware.limit$limit$fn__47880.invoke(limit.clj:37)"
  "query_processor.middleware.cache$maybe_return_cached_results$fn__46460.invoke(cache.clj:204)"
  "query_processor.middleware.optimize_temporal_filters$optimize_temporal_filters$fn__48140.invoke(optimize_temporal_filters.clj:204)"
  "query_processor.middleware.validate_temporal_bucketing$validate_temporal_bucketing$fn__50072.invoke(validate_temporal_bucketing.clj:50)"
  "query_processor.middleware.auto_parse_filter_values$auto_parse_filter_values$fn__45579.invoke(auto_parse_filter_values.clj:43)"
  "query_processor.middleware.wrap_value_literals$wrap_value_literals$fn__41713.invoke(wrap_value_literals.clj:161)"
  "query_processor.middleware.annotate$add_column_info$fn__41588.invoke(annotate.clj:608)"
  "query_processor.middleware.permissions$check_query_permissions$fn__46880.invoke(permissions.clj:81)"
  "query_processor.middleware.pre_alias_aggregations$pre_alias_aggregations$fn__49001.invoke(pre_alias_aggregations.clj:40)"
  "query_processor.middleware.cumulative_aggregations$handle_cumulative_aggregations$fn__47081.invoke(cumulative_aggregations.clj:60)"
  "query_processor.middleware.resolve_joined_fields$resolve_joined_fields$fn__49298.invoke(resolve_joined_fields.clj:102)"
  "query_processor.middleware.resolve_joins$resolve_joins$fn__49611.invoke(resolve_joins.clj:171)"
  "query_processor.middleware.add_implicit_joins$add_implicit_joins$fn__45155.invoke(add_implicit_joins.clj:190)"
  "query_processor.middleware.large_int_id$convert_id_to_string$fn__47844.invoke(large_int_id.clj:59)"
  "query_processor.middleware.format_rows$format_rows$fn__47825.invoke(format_rows.clj:74)"
  "query_processor.middleware.add_default_temporal_unit$add_default_temporal_unit$fn__44449.invoke(add_default_temporal_unit.clj:23)"
  "query_processor.middleware.desugar$desugar$fn__47147.invoke(desugar.clj:21)"
  "query_processor.middleware.binning$update_binning_strategy$fn__45966.invoke(binning.clj:229)"
  "query_processor.middleware.resolve_fields$resolve_fields$fn__46683.invoke(resolve_fields.clj:34)"
  "query_processor.middleware.add_dimension_projections$add_remapping$fn__44804.invoke(add_dimension_projections.clj:314)"
  "query_processor.middleware.add_implicit_clauses$add_implicit_clauses$fn__45033.invoke(add_implicit_clauses.clj:147)"
  "query_processor.middleware.upgrade_field_literals$upgrade_field_literals$fn__50021.invoke(upgrade_field_literals.clj:40)"
  "query_processor.middleware.add_source_metadata$add_source_metadata_for_source_queries$fn__45318.invoke(add_source_metadata.clj:123)"
  "query_processor.middleware.reconcile_breakout_and_order_by_bucketing$reconcile_breakout_and_order_by_bucketing$fn__49173.invoke(reconcile_breakout_and_order_by_bucketing.clj:100)"
  "query_processor.middleware.auto_bucket_datetimes$auto_bucket_datetimes$fn__45526.invoke(auto_bucket_datetimes.clj:147)"
  "query_processor.middleware.resolve_source_table$resolve_source_tables$fn__46730.invoke(resolve_source_table.clj:45)"
  "query_processor.middleware.parameters$substitute_parameters$fn__48983.invoke(parameters.clj:111)"
  "query_processor.middleware.resolve_referenced$resolve_referenced_card_resources$fn__46782.invoke(resolve_referenced.clj:79)"
  "query_processor.middleware.expand_macros$expand_macros$fn__47531.invoke(expand_macros.clj:184)"
  "query_processor.middleware.add_timezone_info$add_timezone_info$fn__45327.invoke(add_timezone_info.clj:15)"
  "query_processor.middleware.splice_params_in_response$splice_params_in_response$fn__49974.invoke(splice_params_in_response.clj:32)"
  "query_processor.middleware.resolve_database_and_driver$resolve_database_and_driver$fn__49184$fn__49188.invoke(resolve_database_and_driver.clj:31)"
  "driver$do_with_driver.invokeStatic(driver.clj:60)"
  "driver$do_with_driver.invoke(driver.clj:56)"
  "query_processor.middleware.resolve_database_and_driver$resolve_database_and_driver$fn__49184.invoke(resolve_database_and_driver.clj:25)"
  "query_processor.middleware.fetch_source_query$resolve_card_id_source_tables$fn__47771.invoke(fetch_source_query.clj:274)"
  "query_processor.middleware.store$initialize_store$fn__49983$fn__49984.invoke(store.clj:11)"
  "query_processor.store$do_with_store.invokeStatic(store.clj:44)"
  "query_processor.store$do_with_store.invoke(store.clj:38)"
  "query_processor.middleware.store$initialize_store$fn__49983.invoke(store.clj:10)"
  "query_processor.middleware.validate$validate_query$fn__50028.invoke(validate.clj:10)"
  "query_processor.middleware.normalize_query$normalize$fn__47907.invoke(normalize_query.clj:22)"
  "query_processor.middleware.add_rows_truncated$add_rows_truncated$fn__45173.invoke(add_rows_truncated.clj:35)"
  "query_processor.middleware.results_metadata$record_and_return_metadata_BANG_$fn__49959.invoke(results_metadata.clj:147)"
  "query_processor.middleware.constraints$add_default_userland_constraints$fn__47024.invoke(constraints.clj:42)"
  "query_processor.middleware.process_userland_query$process_userland_query$fn__49070.invoke(process_userland_query.clj:134)"
  "query_processor.middleware.catch_exceptions$catch_exceptions$fn__46964.invoke(catch_exceptions.clj:173)"
  "query_processor.reducible$async_qp$qp_STAR___38252$thunk__38253.invoke(reducible.clj:103)"
  "query_processor.reducible$async_qp$qp_STAR___38252.invoke(reducible.clj:109)"
  "query_processor.reducible$sync_qp$qp_STAR___38261$fn__38264.invoke(reducible.clj:135)"
  "query_processor.reducible$sync_qp$qp_STAR___38261.invoke(reducible.clj:134)"
  "query_processor$process_userland_query.invokeStatic(query_processor.clj:241)"
  "query_processor$process_userland_query.doInvoke(query_processor.clj:237)"
  "query_processor$fn__50118$process_query_and_save_execution_BANG___50127$fn__50130.invoke(query_processor.clj:253)"
  "query_processor$fn__50118$process_query_and_save_execution_BANG___50127.invoke(query_processor.clj:245)"
  "query_processor$fn__50162$process_query_and_save_with_max_results_constraints_BANG___50171$fn__50174.invoke(query_processor.clj:265)"
  "query_processor$fn__50162$process_query_and_save_with_max_results_constraints_BANG___50171.invoke(query_processor.clj:258)"
  "api.dataset$run_query_async$fn__56409.invoke(dataset.clj:56)"
  "query_processor.streaming$streaming_response_STAR_$fn__56388$fn__56389.invoke(streaming.clj:72)"
  "query_processor.streaming$streaming_response_STAR_$fn__56388.invoke(streaming.clj:71)"
  "async.streaming_response$do_f_STAR_.invokeStatic(streaming_response.clj:65)"
  "async.streaming_response$do_f_STAR_.invoke(streaming_response.clj:63)"
  "async.streaming_response$do_f_async$fn__16088.invoke(streaming_response.clj:84)"],
 :context :ad-hoc,
 :error
 "Column \"source.SUBTOTAL\" not found; SQL statement:\nCREATE FORCE VIEW PUBLIC._16 AS\nSELECT\n    \"source\".CREATED_AT AS CREATED_AT,\n    \"source\".SUBTOTAL AS SUBTOTAL,\n    (1 + 1) AS CC\nFROM (\n    SELECT\n        PARSEDATETIME(FORMATDATETIME(PUBLIC.ORDERS.CREATED_AT, 'yyyyMM'), 'yyyyMM') AS CREATED_AT,\n        COUNT(DISTINCT CASE WHEN (PUBLIC.ORDERS.DISCOUNT > 0.0) THEN PUBLIC.ORDERS.SUBTOTAL ELSE PUBLIC.ORDERS.TOTAL END) AS \"CE bad case field\"\n    FROM PUBLIC.ORDERS\n    GROUP BY PARSEDATETIME(FORMATDATETIME(PUBLIC.ORDERS.CREATED_AT, 'yyyyMM'), 'yyyyMM')\n    ORDER BY 1\n) \"source\" [42122-197]",
 :row_count 0,
 :running_time 0,
 :preprocessed
 {:database 4,
  :query
  {:expressions {:CC [:+ 1 1]},
   :source-metadata
   [{:semantic_type :type/CreationTimestamp,
     :table_id 11,
     :coercion_strategy nil,
     :unit :month,
     :name "CREATED_AT",
     :settings nil,
     :field_ref [:field 86 {:temporal-unit :month}],
     :effective_type :type/DateTime,
     :parent_id nil,
     :id 86,
     :display_name "Created At",
     :fingerprint
     {:global {:distinct-count 10001, :nil% 0.0},
      :type {:type/DateTime {:earliest "2016-05-01T00:56:13.352Z", :latest "2020-04-19T20:07:15.657Z"}}},
     :base_type :type/DateTime}
    {:semantic_type :type/Quantity,
     :table_id 11,
     :coercion_strategy nil,
     :name "CE bad case field",
     :settings {:currency "EUR", :currency_in_header false},
     :field_ref [:aggregation 0],
     :effective_type :type/Float,
     :parent_id nil,
     :id 84,
     :display_name "CE bad case field",
     :fingerprint
     {:global {:distinct-count 340, :nil% 0.0},
      :type
      {:type/Number
       {:min 15.691943673970439,
        :max 148.22900526552291,
        :avg 77.01295465356547,
        :sd 32.53705013056317,
        :q1 49.74894519060184,
        :q3 105.42965746993103}}},
     :base_type :type/BigInteger}],
   :fields [[:field 86 {:temporal-unit :default}] [:field 84 nil] [:expression "CC"]],
   :source-query
   {:source-table 11,
    :aggregation
    [[:aggregation-options
      [:distinct
       [:case
        [[[:>
           [:field 81 nil]
           [:value
            0
            {:base_type :type/Float,
             :effective_type :type/Float,
             :coercion_strategy nil,
             :semantic_type :type/Discount,
             :database_type "DOUBLE",
             :name "DISCOUNT"}]]
          [:field 84 nil]]]
        {:default [:field 88 nil]}]]
      {:name "CE bad case field", :display-name "CE bad case field"}]],
    :breakout [[:field 86 {:temporal-unit :month}]],
    :order-by [[:asc [:field 86 {:temporal-unit :month}]]]},
   :limit 2000},
  :type :query,
  :middleware {:js-int-to-string? true, :add-default-userland-constraints? true},
  :info
  {:executed-by 1,
   :context :ad-hoc,
   :nested? false,
   :query-hash
   [-89, -69, 101, -29, -107, 69, 13, -118, 68, -117, -99, -56, -123, 103, 9, 6, -91, -1, 78, 65, -59, -102, -52, 50,
    35, 114, -60, 16, 108, 57, -48, 105]},
  :constraints {:max-results 10000, :max-results-bare-rows 2000}},
 :data {:rows [], :cols []}}

2021-08-19 13:22:37,299 DEBUG middleware.log :: POST /api/dataset 202 [ASYNC: completed] 1.5 s (13 DB calls) App DB connections: 0/15 Jetty threads: 3/50 (3 idle, 0 queued) (127 total active threads) Queries in flight: 0 (0 queued); h2 DB 4 connections: 0/1 (0 threads blocked)

Information about your Metabase Installation:
Tested 0.39.4 thru 0.40.2 - regression since 0.40.0

Additional context
Slightly similar to #14859, which was fixed in 0.38.0

@flamber flamber added Type:Bug Product defects Priority:P2 Average run of the mill bug Querying/Nested Queries Questions based on other saved questions .Regression Bugs that were previously fixed and/or bugs unintentionally shipped with new features. labels Aug 19, 2021
@nemanjaglumac nemanjaglumac added this to Backlog in Cypress Testing Aug 19, 2021
nemanjaglumac added a commit that referenced this issue Aug 24, 2021
@nemanjaglumac nemanjaglumac added the .Reproduced Issues reproduced in test (usually Cypress) label Aug 24, 2021
@camsaul
Copy link
Member

camsaul commented Sep 7, 2021

I think this is probably related to #15578 and #17770 (not the same issue, but possibly can be fixed together)

@camsaul
Copy link
Member

camsaul commented Oct 20, 2021

Working on getting #16656 merged in to make this sort of thing easy to fix.

@dpsutton
Copy link
Contributor

I'm trying to repro this and not able to get the same error message:

Value does not match schema: {:query {:expressions {:CE (named (not (some-matching-condition? a-clojure.lang.PersistentVector)) "Must be a valid instance of one of these clauses: :expression, :field")}}}

image

image

I can reproduce with the following:
((s/validator FieldOrExpressionDef) [:distinct [:case [[[:> [:field 1 nil] 0] [:field 6 nil]]] {:default [:field 7 nil]}]]) throws the same error.

I'm asking because I'm working on #18513. I've solved that issue and this one feels like it might be similar. The problem there is that when doing type inference on the expression, it returns the field's entire information. For coalesce, it just returns the type inference on the second field. This includes a field-id and this is why it emits a reference to that field rather than to the expression's name. I suspect the error is the same here but i'm unable to reproduce it at the moment.

;; middleware/annotate.clj
(defn infer-expression-type
  "Infer base-type/semantic-type information about an `expression` clause."
  [expression]
  (cond
    (string? expression)
    {:base_type :type/Text}

    (number? expression)
    {:base_type :type/Number}

    (mbql.u/is-clause? :field expression)
    (col-info-for-field-clause {} expression)

    (mbql.u/is-clause? :coalesce expression)
    (infer-expression-type (second expression)) ;; this will return a field's information including id and name. But the id is what is untenable

    (mbql.u/is-clause? :length expression)
    {:base_type :type/BigInteger}

    (mbql.u/is-clause? :case expression)
    (let [[_ clauses] expression]
      (some
       (fn [[_ expression]]
         ;; get the first non-nil val
         (when (and (not= expression nil)
                    (or (not (mbql.u/is-clause? :value expression))
                        (let [[_ value] expression]
                          (not= value nil))))
           (infer-expression-type expression)))
       clauses))

    (mbql.u/datetime-arithmetics? expression)
    {:base_type :type/DateTime}

    (mbql.u/is-clause? mbql.s/string-expressions expression)
    {:base_type :type/Text}

    (mbql.u/is-clause? mbql.s/arithmetic-expressions expression)
    {:base_type :type/Float}

    :else
    {:base_type :type/*}))

Changing that coalesce clause to

    (mbql.u/is-clause? :coalesce expression)
    (select-keys (infer-expression-type (second expression))
                 [:base_type :effective_type :coercion_strategy :semantic_type])

fixes this defect by ensuring it only takes the type information of the target field. The id is so "poisonous" because mb.qp.m.add-implicit-clauses/source-metadata->fields will (understandably) emit [:field field-id nil] when there's a field id in the metadata.

This is why i suspect this is the same issue. A field id propagates from the case type inference leading to emitting selecting that field from the nested query.

@dpsutton
Copy link
Contributor

@camsaul see #19008 for what i think is a similar issue. tldr, computed column metadata can never have a field id in it.

@flamber
Copy link
Contributor Author

flamber commented Nov 17, 2021

@dpsutton You cannot use those functions on Custom Column. You'll have to use it for Custom Expression. The frontend doesn't correctly block functions that cannot be used, but that's a known issue.
CustomExpression: Distinct(case([Discount] > 0, [Subtotal], [Total]))
CustomColumn: 1 + 1
image

@dpsutton
Copy link
Contributor

thank you @flamber . I was being to careless. I've now fixed this in the same manner and PR as #19008

@camsaul camsaul assigned dpsutton and unassigned camsaul Nov 22, 2021
@flamber flamber added this to the 0.41.4 milestone Dec 8, 2021
nemanjaglumac added a commit that referenced this issue Jan 24, 2022
nemanjaglumac added a commit that referenced this issue Jan 25, 2022
github-actions bot pushed a commit that referenced this issue Jan 25, 2022
nemanjaglumac added a commit that referenced this issue Jan 25, 2022
Co-authored-by: Nemanja Glumac <31325167+nemanjaglumac@users.noreply.github.com>
This was referenced Feb 5, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Priority:P2 Average run of the mill bug Querying/Nested Queries Questions based on other saved questions Querying/Notebook/Custom Expression .Regression Bugs that were previously fixed and/or bugs unintentionally shipped with new features. .Reproduced Issues reproduced in test (usually Cypress) Type:Bug Product defects
Projects
Development

Successfully merging a pull request may close this issue.

4 participants