Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Multiple joined Saved Questions, which are based on the same table, causes incorrect field references #18502

Closed
flamber opened this issue Oct 16, 2021 · 6 comments · Fixed by #19791
Assignees
Labels
.Backend Priority:P2 Average run of the mill bug Querying/Nested Queries Questions based on other saved questions .Regression Bugs that were previously fixed and/or bugs unintentionally shipped with new features. .Reproduced Issues reproduced in test (usually Cypress) Type:Bug Product defects
Milestone

Comments

@flamber
Copy link
Contributor

flamber commented Oct 16, 2021

Describe the bug
When joining multiple Saved Questions, which are based on the same table, but with different group-by fields, then query fails because of incorrect table alias in the field reference.

Regression since 0.38.0

To Reproduce

  1. Simple question > Sample Dataset > People - summarize Count group by CreatedAt:Month - save as "Q1"
  2. Simple question > Sample Dataset > People - summarize Count group by BirthDate:Month - save as "Q2"
  3. Custom question > Saved Questions > "Q1" join "Q2" on CreatedAt=BirthDate - fails with incorrect reference
    Column "Q2.CREATED_AT" not found; (other databases might error with ERROR: column Q2.CREATED_AT does not exist)
SQL generated
SELECT
  "Question 941"."CREATED_AT" AS "Question 941__CREATED_AT",
  "source"."count" AS "count",
  "Question 941"."BIRTH_DATE" AS "Question 941__BIRTH_DATE",
  "Question 941"."count" AS "count_2" 
FROM
  (
    SELECT
      parsedatetime(formatdatetime("PUBLIC"."PEOPLE"."CREATED_AT", 'yyyyMM'), 'yyyyMM') AS "CREATED_AT",
      count(*) AS "count" 
    FROM
      "PUBLIC"."PEOPLE" 
    GROUP BY
      parsedatetime(formatdatetime("PUBLIC"."PEOPLE"."CREATED_AT", 'yyyyMM'), 'yyyyMM') 
    ORDER BY
      parsedatetime(formatdatetime("PUBLIC"."PEOPLE"."CREATED_AT", 'yyyyMM'), 'yyyyMM') ASC
  )
  "source" 
  LEFT JOIN
    (
      SELECT
        parsedatetime(formatdatetime("PUBLIC"."PEOPLE"."BIRTH_DATE", 'yyyyMM'), 'yyyyMM') AS "BIRTH_DATE",
        count(*) AS "count" 
      FROM
        "PUBLIC"."PEOPLE" 
      GROUP BY
        parsedatetime(formatdatetime("PUBLIC"."PEOPLE"."BIRTH_DATE", 'yyyyMM'), 'yyyyMM')
    )
    "Question 941" 
    ON parsedatetime(formatdatetime("source"."CREATED_AT", 'yyyyMM'), 'yyyyMM') = parsedatetime(formatdatetime("Question 941"."BIRTH_DATE", 'yyyyMM'), 'yyyyMM')
SQL 0.37.8
SELECT
  "source"."CREATED_AT" AS "CREATED_AT",
  "source"."count" AS "count",
  "Question 941"."BIRTH_DATE" AS "BIRTH_DATE",
  "Question 941"."count" AS "count_2" 
FROM
  (
    SELECT
      parsedatetime(formatdatetime("PUBLIC"."PEOPLE"."CREATED_AT", 'yyyyMM'), 'yyyyMM') AS "CREATED_AT",
      count(*) AS "count" 
    FROM
      "PUBLIC"."PEOPLE" 
    GROUP BY
      parsedatetime(formatdatetime("PUBLIC"."PEOPLE"."CREATED_AT", 'yyyyMM'), 'yyyyMM') 
    ORDER BY
      parsedatetime(formatdatetime("PUBLIC"."PEOPLE"."CREATED_AT", 'yyyyMM'), 'yyyyMM') ASC
  )
  "source" 
  LEFT JOIN
    (
      SELECT
        parsedatetime(formatdatetime("PUBLIC"."PEOPLE"."BIRTH_DATE", 'yyyyMM'), 'yyyyMM') AS "BIRTH_DATE",
        count(*) AS "count" 
      FROM
        "PUBLIC"."PEOPLE" 
      GROUP BY
        parsedatetime(formatdatetime("PUBLIC"."PEOPLE"."BIRTH_DATE", 'yyyyMM'), 'yyyyMM')
    )
    "Question 941" 
    ON "source"."CREATED_AT" = "Question 941"."BIRTH_DATE"
Full stacktrace
2021-10-16 15:12:21,855 INFO middleware.fetch-source-query :: Fetched source query from Card 940: 
 {:source-table 12, :aggregation [[:count]], :breakout [[:field 97 {:temporal-unit :month}]]}

2021-10-16 15:12:21,868 INFO middleware.fetch-source-query :: Fetched source query from Card 941: 
 {:source-table 12, :aggregation [[:count]], :breakout [[:field 90 {:temporal-unit :month}]]}

2021-10-16 15:12:21,914 INFO middleware.fetch-source-query :: Fetched source query from Card 940: 
 {:source-table 12, :aggregation [[:count]], :breakout [[:field 97 {:temporal-unit :month}]]}

2021-10-16 15:12:21,928 INFO middleware.fetch-source-query :: Fetched source query from Card 941: 
 {:source-table 12, :aggregation [[:count]], :breakout [[:field 90 {:temporal-unit :month}]]}

2021-10-16 15:12:23,222 ERROR middleware.catch-exceptions :: Error processing query: null
{:database_id 4,
 :started_at #t "2021-10-16T15:12:21.833981+02:00[Europe/Copenhagen]",
 :via
 [{:status :failed,
   :class clojure.lang.ExceptionInfo,
   :error "Error executing query",
   :stacktrace
   ["--> driver.sql_jdbc.execute$execute_reducible_query$fn__80803.invoke(execute.clj:480)"
    "driver.sql_jdbc.execute$execute_reducible_query.invokeStatic(execute.clj:477)"
    "driver.sql_jdbc.execute$execute_reducible_query.invoke(execute.clj:463)"
    "driver.sql_jdbc.execute$execute_reducible_query.invokeStatic(execute.clj:472)"
    "driver.sql_jdbc.execute$execute_reducible_query.invoke(execute.clj:463)"
    "driver.sql_jdbc$fn__82292.invokeStatic(sql_jdbc.clj:54)"
    "driver.sql_jdbc$fn__82292.invoke(sql_jdbc.clj:52)"
    "driver.h2$fn__81044.invokeStatic(h2.clj:83)"
    "driver.h2$fn__81044.invoke(h2.clj:80)"
    "query_processor.context$executef.invokeStatic(context.clj:59)"
    "query_processor.context$executef.invoke(context.clj:48)"
    "query_processor.context.default$default_runf.invokeStatic(default.clj:68)"
    "query_processor.context.default$default_runf.invoke(default.clj:66)"
    "query_processor.context$runf.invokeStatic(context.clj:45)"
    "query_processor.context$runf.invoke(context.clj:39)"
    "query_processor.reducible$pivot.invokeStatic(reducible.clj:34)"
    "query_processor.reducible$pivot.invoke(reducible.clj:31)"
    "query_processor.middleware.mbql_to_native$mbql__GT_native$fn__47903.invoke(mbql_to_native.clj:25)"
    "query_processor.middleware.check_features$check_features$fn__47017.invoke(check_features.clj:39)"
    "query_processor.middleware.limit$limit$fn__47889.invoke(limit.clj:37)"
    "query_processor.middleware.cache$maybe_return_cached_results$fn__46469.invoke(cache.clj:204)"
    "query_processor.middleware.optimize_temporal_filters$optimize_temporal_filters$fn__48149.invoke(optimize_temporal_filters.clj:204)"
    "query_processor.middleware.validate_temporal_bucketing$validate_temporal_bucketing$fn__50081.invoke(validate_temporal_bucketing.clj:50)"
    "query_processor.middleware.auto_parse_filter_values$auto_parse_filter_values$fn__45588.invoke(auto_parse_filter_values.clj:43)"
    "query_processor.middleware.wrap_value_literals$wrap_value_literals$fn__41716.invoke(wrap_value_literals.clj:161)"
    "query_processor.middleware.annotate$add_column_info$fn__41591.invoke(annotate.clj:608)"
    "query_processor.middleware.permissions$check_query_permissions$fn__46889.invoke(permissions.clj:81)"
    "query_processor.middleware.pre_alias_aggregations$pre_alias_aggregations$fn__49010.invoke(pre_alias_aggregations.clj:40)"
    "query_processor.middleware.cumulative_aggregations$handle_cumulative_aggregations$fn__47090.invoke(cumulative_aggregations.clj:60)"
    "query_processor.middleware.resolve_joined_fields$resolve_joined_fields$fn__49307.invoke(resolve_joined_fields.clj:102)"
    "query_processor.middleware.resolve_joins$resolve_joins$fn__49620.invoke(resolve_joins.clj:171)"
    "query_processor.middleware.add_implicit_joins$add_implicit_joins$fn__45164.invoke(add_implicit_joins.clj:190)"
    "query_processor.middleware.large_int_id$convert_id_to_string$fn__47853.invoke(large_int_id.clj:59)"
    "query_processor.middleware.format_rows$format_rows$fn__47834.invoke(format_rows.clj:74)"
    "query_processor.middleware.add_default_temporal_unit$add_default_temporal_unit$fn__44458.invoke(add_default_temporal_unit.clj:23)"
    "query_processor.middleware.desugar$desugar$fn__47156.invoke(desugar.clj:21)"
    "query_processor.middleware.binning$update_binning_strategy$fn__45975.invoke(binning.clj:229)"
    "query_processor.middleware.resolve_fields$resolve_fields$fn__46692.invoke(resolve_fields.clj:34)"
    "query_processor.middleware.add_dimension_projections$add_remapping$fn__44813.invoke(add_dimension_projections.clj:314)"
    "query_processor.middleware.add_implicit_clauses$add_implicit_clauses$fn__45042.invoke(add_implicit_clauses.clj:147)"
    "query_processor.middleware.upgrade_field_literals$upgrade_field_literals$fn__50030.invoke(upgrade_field_literals.clj:40)"
    "query_processor.middleware.add_source_metadata$add_source_metadata_for_source_queries$fn__45327.invoke(add_source_metadata.clj:123)"
    "query_processor.middleware.reconcile_breakout_and_order_by_bucketing$reconcile_breakout_and_order_by_bucketing$fn__49182.invoke(reconcile_breakout_and_order_by_bucketing.clj:100)"
    "query_processor.middleware.auto_bucket_datetimes$auto_bucket_datetimes$fn__45535.invoke(auto_bucket_datetimes.clj:147)"
    "query_processor.middleware.resolve_source_table$resolve_source_tables$fn__46739.invoke(resolve_source_table.clj:45)"
    "query_processor.middleware.parameters$substitute_parameters$fn__48992.invoke(parameters.clj:111)"
    "query_processor.middleware.resolve_referenced$resolve_referenced_card_resources$fn__46791.invoke(resolve_referenced.clj:79)"
    "query_processor.middleware.expand_macros$expand_macros$fn__47540.invoke(expand_macros.clj:184)"
    "query_processor.middleware.add_timezone_info$add_timezone_info$fn__45336.invoke(add_timezone_info.clj:15)"
    "query_processor.middleware.splice_params_in_response$splice_params_in_response$fn__49983.invoke(splice_params_in_response.clj:32)"
    "query_processor.middleware.resolve_database_and_driver$resolve_database_and_driver$fn__49193$fn__49197.invoke(resolve_database_and_driver.clj:31)"
    "driver$do_with_driver.invokeStatic(driver.clj:60)"
    "driver$do_with_driver.invoke(driver.clj:56)"
    "query_processor.middleware.resolve_database_and_driver$resolve_database_and_driver$fn__49193.invoke(resolve_database_and_driver.clj:25)"
    "query_processor.middleware.fetch_source_query$resolve_card_id_source_tables$fn__47780.invoke(fetch_source_query.clj:273)"
    "query_processor.middleware.store$initialize_store$fn__49992$fn__49993.invoke(store.clj:11)"
    "query_processor.store$do_with_store.invokeStatic(store.clj:44)"
    "query_processor.store$do_with_store.invoke(store.clj:38)"
    "query_processor.middleware.store$initialize_store$fn__49992.invoke(store.clj:10)"
    "query_processor.middleware.validate$validate_query$fn__50037.invoke(validate.clj:10)"
    "query_processor.middleware.normalize_query$normalize$fn__47916.invoke(normalize_query.clj:22)"
    "query_processor.middleware.add_rows_truncated$add_rows_truncated$fn__45182.invoke(add_rows_truncated.clj:35)"
    "query_processor.middleware.results_metadata$record_and_return_metadata_BANG_$fn__49968.invoke(results_metadata.clj:147)"
    "query_processor.middleware.constraints$add_default_userland_constraints$fn__47033.invoke(constraints.clj:42)"
    "query_processor.middleware.process_userland_query$process_userland_query$fn__49079.invoke(process_userland_query.clj:134)"
    "query_processor.middleware.catch_exceptions$catch_exceptions$fn__46973.invoke(catch_exceptions.clj:173)"
    "query_processor.reducible$async_qp$qp_STAR___38255$thunk__38256.invoke(reducible.clj:103)"
    "query_processor.reducible$async_qp$qp_STAR___38255$fn__38258.invoke(reducible.clj:108)"],
   :error_type :invalid-query,
   :ex-data
   {:sql
    "-- Metabase:: userID: 1 queryType: MBQL queryHash: 553946f7b6571f0e7b4012361a812be8072c469a70119f5796510cdbf55322db\nSELECT \"Question 941\".\"CREATED_AT\" AS \"Question 941__CREATED_AT\", \"source\".\"count\" AS \"count\", \"Question 941\".\"BIRTH_DATE\" AS \"Question 941__BIRTH_DATE\", \"Question 941\".\"count\" AS \"count_2\" FROM (SELECT parsedatetime(formatdatetime(\"PUBLIC\".\"PEOPLE\".\"CREATED_AT\", 'yyyyMM'), 'yyyyMM') AS \"CREATED_AT\", count(*) AS \"count\" FROM \"PUBLIC\".\"PEOPLE\" GROUP BY parsedatetime(formatdatetime(\"PUBLIC\".\"PEOPLE\".\"CREATED_AT\", 'yyyyMM'), 'yyyyMM') ORDER BY parsedatetime(formatdatetime(\"PUBLIC\".\"PEOPLE\".\"CREATED_AT\", 'yyyyMM'), 'yyyyMM') ASC) \"source\" LEFT JOIN (SELECT parsedatetime(formatdatetime(\"PUBLIC\".\"PEOPLE\".\"BIRTH_DATE\", 'yyyyMM'), 'yyyyMM') AS \"BIRTH_DATE\", count(*) AS \"count\" FROM \"PUBLIC\".\"PEOPLE\" GROUP BY parsedatetime(formatdatetime(\"PUBLIC\".\"PEOPLE\".\"BIRTH_DATE\", 'yyyyMM'), 'yyyyMM')) \"Question 941\" ON parsedatetime(formatdatetime(\"source\".\"CREATED_AT\", 'yyyyMM'), 'yyyyMM') = parsedatetime(formatdatetime(\"Question 941\".\"BIRTH_DATE\", 'yyyyMM'), 'yyyyMM') LIMIT 2000",
    :params nil,
    :type :invalid-query}}],
 :state "42S22",
 :error_type :invalid-query,
 :json_query
 {:constraints {:max-results 10000, :max-results-bare-rows 2000},
  :type :query,
  :middleware {:js-int-to-string? true, :ignore-cached-results? false},
  :database 4,
  :query
  {:source-table "card__940",
   :joins
   [{:fields :all,
     :source-table "card__941",
     :condition
     [:=
      [:field "CREATED_AT" {:base-type :type/DateTime}]
      [:field "BIRTH_DATE" {:base-type :type/DateTime, :join-alias "Question 941"}]],
     :alias "Question 941"}]},
  :parameters [],
  :async? true,
  :cache-ttl nil},
 :native
 {:query
  "SELECT \"Question 941\".\"CREATED_AT\" AS \"Question 941__CREATED_AT\", \"source\".\"count\" AS \"count\", \"Question 941\".\"BIRTH_DATE\" AS \"Question 941__BIRTH_DATE\", \"Question 941\".\"count\" AS \"count_2\" FROM (SELECT parsedatetime(formatdatetime(\"PUBLIC\".\"PEOPLE\".\"CREATED_AT\", 'yyyyMM'), 'yyyyMM') AS \"CREATED_AT\", count(*) AS \"count\" FROM \"PUBLIC\".\"PEOPLE\" GROUP BY parsedatetime(formatdatetime(\"PUBLIC\".\"PEOPLE\".\"CREATED_AT\", 'yyyyMM'), 'yyyyMM') ORDER BY parsedatetime(formatdatetime(\"PUBLIC\".\"PEOPLE\".\"CREATED_AT\", 'yyyyMM'), 'yyyyMM') ASC) \"source\" LEFT JOIN (SELECT parsedatetime(formatdatetime(\"PUBLIC\".\"PEOPLE\".\"BIRTH_DATE\", 'yyyyMM'), 'yyyyMM') AS \"BIRTH_DATE\", count(*) AS \"count\" FROM \"PUBLIC\".\"PEOPLE\" GROUP BY parsedatetime(formatdatetime(\"PUBLIC\".\"PEOPLE\".\"BIRTH_DATE\", 'yyyyMM'), 'yyyyMM')) \"Question 941\" ON parsedatetime(formatdatetime(\"source\".\"CREATED_AT\", 'yyyyMM'), 'yyyyMM') = parsedatetime(formatdatetime(\"Question 941\".\"BIRTH_DATE\", 'yyyyMM'), 'yyyyMM') LIMIT 2000",
  :params nil},
 :status :failed,
 :class org.h2.jdbc.JdbcSQLException,
 :stacktrace
 ["org.h2.message.DbException.getJdbcSQLException(DbException.java:357)"
  "org.h2.message.DbException.get(DbException.java:179)"
  "org.h2.message.DbException.get(DbException.java:155)"
  "org.h2.expression.ExpressionColumn.optimize(ExpressionColumn.java:150)"
  "org.h2.expression.Alias.optimize(Alias.java:51)"
  "org.h2.command.dml.Select.prepare(Select.java:858)"
  "org.h2.command.Parser.prepareCommand(Parser.java:283)"
  "org.h2.engine.Session.prepareLocal(Session.java:611)"
  "org.h2.engine.Session.prepareCommand(Session.java:549)"
  "org.h2.jdbc.JdbcConnection.prepareCommand(JdbcConnection.java:1247)"
  "org.h2.jdbc.JdbcStatement.executeInternal(JdbcStatement.java:217)"
  "org.h2.jdbc.JdbcStatement.execute(JdbcStatement.java:205)"
  "com.mchange.v2.c3p0.impl.NewProxyStatement.execute(NewProxyStatement.java:75)"
  "--> driver.sql_jdbc.execute$fn__80723.invokeStatic(execute.clj:344)"
  "driver.sql_jdbc.execute$fn__80723.invoke(execute.clj:342)"
  "driver.sql_jdbc.execute$execute_statement_or_prepared_statement_BANG_.invokeStatic(execute.clj:352)"
  "driver.sql_jdbc.execute$execute_statement_or_prepared_statement_BANG_.invoke(execute.clj:349)"
  "driver.sql_jdbc.execute$execute_reducible_query$fn__80803.invoke(execute.clj:478)"
  "driver.sql_jdbc.execute$execute_reducible_query.invokeStatic(execute.clj:477)"
  "driver.sql_jdbc.execute$execute_reducible_query.invoke(execute.clj:463)"
  "driver.sql_jdbc.execute$execute_reducible_query.invokeStatic(execute.clj:472)"
  "driver.sql_jdbc.execute$execute_reducible_query.invoke(execute.clj:463)"
  "driver.sql_jdbc$fn__82292.invokeStatic(sql_jdbc.clj:54)"
  "driver.sql_jdbc$fn__82292.invoke(sql_jdbc.clj:52)"
  "driver.h2$fn__81044.invokeStatic(h2.clj:83)"
  "driver.h2$fn__81044.invoke(h2.clj:80)"
  "query_processor.context$executef.invokeStatic(context.clj:59)"
  "query_processor.context$executef.invoke(context.clj:48)"
  "query_processor.context.default$default_runf.invokeStatic(default.clj:68)"
  "query_processor.context.default$default_runf.invoke(default.clj:66)"
  "query_processor.context$runf.invokeStatic(context.clj:45)"
  "query_processor.context$runf.invoke(context.clj:39)"
  "query_processor.reducible$pivot.invokeStatic(reducible.clj:34)"
  "query_processor.reducible$pivot.invoke(reducible.clj:31)"
  "query_processor.middleware.mbql_to_native$mbql__GT_native$fn__47903.invoke(mbql_to_native.clj:25)"
  "query_processor.middleware.check_features$check_features$fn__47017.invoke(check_features.clj:39)"
  "query_processor.middleware.limit$limit$fn__47889.invoke(limit.clj:37)"
  "query_processor.middleware.cache$maybe_return_cached_results$fn__46469.invoke(cache.clj:204)"
  "query_processor.middleware.optimize_temporal_filters$optimize_temporal_filters$fn__48149.invoke(optimize_temporal_filters.clj:204)"
  "query_processor.middleware.validate_temporal_bucketing$validate_temporal_bucketing$fn__50081.invoke(validate_temporal_bucketing.clj:50)"
  "query_processor.middleware.auto_parse_filter_values$auto_parse_filter_values$fn__45588.invoke(auto_parse_filter_values.clj:43)"
  "query_processor.middleware.wrap_value_literals$wrap_value_literals$fn__41716.invoke(wrap_value_literals.clj:161)"
  "query_processor.middleware.annotate$add_column_info$fn__41591.invoke(annotate.clj:608)"
  "query_processor.middleware.permissions$check_query_permissions$fn__46889.invoke(permissions.clj:81)"
  "query_processor.middleware.pre_alias_aggregations$pre_alias_aggregations$fn__49010.invoke(pre_alias_aggregations.clj:40)"
  "query_processor.middleware.cumulative_aggregations$handle_cumulative_aggregations$fn__47090.invoke(cumulative_aggregations.clj:60)"
  "query_processor.middleware.resolve_joined_fields$resolve_joined_fields$fn__49307.invoke(resolve_joined_fields.clj:102)"
  "query_processor.middleware.resolve_joins$resolve_joins$fn__49620.invoke(resolve_joins.clj:171)"
  "query_processor.middleware.add_implicit_joins$add_implicit_joins$fn__45164.invoke(add_implicit_joins.clj:190)"
  "query_processor.middleware.large_int_id$convert_id_to_string$fn__47853.invoke(large_int_id.clj:59)"
  "query_processor.middleware.format_rows$format_rows$fn__47834.invoke(format_rows.clj:74)"
  "query_processor.middleware.add_default_temporal_unit$add_default_temporal_unit$fn__44458.invoke(add_default_temporal_unit.clj:23)"
  "query_processor.middleware.desugar$desugar$fn__47156.invoke(desugar.clj:21)"
  "query_processor.middleware.binning$update_binning_strategy$fn__45975.invoke(binning.clj:229)"
  "query_processor.middleware.resolve_fields$resolve_fields$fn__46692.invoke(resolve_fields.clj:34)"
  "query_processor.middleware.add_dimension_projections$add_remapping$fn__44813.invoke(add_dimension_projections.clj:314)"
  "query_processor.middleware.add_implicit_clauses$add_implicit_clauses$fn__45042.invoke(add_implicit_clauses.clj:147)"
  "query_processor.middleware.upgrade_field_literals$upgrade_field_literals$fn__50030.invoke(upgrade_field_literals.clj:40)"
  "query_processor.middleware.add_source_metadata$add_source_metadata_for_source_queries$fn__45327.invoke(add_source_metadata.clj:123)"
  "query_processor.middleware.reconcile_breakout_and_order_by_bucketing$reconcile_breakout_and_order_by_bucketing$fn__49182.invoke(reconcile_breakout_and_order_by_bucketing.clj:100)"
  "query_processor.middleware.auto_bucket_datetimes$auto_bucket_datetimes$fn__45535.invoke(auto_bucket_datetimes.clj:147)"
  "query_processor.middleware.resolve_source_table$resolve_source_tables$fn__46739.invoke(resolve_source_table.clj:45)"
  "query_processor.middleware.parameters$substitute_parameters$fn__48992.invoke(parameters.clj:111)"
  "query_processor.middleware.resolve_referenced$resolve_referenced_card_resources$fn__46791.invoke(resolve_referenced.clj:79)"
  "query_processor.middleware.expand_macros$expand_macros$fn__47540.invoke(expand_macros.clj:184)"
  "query_processor.middleware.add_timezone_info$add_timezone_info$fn__45336.invoke(add_timezone_info.clj:15)"
  "query_processor.middleware.splice_params_in_response$splice_params_in_response$fn__49983.invoke(splice_params_in_response.clj:32)"
  "query_processor.middleware.resolve_database_and_driver$resolve_database_and_driver$fn__49193$fn__49197.invoke(resolve_database_and_driver.clj:31)"
  "driver$do_with_driver.invokeStatic(driver.clj:60)"
  "driver$do_with_driver.invoke(driver.clj:56)"
  "query_processor.middleware.resolve_database_and_driver$resolve_database_and_driver$fn__49193.invoke(resolve_database_and_driver.clj:25)"
  "query_processor.middleware.fetch_source_query$resolve_card_id_source_tables$fn__47780.invoke(fetch_source_query.clj:273)"
  "query_processor.middleware.store$initialize_store$fn__49992$fn__49993.invoke(store.clj:11)"
  "query_processor.store$do_with_store.invokeStatic(store.clj:44)"
  "query_processor.store$do_with_store.invoke(store.clj:38)"
  "query_processor.middleware.store$initialize_store$fn__49992.invoke(store.clj:10)"
  "query_processor.middleware.validate$validate_query$fn__50037.invoke(validate.clj:10)"
  "query_processor.middleware.normalize_query$normalize$fn__47916.invoke(normalize_query.clj:22)"
  "query_processor.middleware.add_rows_truncated$add_rows_truncated$fn__45182.invoke(add_rows_truncated.clj:35)"
  "query_processor.middleware.results_metadata$record_and_return_metadata_BANG_$fn__49968.invoke(results_metadata.clj:147)"
  "query_processor.middleware.constraints$add_default_userland_constraints$fn__47033.invoke(constraints.clj:42)"
  "query_processor.middleware.process_userland_query$process_userland_query$fn__49079.invoke(process_userland_query.clj:134)"
  "query_processor.middleware.catch_exceptions$catch_exceptions$fn__46973.invoke(catch_exceptions.clj:173)"
  "query_processor.reducible$async_qp$qp_STAR___38255$thunk__38256.invoke(reducible.clj:103)"
  "query_processor.reducible$async_qp$qp_STAR___38255$fn__38258.invoke(reducible.clj:108)"],
 :context :question,
 :error
 "Column \"Question 941.CREATED_AT\" not found; SQL statement:\n-- Metabase:: userID: 1 queryType: MBQL queryHash: 553946f7b6571f0e7b4012361a812be8072c469a70119f5796510cdbf55322db\nSELECT \"Question 941\".\"CREATED_AT\" AS \"Question 941__CREATED_AT\", \"source\".\"count\" AS \"count\", \"Question 941\".\"BIRTH_DATE\" AS \"Question 941__BIRTH_DATE\", \"Question 941\".\"count\" AS \"count_2\" FROM (SELECT parsedatetime(formatdatetime(\"PUBLIC\".\"PEOPLE\".\"CREATED_AT\", 'yyyyMM'), 'yyyyMM') AS \"CREATED_AT\", count(*) AS \"count\" FROM \"PUBLIC\".\"PEOPLE\" GROUP BY parsedatetime(formatdatetime(\"PUBLIC\".\"PEOPLE\".\"CREATED_AT\", 'yyyyMM'), 'yyyyMM') ORDER BY parsedatetime(formatdatetime(\"PUBLIC\".\"PEOPLE\".\"CREATED_AT\", 'yyyyMM'), 'yyyyMM') ASC) \"source\" LEFT JOIN (SELECT parsedatetime(formatdatetime(\"PUBLIC\".\"PEOPLE\".\"BIRTH_DATE\", 'yyyyMM'), 'yyyyMM') AS \"BIRTH_DATE\", count(*) AS \"count\" FROM \"PUBLIC\".\"PEOPLE\" GROUP BY parsedatetime(formatdatetime(\"PUBLIC\".\"PEOPLE\".\"BIRTH_DATE\", 'yyyyMM'), 'yyyyMM')) \"Question 941\" ON parsedatetime(formatdatetime(\"source\".\"CREATED_AT\", 'yyyyMM'), 'yyyyMM') = parsedatetime(formatdatetime(\"Question 941\".\"BIRTH_DATE\", 'yyyyMM'), 'yyyyMM') LIMIT 2000 [42122-197]",
 :row_count 0,
 :running_time 0,
 :preprocessed
 {:constraints {:max-results 10000, :max-results-bare-rows 2000},
  :type :query,
  :middleware {:js-int-to-string? true, :ignore-cached-results? false},
  :info
  {:executed-by 1,
   :context :question,
   :card-id 942,
   :query-hash
   [85, 57, 70, -9, -74, 87, 31, 14, 123, 64, 18, 54, 26, -127, 43, -24, 7, 44, 70, -102, 112, 17, -97, 87, -106, 81,
    12, -37, -11, 83, 34, -37]},
  :database 4,
  :query
  {:source-metadata
   [{:description "The date the user record was created. Also referred to as the user’s \"join date\"",
     :special_type "type/CreationTimestamp",
     :unit :month,
     :name "CREATED_AT",
     :field_ref [:field 97 {:temporal-unit :month, :join-alias "Question 941"}],
     :id 97,
     :display_name "Created At",
     :fingerprint
     {:global {:distinct-count 2500, :nil% 0.0},
      :type {:type/DateTime {:earliest "2016-04-20T03:35:18.752Z", :latest "2019-04-19T20:06:27.300Z"}}},
     :base_type :type/DateTime}
    {:name "count",
     :display_name "Count",
     :base_type :type/BigInteger,
     :special_type "type/Quantity",
     :field_ref [:aggregation 0],
     :fingerprint
     {:global {:distinct-count 23, :nil% 0.0},
      :type
      {:type/Number
       {:min 26.0, :q1 64.12917130661303, :q3 72.375, :max 91.0, :sd 10.356266327796531, :avg 67.56756756756756}}}}],
   :fields
   [[:field 97 {:temporal-unit :default, :join-alias "Question 941"}]
    [:field "count" {:base-type :type/BigInteger}]
    [:field 90 {:join-alias "Question 941"}]
    [:field "count" {:base-type :type/BigInteger, :join-alias "Question 941"}]],
   :joins
   [{:strategy :left-join,
     :condition
     [:= [:field 97 {:temporal-unit :month}] [:field 90 {:temporal-unit :month, :join-alias "Question 941"}]],
     :alias "Question 941",
     :source-query
     {:source-table 12,
      :aggregation [[:aggregation-options [:count] {:name "count"}]],
      :breakout [[:field 90 {:temporal-unit :month}]]},
     :source-metadata
     [{:description "The date of birth of the user",
       :special_type "type/Birthdate",
       :unit :month,
       :name "BIRTH_DATE",
       :field_ref [:field 90 {:temporal-unit :month}],
       :id 90,
       :display_name "Birth Date",
       :fingerprint
       {:global {:distinct-count 2308, :nil% 0.0},
        :type {:type/DateTime {:earliest "1958-04-26T06:00:00.000Z", :latest "2000-04-03T06:00:00.000Z"}}},
       :base_type :type/DateTime}
      {:name "count",
       :display_name "Count",
       :base_type :type/BigInteger,
       :special_type "type/Quantity",
       :field_ref [:aggregation 0],
       :fingerprint
       {:global {:distinct-count 12, :nil% 0.0},
        :type
        {:type/Number
         {:min 1.0,
          :q1 3.357247182083807,
          :q3 6.450040917105702,
          :max 12.0,
          :sd 2.2200276460890356,
          :avg 5.030181086519114}}}}]}],
   :source-query
   {:source-table 12,
    :aggregation [[:aggregation-options [:count] {:name "count"}]],
    :breakout [[:field 97 {:temporal-unit :month}]],
    :order-by [[:asc [:field 97 {:temporal-unit :month}]]]},
   :limit 2000},
  :async? true},
 :data {:rows [], :cols []}}

2021-10-16 15:12:23,236 DEBUG middleware.log :: POST /api/card/942/query 202 [ASYNC: completed] 1.4 s (14 DB calls) App DB connections: 0/7 Jetty threads: 3/50 (2 idle, 0 queued) (56 total active threads) Queries in flight: 1 (0 queued); h2 DB 4 connections: 0/2 (0 threads blocked)

Information about your Metabase Installation:
Tested 0.37.8 thru 0.41.0 and master - regression since 0.38.0

Additional context
Perhaps related to #14787 and #16389 and #17770, which are also regressions since 0.38.0, but this behaves differently.

@camsaul
Copy link
Member

camsaul commented Oct 25, 2021

This is probably going to be part of the #18580 group of fixes

camsaul added a commit that referenced this issue Jan 19, 2022
@camsaul
Copy link
Member

camsaul commented Jan 20, 2022

Generated SQL looks like (Postgres)

SELECT 
  "Q2"."created_at" AS "Q2__created_at",
  "source"."count"  AS "count",
  "Q2"."birth_date" AS "Q2__birth_date",
  "Q2"."count"      AS "Q2__count" 
FROM (
  SELECT 
    date_trunc('month', CAST("public"."people"."created_at" AS timestamp)) AS "created_at",
    count(*)                                                               AS "count" 
  FROM 
    "public"."people" 
  GROUP BY 
    date_trunc('month', CAST("public"."people"."created_at" AS timestamp)) 
  ORDER BY 
    date_trunc('month', CAST("public"."people"."created_at" AS timestamp)) ASC
) "source" 
LEFT JOIN (
  SELECT 
    date_trunc('month', CAST("public"."people"."birth_date" AS timestamp)) AS "birth_date",
    count(*)                                                               AS "count" 
  FROM 
    "public"."people" 
  GROUP BY 
    date_trunc('month', CAST("public"."people"."birth_date" AS timestamp)) 
  ORDER BY 
    date_trunc('month', CAST("public"."people"."birth_date" AS timestamp)) ASC
) "Q2"
  ON date_trunc('month', CAST("source"."created_at" AS timestamp))
   = date_trunc('month', CAST("Q2"."birth_date" AS timestamp)) 
LIMIT 3

@camsaul
Copy link
Member

camsaul commented Jan 20, 2022

I noticed the above SQL looked really silly with all that unnecesary casting so I made a few adjustments and the improved SQL is now:

SELECT 
  "Q2"."created_at" AS "Q2__created_at",
  "source"."count"  AS "count",
  "Q2"."birth_date" AS "Q2__birth_date",
  "Q2"."count"      AS "Q2__count" 
FROM (
  SELECT 
    date_trunc('month', "public"."people"."created_at") AS "created_at",
    count(*)                                            AS "count" 
  FROM 
    "public"."people" 
  GROUP BY 
    date_trunc('month', "public"."people"."created_at") 
  ORDER BY 
    date_trunc('month', "public"."people"."created_at") ASC
) "source" 
LEFT JOIN (
  SELECT 
    date_trunc('month', "public"."people"."birth_date") AS "birth_date",
    count(*)                                            AS "count" 
  FROM 
    "public"."people" 
  GROUP BY 
    date_trunc('month', "public"."people"."birth_date") 
  ORDER BY 
    date_trunc('month', "public"."people"."birth_date") ASC
) "Q2"
  ON date_trunc('month', "source"."created_at")
   = date_trunc('month', "Q2"."birth_date") 
LIMIT 3

I might split that stuff on into a separate PR or just do it as part of the fix for this issue.

@camsaul
Copy link
Member

camsaul commented Jan 20, 2022

The error these days is

Error executing query: ERROR: column Q2.created_at does not exist

Clearly someone is on drugs. For some reason it thinks created_at comes from Q2 when actually it comes from source

@camsaul
Copy link
Member

camsaul commented Jan 20, 2022

Split the PG cast stuff off into #19790

@camsaul
Copy link
Member

camsaul commented Jan 20, 2022

Ok, so this is a bug in the QP resolve-joined-fields middleware.

As you can see it's incorrectly rewriting the !default.created_at in the top-level :fields to !default.&Q2.created_at. I think it's assuming that because created_at could theoretically come from the join (since the join is against people) that it does come from the join, which is actually not the case

Before:

(mt/mbql-query people
  {:source-query {:source-table $$people
                  :breakout     [!month.created_at]
                  :aggregation  [[:count]]
                  :order-by     [[:asc !month.created_at]]}
   :joins        [{:source-query {:source-table $$people
                                  :breakout     [!month.birth_date]
                                  :aggregation  [[:count]]
                                  :order-by     [[:asc !month.birth_date]]}
                   :alias        "Q2"
                   :condition    [:= !month.created_at !month.&Q2.birth_date]
                   :fields       [&Q2.birth_date &Q2.*count/BigInteger]
                   :strategy     :left-join}]
   :fields       [!default.created_at
                  *count/BigInteger
                  &Q2.birth_date
                  &Q2.*count/BigInteger]
   :limit        3})

After:

(mt/mbql-query people
  {:source-query {:source-table $$people
                  :breakout     [!month.created_at]
                  :aggregation  [[:count]]
                  :order-by     [[:asc !month.created_at]]}
   :joins        [{:source-query {:source-table $$people
                                  :breakout     [!month.birth_date]
                                  :aggregation  [[:count]]
                                  :order-by     [[:asc !month.birth_date]]}
                   :alias        "Q2"
                   :condition    [:= !month.created_at !month.&Q2.birth_date]
                   :fields       [&Q2.birth_date &Q2.*count/BigInteger]
                   :strategy     :left-join}]
   :fields       [!default.&Q2.created_at
                  *count/BigInteger
                  &Q2.birth_date
                  &Q2.*count/BigInteger]
   :limit        3})

This was referenced May 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
.Backend Priority:P2 Average run of the mill bug Querying/Nested Queries Questions based on other saved questions .Regression Bugs that were previously fixed and/or bugs unintentionally shipped with new features. .Reproduced Issues reproduced in test (usually Cypress) Type:Bug Product defects
Projects
Development

Successfully merging a pull request may close this issue.

3 participants