MDEV-24931: Fix Bitmap<64>::is_prefix assertion with >64-column NATURAL JOIN on derived table by bodyhedia44 · Pull Request #4756 · MariaDB/server

bodyhedia44 · 2026-03-08T22:49:12Z

Problem

When a NATURAL JOIN operates on a derived table (or view with derived_merge=off) having more than 64 columns, the server crashes with:
Assertion `prefix_size <= width' failed in Bitmap<64u>::is_prefix

UBSAN also reports: shift exponent 32 is too large for 32-bit type 'int'

Root Cause

generate_derived_keys_for_table() in sql_select.cc creates derived keys with no cap on the number of key parts. When a 65-column table is NATURAL JOINed:

Shift overflow: (key_part_map)(1 << parts) — when parts >= 32, this is undefined behavior (shifting a 32-bit 1 by 32+ bits).
Bitmap overflow: A derived key with 65 parts is created, but key_map (Bitmap<64>) and key_part_map (uint64) can only track 64 bits.
Crash: make_join_statistics() calls eq_part.is_prefix(65) on a Bitmap<64>, hitting the assertion.

How to Reproduce

CREATE DATABASE IF NOT EXISTS test;
USE test;

DROP TABLE IF EXISTS t;

CREATE TABLE t (c1 INT,c2 INT,c3 INT,c4 INT,c5 INT,c6 INT,c7 INT,c8 INT,c9 INT,c10 INT,c11 INT,c12 INT,c13 INT,c14 INT,c15 INT,c16 INT,c17 INT,c18 INT,c19 INT,c20 INT,c21 INT,c22 INT,c23 INT,c24 INT,c25 INT,c26 INT,c27 INT,c28 INT,c29 INT,c30 INT,c31 INT,c32 INT,c33 INT,c34 INT,c35 INT,c36 INT,c37 INT,c38 INT,c39 INT,c40 INT,c41 INT,c42 INT,c43 INT,c44 INT,c45 INT,c46 INT,c47 INT,c48 INT,c49 INT,c50 INT,c51 INT,c52 INT,c53 INT,c54 INT,c55 INT,c56 INT,c57 INT,c58 INT,c59 INT,c60 INT,c61 INT,c62 INT,c63 INT,c64 INT,c65 INT) ENGINE=InnoDB;

SET SESSION optimizer_switch='derived_merge=off';

SELECT * FROM t AS a NATURAL JOIN (SELECT * FROM t) AS b;

DROP TABLE t;

Fix

Cap derived keys at sizeof(key_part_map) * 8 (64) parts in generate_derived_keys_for_table().
Excess key parts beyond 64 are excluded (keyuse->key = MAX_KEY).
Fix the shift to cast before shifting: (key_part_map) 1 << parts instead of (key_part_map)(1 << parts).

Test

Added main.mdev24931 with three cases:

65-column derived table via subquery (the crash case).
65-column view with derived_merge=off (original bug report).
64-column boundary test (should always work).

gkodinov

Thank you for your contribution. This is a preliminary review.

gkodinov · 2026-03-09T09:49:07Z

  KEYUSE *first_keyuse= keyuse;
  uint prev_part= keyuse->keypart;
  uint parts= 0;
+  uint max_parts= sizeof(key_part_map) * 8;


This should not be sizeof(key_part_map) IMHO. I'd use table->file->max_key_parts() to get the info from the SE.

gkodinov · 2026-03-09T09:51:42Z

    while (i < count && keyuse->used_tables == first_keyuse->used_tables &&
           keyuse->keypart == prev_part);
-    parts++;
+    if (parts < max_parts)


why the extra check here? You are already checking above.

bodyhedia44 · 2026-03-10T10:40:31Z

Done

gkodinov

Thanks. LGTM. Please stand by for the final review.

mariadb-RexJohnston

Please rebase your commit onto 10.11.

mariadb-RexJohnston · 2026-03-11T01:38:03Z

+SET SESSION optimizer_switch= @save_optimizer_switch;
+DROP VIEW v1;
+DROP TABLE t1;
+#


please collapse these 2 tests into one and add a test for joining a CTE.

mariadb-RexJohnston · 2026-03-11T01:39:01Z

+SELECT * FROM t64 AS a NATURAL JOIN (SELECT * FROM t64) AS b;
+c1	c2	c3	c4	c5	c6	c7	c8	c9	c10	c11	c12	c13	c14	c15	c16	c17	c18	c19	c20	c21	c22	c23	c24	c25	c26	c27	c28	c29	c30	c31	c32	c33	c34	c35	c36	c37	c38	c39	c40	c41	c42	c43	c44	c45	c46	c47	c48	c49	c50	c51	c52	c53	c54	c55	c56	c57	c58	c59	c60	c61	c62	c63	c64
+SET SESSION optimizer_switch= @save_optimizer_switch;
+DROP TABLE t64;


Does this test fail without your patch?

mariadb-RexJohnston · 2026-03-11T01:46:13Z

+SELECT * FROM t64 AS a NATURAL JOIN (SELECT * FROM t64) AS b;
+
+SET SESSION optimizer_switch= @save_optimizer_switch;
+DROP TABLE t64;


Please add your tests to derived_opt.test

mariadb-RexJohnston · 2026-03-11T02:02:42Z

+  uint max_parts= table->file->max_key_parts();
  uint i= 0;

  for ( ; i < count && key_count < keys; )


could we not just clip count at table->file->max_key_parts() ?

CLAassistant · 2026-03-11T13:47:20Z

All committers have signed the CLA.

mariadb-RexJohnston · 2026-03-11T21:03:26Z

    {
      keyuse->key= table->s->keys;
-      keyuse->keypart_map= (key_part_map) (1 << parts);     
+      keyuse->keypart_map= (key_part_map) 1 << parts;


please don't change the formatting on existing code.

mariadb-RexJohnston · 2026-03-11T21:03:48Z

-                               get_next_field_for_derived_key, 
+        if (table->add_tmp_key(table->s->keys, parts,
+                               get_next_field_for_derived_key,
                               (uchar *) &first_keyuse,


mariadb-RexJohnston · 2026-04-01T22:10:10Z

If i checkout your PR, undo your changes to sql_select.cc with
git restore -s c43bf13631f sql_select.cc
then run
mtr main.derived_opt
i would expect the test to fail. It doesn't here. These sorts of differences are usually due to differing storage engines.

As a general approach, it is better to correct data structures upstream than make the rest of the code deal with invalid data.
Here i would expect that the keyuse_array size, incremented beyond what is valid by add_key_part(), shouldn't be.

…AL JOIN on derived table When a NATURAL JOIN operates on a derived table (or view with derived_merge=off) having more than 64 columns, the optimizer's generate_derived_keys_for_table() would: 1) Overflow a 32-bit shift: (key_part_map)(1 << parts) when parts >= 32 2) Create a derived key with more parts than Bitmap<64>/key_part_map can hold 3) Crash on DBUG_ASSERT(prefix_size <= width) in Bitmap<64>::is_prefix(65) Fix: cap derived keys at sizeof(key_part_map)*8 (64) parts. Excess columns are excluded from the key (keyuse->key = MAX_KEY). Also fix the shift to cast before shifting: (key_part_map) 1 << parts.

bodyhedia44 · 2026-04-02T13:01:31Z

@mariadb-RexJohnston
Hi Rex, thanks for the review.

You're right. I reverted sql_select.cc and ran mtr main.derived_opt it passed. The reason is the test didn't specify ENGINE=InnoDB, so it used MyISAM (the default in MTR). The bug only triggers with InnoDB. I confirmed this by running a standalone test: same 65-column NATURAL JOIN query crashes with ENGINE=InnoDB but passes with MyISAM.

I fixed the test to use --source include/have_innodb.inc and ENGINE=InnoDB. Now mtr main.derived_opt fails without the fix and passes with it.

secondly, I moved the fix upstream to add_key_part(), where the keyuse_array grows. For materialized derived tables, it now stops adding hash-join KEYUSE entries once max_key_parts() is reached, so generate_derived_keys_for_table() never sees more entries than it can handle. I removed the downstream cap (the max_parts check and the while-loop marking excess entries) from generate_derived_keys_for_table().

gkodinov · 2026-04-23T11:27:18Z

@bodyhedia44 have you forgotten to push the changes you've described? I see no updated commit

bodyhedia44 · 2026-04-23T11:40:20Z

@gkodinov
No, I push it before my message, not after it
I am now waiting @mariadb-RexJohnston review on them

gkodinov added the External Contribution All PRs from entities outside of MariaDB Foundation, Corporation, Codership agreements. label Mar 9, 2026

gkodinov requested changes Mar 9, 2026

View reviewed changes

bodyhedia44 force-pushed the 10.6 branch from 4388cd6 to 8c48e02 Compare March 10, 2026 10:40

bodyhedia44 requested a review from gkodinov March 10, 2026 10:40

gkodinov approved these changes Mar 10, 2026

View reviewed changes

gkodinov requested a review from mariadb-RexJohnston March 10, 2026 13:26

gkodinov assigned mariadb-RexJohnston Mar 10, 2026

mariadb-RexJohnston requested changes Mar 11, 2026

View reviewed changes

bodyhedia44 force-pushed the 10.6 branch from 8c48e02 to ff1e21b Compare March 11, 2026 13:46

bodyhedia44 changed the base branch from 10.6 to 10.11 March 11, 2026 13:47

bodyhedia44 requested a review from mariadb-RexJohnston March 11, 2026 13:52

mariadb-RexJohnston reviewed Mar 11, 2026

View reviewed changes

bodyhedia44 force-pushed the 10.6 branch from ff1e21b to 16e196b Compare March 11, 2026 22:29

bodyhedia44 requested a review from mariadb-RexJohnston March 11, 2026 22:29

bodyhedia44 force-pushed the 10.6 branch from 16e196b to 2e900e1 Compare April 2, 2026 12:54

bodyhedia44 force-pushed the 10.6 branch from 2e900e1 to 6753c81 Compare April 2, 2026 12:58

Uh oh!

Conversation

bodyhedia44 commented Mar 8, 2026

Problem

Root Cause

How to Reproduce

Fix

Test

Uh oh!

gkodinov left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

bodyhedia44 commented Mar 10, 2026

Uh oh!

gkodinov left a comment

Choose a reason for hiding this comment

Uh oh!

mariadb-RexJohnston left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

CLAassistant commented Mar 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mariadb-RexJohnston commented Apr 1, 2026

Uh oh!

bodyhedia44 commented Apr 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gkodinov commented Apr 23, 2026

Uh oh!

bodyhedia44 commented Apr 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

4 participants

CLAassistant commented Mar 11, 2026 •

edited

Loading

bodyhedia44 commented Apr 2, 2026 •

edited

Loading

bodyhedia44 commented Apr 23, 2026 •

edited

Loading