[pull] main from MaterializeInc:main#1040
Merged
Merged
Conversation
`DROP VIEW v1, v2` (and analogously for other item types) failed with "still depended upon by ..." when v2 depended on v1, even though v2 was explicitly named in the same statement. `plan_drop_objects` resolved names one at a time and ran the non-CASCADE dependency check inside `plan_drop_item` before knowing the full set of objects being dropped, so a dependent that was itself being dropped still blocked the drop. Split `plan_drop_item` into a name-resolution step (`plan_drop_item_name`) and a dependency-check helper (`ensure_no_blocking_dependents`) that takes a set of co-dropped item ids. `plan_drop_objects` now resolves all names first, then runs the dependency check with the full set of explicitly-named items, skipping dependents that are themselves being dropped. The two CREATE OR REPLACE callers of `plan_drop_item` are unaffected: they pass an empty co-dropped set and preserve the prior behavior. This matches PostgreSQL, which lets you drop an object together with its dependents in a single statement without CASCADE. Fixes SQL-151
### Motivation Closes https://linear.app/materializeinc/issue/SQL-119/create-documentation-for-oidc-auth ### Description Documentation for the new Oidc feature! Most of this was generated from this prompt https://www.notion.so/materialize/SSO-Documentation-Outline-2e013f48d37b806591b5c3b40304d028 with a few small edits The behavior still relies on the following PRs to be merged: -`autoprovision` column in `mz_roles` #35325 - Console OIDC login flow https://github.com/MaterializeInc/materialize/pull/35440/files - Console OIDC generate token leedqin#1 TODOs from my side: - Test Microsoft Entra flows. I've already tested Okta+Ory Hydra. Qindeel has tested Auth0. - Test OIDC CLI integrations (e.g. `oauth2c`)
We also need to trigger the self-managed deploy again. Edit: Doing that now: https://buildkite.com/materialize/publish-helm-charts/builds/168
The `complete-release` CI job is failing with:
```
File ".../materialize/release/mark_release_complete.py", line 37, in main
frontmatter.dump(metadata, f, sort_keys=False)
TypeError: a bytes-like object is required, not 'str'
```
`python-frontmatter` was bumped 1.1.0 → 1.3.0 in #36710 (2026-05-25). In
1.1.0, `frontmatter.dump()` encoded the content to bytes
(`.encode(encoding)`) before writing, so the doc file was opened in
binary mode (`"wb"`). In 1.3.0, `dump()` writes a `str` directly and
expects a text-mode file, so writing to the `"wb"` handle now raises
`TypeError`.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Remove these sections if your commit already has a good description! ### Motivation Now that we can provide a path to restrict roles using mcp from accessing public catalog objects we should enable this by default. cc @bobbyiliev ### Description What does this PR actually do? Focus on the approach and any non-obvious decisions. The diff shows the code --- use this space to explain what the diff *can't* tell a reviewer. ### Verification How do you know this change is correct? Describe new or existing automated tests, or manual steps you took. --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Test run: https://buildkite.com/materialize/nightly/builds/15335 Should currently fail because of MaterializeInc/database-issues#10099
…ifiable (#36775) Allow environment superusers to configure OIDC group-sync settings via ALTER SYSTEM SET without needing mz_system access. This is appropriate since these control how a customer's own OIDC claims map to roles. Fixes SQL-321
We saw an issue where a source failed because the kafka low watermarks were beyond the `resume_upper` for that source, indicating there was data compacted away that we would never get back. However, because the source emitted an error at the timestamp of the snapshot, and then exited, implicitly dropping its capabilities, the remap operator believed that all messages up until the input upper (high watermarks of the kafka partitions) had been completed, and advanced the `resume_uppers` to the high watermarks. This is normal procedure when emitting definite errors that indicate the source is irreparably broken. However, at this point the kafka replication operator emitted a `HealthStatusUpdate` with `halting = true`. This meant that the healthcheck operator would then issue a suspend and restart command to the storage controller. When restarting, because the resume uppers had advanced, the source was able to continue reading from the previous frontier, and the status appeared healthy until querying the source table, when you would get the low watermark error. Instead, we should emit a `HealthStatusUpdate::stalled` which will tell the healthcheck operator not to issue a restart command, and the source will stay visibly stalled rather than appear healthy.
Fixes [CNS-80. ](https://linear.app/materializeinc/issue/CNS-80/cluster-sizes-are-out-of-order)`ORDER BY credits_per_hour` to sort cluster replica sizes by cost ascending. ### Verification <img width="3018" height="1558" alt="image" src="https://github.com/user-attachments/assets/a70cc06b-c293-425c-a8dd-a6c6aa93a9cd" />
### Motivation Fixes database-issues#11268: `generate_subscripts` was not respecting custom lower bounds on arrays. When an array was created with a custom lower bound (e.g., via `array_fill` with a lower bound parameter), `generate_subscripts` would incorrectly start from 1 instead of the array's actual lower bound. ### Description The issue was in `generate_subscripts_array` in `src/expr/src/relation/func.rs`. The function was calling `generate_series` with only the `lower_bound` and `length` parameters, but `generate_series` expects the upper bound as the second argument, not the length. The fix: 1. Extract `lower_bound` and `length` from the dimension into separate variables 2. Calculate the actual `upper_bound` as `lower_bound + length - 1` (since subscripts are inclusive on both ends) 3. Add overflow checking when computing the upper bound 4. Pass the correct `lower_bound` and `upper_bound` to `generate_series` This ensures that `generate_subscripts` correctly generates subscripts from the array's actual lower bound through its upper bound, inclusive. ### Verification Added two regression tests in `test/sqllogictest/table_func.slt`: 1. Single-dimensional array with custom lower bound 5 and length 3, expecting subscripts 5, 6, 7 2. Multi-dimensional array with custom lower bounds (10, 20) and lengths (2, 3), expecting subscripts 20, 21, 22 for dimension 2 Both tests verify the fix works correctly for arrays created with `array_fill` using custom lower bounds. https://claude.ai/code/session_01C2n4d7MhwFRXVezyxs3DgQ --------- Co-authored-by: Claude <noreply@anthropic.com>
### Motivation Closes MaterializeInc/database-issues#11278 Both `array_lower()` and `array_upper()` ignored the array's custom lower bound: - `array_lower()` always returned `1`, regardless of the array's actual lower bound. - `array_upper()` returned the dimension's *length* rather than its upper bound (`lower_bound + length - 1`). For arrays with the default lower bound of `1` these happen to be correct (since `upper_bound == length`), so the bug only surfaced for arrays with a non-default lower bound, e.g. those built by `array_fill()`: ``` SELECT array_lower(array_fill(0, ARRAY[3], ARRAY[5]), 1); -- returned 1, should be 5 SELECT array_upper(array_fill(0, ARRAY[3], ARRAY[5]), 1); -- returned 3, should be 7 ``` PostgreSQL returns `5` and `7` respectively. ### Description - **`array_lower()`**: now returns the dimension's actual lower bound via `dim.dimension_bounds()` instead of hardcoding `1`. The return type changed from `Option<i32>` to `Result<Option<i32>, EvalError>` (matching `array_upper()`) so a bound that does not fit in `i32` is reported as an error. - **`array_upper()`**: now returns the dimension's upper bound via `dim.dimension_bounds()` instead of `dim.length`. - Reduced the `sqlfunc` attribute lists on both functions to just `is_infix_op = true`; `output_type`, `sqlname`, `propagates_nulls`, and `introduces_nulls` are inferred by the macro from the signature and matched the previous explicit values. ### Verification - Added a unit test (`array_lower_upper_respect_lower_bound`) in `src/expr/src/scalar/func.rs` covering default, custom positive, negative, and out-of-range dimensions for both functions. - Added test cases in `test/sqllogictest/array_fill.slt` for single-dimension custom (`5`), negative (`-3`), and multi-dimensional (`4`, `3`) lower bounds. https://claude.ai/code/session_01A4oEvWwfEb6yE8FYgPAfpQ --------- Co-authored-by: Claude <noreply@anthropic.com>
### Motivation Fixes CLU-75 / database-issues#11265: `SUM(float8)` returns incorrect results for large finite values when read from a table. ```sql CREATE TABLE float_boundary (f1 float8); INSERT INTO float_boundary VALUES (1.1e31), (-1.1e31); SELECT SUM(f1) FROM float_boundary; -- before: -5.960464477539063e-8 -- after: 0 ``` ### Root cause The accumulable float sum maps each input onto a fixed-point `i128` domain (scaled by `2^24`) and combines accumulators with **wrapping** arithmetic so the representation forms a commutative group, as required for incremental maintenance with retractions. The per-datum conversion, however, used a **saturating** `as i128` cast. Values whose scaled magnitude exceeds `i128`'s range (`|n| > ~1.01e31`) saturated to `i128::MAX` or `i128::MIN`. Because `i128`'s range is asymmetric (`MIN = -(MAX + 1)`), summing two cancelling values such as `1.1e31` and `-1.1e31` produced `MAX + MIN = -1`, surfacing as `-5.96e-8` (`= -1 / 2^24`) instead of `0`. (Without a table the query is constant-folded through the scalar path and was already correct.) ### Changes - Add `float_to_fixed_point`, which computes `trunc(n * 2^24)` reduced modulo `2^128` (wrapping) directly from the `f64` bit pattern. This makes the conversion a group homomorphism consistent with the wrapping accumulation, so large finite values whose mathematical sum is representable now yield the correct result. Working in the integer domain also avoids forming the intermediate `f64` product `n * FLOAT_SCALE`, which could itself overflow to infinity. - Replace the remaining raw `as` casts on the finalize path with `cast_lossy`, adding the `i128 -> f64` and `f64 -> f32` impls in `mz_ore::cast`. - Add Rust unit tests for `float_to_fixed_point` (agreement with the old cast in range, truncation toward zero, subnormals, cancellation of out-of-range values, full accumulate/finalize path) and a `float.slt` regression test mirroring the issue. Note: sums whose **mathematical result** is itself outside the representable fixed-point range (e.g. `SUM` of `1e31 + 1e31 = 2e31`) remain approximate — the fixed-point domain simply cannot represent them — and continue to log the existing overflow warning. This change makes the *representable* cases (including exact cancellation) correct. ### Tip `cargo test -p mz-compute --lib render::reduce::tests` https://claude.ai/code/session_01HXYJ6TAz8GawjxdC6TNr8R --- _Generated by [Claude Code](https://claude.ai/code/session_01HXYJ6TAz8GawjxdC6TNr8R)_ --------- Co-authored-by: Claude <noreply@anthropic.com>
### Motivation CLU-65: introduce an explicit pager so we can mark cold columnar data and pull it back on demand, choosing at runtime between an `MADV_COLD`-based swap backend and a file-backed scratch backend. Today Materialize spills to Linux swap and pays direct-reclaim latency on user threads. Design lives at `doc/developer/design/20260504_pager.md`. Depends on CLU-64 (`Column::Aligned` becomes `Vec<u64>`), which makes columnar buffers a natural caller. ### Description `mz_ore::pager` exposes a small handle-oriented API (`pageout`, `read_at_many`, `read_at`, `take`) over a global atomic backend selector. The handle's backend is fixed at `pageout` time so live config flips do not invalidate existing data. Two backends: * `Swap` keeps the input `Vec<u64>` chains resident and calls `MADV_COLD` on the page-aligned subrange. Reads `extend_from_slice` across chunk boundaries; `take` is zero-copy when the handle holds a single chunk. * `File` writes one named scratch file per handle in a per-process subdirectory under the configured scratch root. No file descriptor is retained across calls — `pageout` opens, vector-writes via `Write::write_vectored`, and closes; reads reopen and `pread`. A reaper sweeps stale subdirectories from crashed predecessors at config time. Documented as `doc/developer/design/20260504_pager.md`. ### Verification Unit + integration tests cover round-trip on both backends, scatter/gather correctness, drop-without-take reclaim, the swap fast path's pointer identity, and backend flip mid-process. Two micro-benches plus a merge-batcher example (`cargo run --release --features pager --example pager_merge`) for sizing the swap-vs-file trade-off under real memory pressure. Sanity numbers on a 32 GiB working set with 16 GiB cap, ext4 scratch: file finishes the merge pass in ~67 s vs swap's ~210 s, dominated by swap-in TLB shootdowns on the swap path. --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
10 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
See Commits and Changes for more details.
Created by
pull[bot] (v2.0.0-alpha.4)
Can you help keep this open source service alive? 💖 Please sponsor : )