v0.4.0

Highlights

• Vector runtime only: For faster performance, all queries now run in the vector runtime, and many functions and operators gained native vector implementations.
• Unified binary format: The row-based BSUP and column-based CSUP formats are merged into a single BSUP format that can hold both. Files and databases from earlier releases are not readable by this release (see below).
• Persistent database revived: Tables (formerly pools) can now be created and loaded with SQL CREATE TABLE and INSERT INTO, including typed tables that check every value loaded into them, and CREATE SUPER TABLE makes a table of any SuperSQL type, such as a sum type computed from the data. attach/detach select a database, tables can be unsorted, and queries against them are type-checked.
• Option types and none values replace error("missing"), with new ?? and ?. operators and ok()/is_ok() functions.
• Stronger type checking: Static type checking of inputs by default where possible, and strict checking of unions.
• Formats from file extensions: The format of a local file is now determined by its extension rather than auto-detected (-i auto still requests auto-detection). Likewise, the output format now follows the extension given to -o.

BREAKING CHANGES

Users of earlier releases should expect to regenerate any saved binary files and databases from their original sources, and to update some queries. If you have valuable saved data you can't easily regenerate, please speak up on community Slack or open an issue for assistance.

• The binary formats have been unified. The column-oriented format formerly called CSUP is now called BSUP, and the row-oriented format formerly called BSUP lives on as row frames inside the new BSUP, which can mix row and column frames in one file. Binary output is now column-oriented by default (-rows writes row frames). super cannot read BSUP or CSUP files written by earlier releases, and the csup format name and the -bsup.* flags are gone (#6844, #6994, #7050, #7145, #7181, #7289, #7361, #7389, #7394, #7399, #7434, #7573).
• Databases created by earlier releases cannot be opened. Data objects and metadata are now stored in the new BSUP format, and object and commit IDs are now UUIDv7 rather than KSUID (#7182, #7401, #7424, #7425, #7427, #7430, #7573).
• Database pools are now called tables in commands, flags, and queries (#7472).
• super db auth and super db login are replaced by super db attach and super db detach, and the SUPER_DB environment variable has been removed. Commands use the attached database unless -attach is given (#7467, #7511, #7519).
• The super db vector commands have been removed (#7395).
• The vector runtime is now the only runtime. The -sam and -vam flags and the SUPER_RUNTIME environment variable have been removed (#7072).
• The format of a local file is now determined from its extension (including .gz-compressed files) or from -i. Auto-detection is now the default only for standard input and for HTTP, HTTPS, and S3 URLs without a recognized extension, so a local file with a missing or unrecognized extension now requires -i, which can be -i auto to request auto-detection (or format auto on a from operator) (#7501, #7514, #7520, #7531).
• The -samplesize flag has been removed. BSUP, Parquet, Arrow, and SUP files are now statically type-checked by default, -static extends this to all inputs, and -dynamic turns it off. Type checking is now strict about unions and about fields that may be missing, so a query over heterogeneous data may need ok() where it previously ran. For example, values y over a statically typed input is now an error ("field may be missing (consider ok())") if y is absent from some values (#7292, #7312, #7528, #7539, #7541, #7564).
• Referencing a missing field now produces a "no such field" error rather than error("missing"). The has() and missing() functions are replaced by ok() and is_ok(), and the quiet() function and special handling of error("quiet") have been removed (#7232, #7288, #7290, #7297, #7502, #7508).
• SUP syntax changes: Named types are now declared with type statements (e.g., type port=uint16) instead of inline (e.g., 80::(port=uint16)), sets and maps are written set[...] and map{...} instead of |[...]| and |{...}|, and a none value is written none instead of _. The -persist flag has been removed since named types can no longer be redefined (#6790, #6821, #6828, #7258).
• Unions are now flat: An anonymous union can no longer contain another anonymous union (a named union still can) (#6809).
• The non-aggregate max() and min() functions are renamed greatest() and least(), as in other SQL implementations (#6963, #6965, #6966).
• The type fusion operators and functions have been renamed: blend is now fuse, the former fuse (which produces fusion types) is now superfuse, unblend() is now defuse(), and the former defuse() is now superdefuse(). Because the names were swapped, a query that uses fuse or defuse() still runs but now does what blend or unblend() did. This restores the meaning fuse had before v0.3.0 (#7563).
• The time type is renamed timestamp, following Spark SQL and BigQuery, and is still a nanosecond-precision instant in UTC. The name time now refers to the SQL time-of-day type, which isn't implemented yet, so a cast such as x::time now produces an error (#7510).
• The JSUP format has been removed. The Python client has moved to superdb-python and now uses Arrow IPC to receive query results (#6838, #6858, #7028).
• The table output format has been removed (#7440).
• flatten() now returns {path,value} records rather than {key,value}, and it now descends into arrays, sets, and maps, using integer positions and map keys as path elements. unflatten() takes the new form. Similarly, fields() is replaced by paths(), which returns the same paths as flatten(), e.g., paths({a:[1,2]}) is [["a",0],["a",1]] where fields() returned [["a"]] (#7567, #7571).
• The grep() function and search expressions no longer match field names (#7343).
• The S3 endpoint environment variable is now AWS_ENDPOINT_URL_S3 instead of AWS_S3_ENDPOINT, following AWS convention (#7307).
• The -fusemem flag has been removed (#7078).

Added

• Option types (e.g., option(int64)) and none values, along with the ?? none-coalescing and ?. optional-chaining operators (#6881, #6894, #6900, #6903, #7249, #7254, #7269, #7279, #7311, #7327, #7358, #7380, #7478, #7491, #7505, #7522, #7552).
• Recursive types, e.g., type JSON=string|float64|bool|null|[JSON]|map{string:JSON} (#6841, #7458).
• Native uuid type (#7512).
• Reserved types for SQL compatibility, such as date, time, interval, decimal, int128, geometry, and geography, which give these types a place in the type system and binary format ahead of their implementation. Casting to one of them currently produces a "type not yet implemented" error (#7510).
• The any type, for use in type expressions; a value cast to any becomes a fusion(all) value (#6827, #6978, #7063, #7089, #7554).
• A type declaration can bind a name to a type computed by a query, e.g., type T = (from a.sup | fuse(this)) (#7558).
• The dot operator and string indexing can navigate into type values, e.g., <{a:{b:string}}>.a.b and <{a:{b:string}}>["a"]["b"] are <string>, including through named, option, and fusion types, into array, set, and map element types, and across the members of a union (#7565, #7570).
• New keys() and values() functions return the field names and values of a record, or the keys and values of a map (#7571).
• New optionize() function, which turns missing|T unions into option(T), e.g., to make the type from fuse() have optional fields (#7556).
• New sum_type() function, which forms a union of named types from a map of names to types (#7557).
• New array_agg aggregate function, including support for distinct (#7165, #7225).
• Method-style call syntax, where the value before the dot becomes the first argument, e.g., s.upper() is equivalent to upper(s), s.split(",") to split(s, ","), and calls can be chained, as in s.trim().upper() (#7281).
• Command-line arguments after -- are passed to the query as constants (#7364).
• The output format is now inferred from the file extension given to -o when -f isn't specified (#7331).
• super compile now accepts -i (#7538).
• A from operator that reads files can now have a parent operator, scanning its files once per input value (#7317).
• from with an expression argument (e.g., a URL built with an f-string) now works in super db queries (#7294).
• Database tables can now be created without a sort key (the new default) (#7438).
• SQL CREATE TABLE, DROP TABLE, and INSERT INTO statements for database tables, with column types, NOT NULL, and ORDER BY, plus CREATE SUPER TABLE for a table of any SuperSQL type (e.g., a sum type from sum_type()) and super db create -type. Values inserted or loaded into a typed table are cast to its type, and nothing is committed if any value can't be cast. A query can now also be a script of semicolon-separated statements that share declarations, with a status record output after each statement (#7562).
• New -objectcap, -framecap, and -inputcap flags control the size of database storage objects, BSUP frames, and batches of input (#7437).
• Queries against database tables are now type-checked, and :objects output now includes each object's type (#7513).
• Named types in superdefuse() (#6767).
• Enum types in upcast() and downcast() (#6853, #6864).
• The Parquet reader now supports the map type (#7098).

Changed

• Fusion values are now automatically defused in all output formats except BSUP. Use the new -fusion flag to see them in SUP output. The superfuse operator now defuses any fusion values in its input (#7036, #7037).
• Type fusion now preserves named types, produces simpler fused types, and computes the same fused type regardless of input order. A field that is absent from some inputs now has a union type with missing (e.g., int64|missing) rather than becoming an optional field, and optionize() converts such a type to one with optional fields. An empty array, set, or map now has element type none rather than null (#6760, #6766, #6777, #6798, #6819, #7374, #7375, #7491, #7553).
• In SQL, a reference to a missing field now evaluates to null (#7295).
• collect() now returns an empty array when there's nothing to collect, and collect(), array_agg(), and union() now keep null values, and collect() and array_agg() preserve input order (#7165, #7166, #7167, #7392, #7405).
• join no longer treats two null keys as matching, per SQL semantics (#7238).
• The JSON reader now reports an error for duplicate field names (the last value was previously kept), and its parse errors now include line numbers (#6868, #7111, #7112).
• JSON output now omits fields whose value is none (#6780, #7322).
• typeof() now returns the subtype of a fusion value (#7147, #7158).
• Casting a record to a union of record types now converts it to the member it best matches, converting field values as needed, where previously it had to match a member exactly. Casting a missing value or field to an option type now gives none (#7555).
• under() of a type value now unwraps one level of an array, set, option, union, or map type, e.g., under(<[int64]>) is <int64> where it was previously <[int64]> (#7572).
• A format argument on a from operator still takes precedence, but -i now takes precedence over a file's extension (#7344).
• File paths in output and messages are now shown without a file:// prefix (#7515).
• A clearer error message when a from operator has a parent but its argument isn't an expression (#7082).
• The GitHub organization and Go module path are now github.com/superdb/super (#7377).
• Homebrew installation is now brew install super from the core Homebrew casks instead of our own tap (#6796, #6835).
• Windows release binaries are now signed (#6761).
• Go 1.27 or later is now required to build super (#6778, #7231, #7318).

Fixed

• Fix many problems with fusion values in operators, functions, and expressions (e.g., put, rename, unnest, collect(), typeof(), casts, comparisons, and search expressions) (#6941, #6990, #7001, #7019, #7031, #7032, #7034, #7043, #7044, #7045, #7051, #7133, #7134, #7136, #7137, #7138, #7142, #7143, #7144, #7146, #7150, #7151, #7153, #7154, #7155, #7156, #7159, #7171, #7177, #7179, #7180, #7183, #7187, #7188, #7189, #7191, #7193, #7195, #7198, #7199, #7200, #7201, #7202, #7207, #7219, #7221, #7222, #7381).
• Fix many problems in type fusion and in the defuse(), superdefuse(), upcast(), downcast(), and infer functions and operators (#6768, #6770, #6771, #6774, #6786, #6815, #6845, #6848, #6849, #6854, #6869, #6871, #6882, #6904, #6930, #6932, #6937, #6979, #6998, #6999, #7000, #7005, #7006, #7008, #7011, #7021, #7030, #7129, #7162).
• Fix many functions, operators, and expressions that mishandled union-typed values (#6836, #6837, #6888, #6889, #6901, #6902, #6905, #6911, #6912, #6913, #6914, #6922, #6934, #7022, #7026, #7038).
• Fix incorrect results (missing or extra values) when filtering columnar files with union-typed fields, and a related panic (#7100, #7118, #7119, #7124).
• Fix incorrect or inconsistent aggregation results on columnar (formerly CSUP) input, and on aggregates with distinct (#6782, #6793, #6843).
• Fix incorrect results from a switch containing an aggregate (#6953).
• Fix deadlocks in some queries, such as from /dev/stdin | top 1 a and fork combined with sort (#7455, #7456).
• Fix a columnar (formerly CSUP) writer bug that could misalign values in union columns (#6784, #6908).
• Fix super db manage so it compacts data again (#7523).
• Fix a panic in super db ls on a database with no tables (#7485).
• Fix Parquet reading of null-typed values, writing of optional fields, and projection of lists, maps, and empty projections (#7097, #7104, #7169, #7241, #7460).
• Fix Arrow writer handling of named types and of a union of null and a nested union (#6772, #7012).
• Fix SQL queries that returned errors or too few columns on some BSUP input (e.g., select * with named types, or nested field references) (#7545, #7548).
• Fix none appearing in CSV output for a missing field, and spurious "wrong number of fields" errors reading non-comma-delimited CSV files (#7239, #7483).
• Fix reading /dev/stdin by name on macOS (#7049, #7067).
• Fix date_part() and strftime() handling of unions and nulls (#6884, #6885).
• Fix <= and >= comparisons of bool values (#6976).
• Fix collect_map() handling of a named type inside a union (#6967).
• Fix panics and lost named types in count() grouped by a string (#7080, #7084).
• Fix a precision bug when casting to float16 or float32 (#7160).
• Fix the Zeek reader treating a time or interval with more than nine fractional digits as zero (#7496).
• Fix duplicate fields in record expressions with spreads, and a panic on {...r,a:none} (#7530).
• Fix duplicate error messages for cut field assignments without a right-hand side (#7087).
• Fix group by ordinals when a column alias matches an input column name, and a panic when used with select * (#7326, #7435).
• Fix a panic on SQL select with conflicting column aliases (e.g., select 1 as a, 2 as a, 3 as a_1) (#7250).
• Fix select * failing with "cannot be used for dynamic inputs" on some BSUP input (#7463).
• Fix a panic in unnest {a,b} when b is an empty array (#7208).
• Fix empty field names produced by lambdas in record expressions (#7266).
• Fix a function call that failed depending on the order of references to a const (#7321).
• Fix named types on constant values (#7547).
• Fix SUP output of a named type whose underlying type is another named type (#6811).
• Fix panics: in collect(), any(), or superfuse() downstream of sort; after a from with an expression argument; when outputting an error-valued constant; and when evaluating a recursive function concurrently (#7323, #7406, #7416, #7417).

Performance

• A new, faster JSON reader (#6764, #6789, #6817, #6818, #6847, #6867, #6935, #6981, #7113, #7157, #7161, #7194, #7226, #7462).
• Native vector implementations of defuse(), superdefuse(), upcast(), downcast(), collect(), greatest(), least(), fuse, and superfuse, which previously used the sequential runtime (#6773, #6820, #6829, #6856, #6863, #7035).
• Faster hash joins (#7252, #7253, #7255).
• Better parallelism on sorted database tables for queries that use values, which includes every SQL SELECT (#7559).
• Faster unnest (#7246).
• Faster queries on records with nullable fields (#7340).
• Faster Parquet reading, including when it's type-checked (#7115, #7277, #7345).
• BSUP and Parquet inputs stay in vector form throughout a query, and BSUP files now record their fused type so type checking doesn't need to scan the data (#7090, #7094, #7105).
• Filter and search pushdown into BSUP files (#7450, #7453, #7550).
• Faster queries with subqueries, including caching of uncorrelated subqueries (#7248, #7351).
• Faster database ingest when the sort key is a primitive type (#7429).

Security

• Upgrade github.com/golang-jwt/jwt/v4 to v4.5.2 (CVE-2024-51744, CVE-2025-30204) (#7099).
• Upgrade github.com/apache/arrow-go/v18 to v18.8.0, which pulls in a fixed github.com/apache/thrift (CVE-2026-43871) (#7309).