Skip to content

RumbleDB 3.1.0 "Pistachio Tree"

Latest

Choose a tag to compare

@ghislainfourny ghislainfourny released this 09 Nov 12:08
· 39 commits to master since this release
64ab60c

RumbleDB 3.1.0 Pistachio Tree passes 100% of applicable tests of the qt3tests test suite for XQuery 3.1. Since JSONiq is based on XQuery 3.1, this also aligns JSONiq with the standard (in particular full support of all builtin types, all builtin functions, serialization, all expressions, etc).

It is the result the work of amazing students at ETH Zurich.

Support for Spark 4.0, 4.1, 4.2, and 4.3.
We also added jars for Spark 3.5 with both Scala 2.12 and 2.13, as some users may still be using it and we want them to also benefit from the improved language support.

Amazon EMR 8.x supports Spark 4.x and is compatible with RumbleDB: we tested it successfully. EMR 7 should be used with the jars for Spark 3.5 and Scala 2.12.

Java 17 or 21 is required for Spark 4.0, 4.1, and 4.2.
Java 17 is required for Spark 3.5.

Contributors (Ghislain Fourny's students at ETH): Stefan Irimescu, Renato Marroquin, Rodrigo Bruno, Falko Noé, Ioana Stefan, Andrea Rinaldi, Stevan Mihajlovic, Mario Arduini, Can Berker Çıkış, Elwin Stephan, David Dao, Zirun Wang, Ingo Müller, Dan-Ovidiu Graur, Thomas Zhou, Olivier Goerens, Alexandru Meterez, Pierre Motard, Remo Röthlisberger, Dominik Bruggisser, David Loughlin, David Buzatu, Marco Schöb, Maciej Byczko, Abishek Ramdas, Matteo Agnoletto, Dwij Dixit, Omar Hammoud, Henrik Pätzold, Jimmy Cai, Athena Wang.

Test suite status

31387 in total pass.

Inapplicable tests include: XPath, XSLT (not supported), past versions of XQuery (1.0, 3.0), inapplicable Unicode versions, and tests checking the behavior of engines that do not support optional features, because we do support all optional features.
Currently intentionally skipped tests:

  • XML Schema 1.1 tests (because we are waiting until the Xerces library dependency's support of XML Schema 1.1 is pushed to Maven, which is out of our control)
  • HTTP Remote (because a test uses timeanddate.com, which now has captcha preventing programmatic queries)

New features and capabilities

  • Added dynamic XQuery module loading with fn:load-xquery-module.
  • Added collections containing a mix of XML, JSON, and JSON Lines files.
  • Added support for optional integer numbering sequences.
  • Added collation support to more string functions.
  • Added collation support to deep-equal, index-of, min, and max.
  • Added configurable XML Schema datatype and regular-expression semantics.
  • Added schema validation when converting JSON to XML.
  • Added escaping options and fallback callbacks to json-to-xml.
  • Added typed XML element and attribute tests in paths.
  • Added support for dotted names in JSONiq XML constructors.
  • Added support for encoding declarations in XQuery source files.
  • Allowed window clauses to process distributed input.
  • Added ways for tools to list built-in types and schema-declared names.

Query checking and type inference

  • Improved type inference for JSONiq objects.
  • Improved handling of values that can have several possible types.
  • Improved tracking of whether expressions return zero, one, or several values.
  • Added initial schema-aware static typing.
  • Used XML schemas to infer child and attribute path results.
  • Used XML schemas to infer descendant path results.
  • Improved detection of invalid XML paths before execution.
  • Improved inference of XML element names and namespaces.
  • Propagated the context item’s type into path expressions.
  • Detected invalid data() calls on mixed-content XML elements earlier.
  • Made conversion of nodes to values more consistent.
  • Preserved sequence sizes correctly when checking comparisons.
  • Enforced declared types in for clauses and quantified expressions.
  • Applied function return-value conversions consistently.
  • Fixed type checking for complete sequences in typeswitch.
  • Corrected return types for XML parsing functions.
  • Corrected the inferred return type of random-generator next.
  • Improved compatibility between dateTimeStamp subtypes and DataFrames.
  • Let developer tools recover from translation failures in incomplete queries.

Modules, variables, and functions

  • Fixed cyclic module dependencies that could cause stack overflows.
  • Detected a module’s language from its version declaration.
  • Fixed dependencies between global variable initializers.
  • Avoided falsely reporting valid global initializers as circular.
  • Reported genuine circular variable dependencies correctly.
  • Made named functions available while global variables are initialized.
  • Preserved the defining module’s context for imported functions.
  • Shared the initial context item correctly across modules.
  • Checked that context-item declarations across modules agree.
  • Rejected duplicate context-item declarations in JSONiq.
  • Rejected conflicting imported variable declarations.
  • Respected private declarations across module imports.
  • Preserved imported modules’ locations and namespace scopes.
  • Reported the correct error when an imported resource is not a library module.
  • Preserved captured variables through higher-order function calls.
  • Preserved the proper context when calling function references.
  • Fixed partial-function application edge cases.
  • Made tail-call handling work across different kinds of function calls.
  • Made exit leave the current function correctly.
  • Preserved original initialization errors during Spark serialization.

XML parsing, construction, and navigation

  • Preserved whitespace when parsing XML documents and fragments.
  • Fixed handling of whitespace in empty CDATA sections.
  • Added support for text declarations in XML fragments.
  • Preserved inherited namespaces when serializing selected XML nodes.
  • Fixed namespace scope in XML constructors.
  • Fixed namespace handling in computed element names.
  • Accepted expanded QName strings as computed element names.
  • Preserved lexical QName spelling where required.
  • Fixed XML axis ordering and context handling.
  • Assigned correct document positions to nested constructed nodes.
  • Rejected non-node values in intermediate path steps correctly.
  • Reported the correct error for axis steps on non-node values.
  • Corrected argument checks in root and namespace functions.
  • Fixed fn:root signatures and node handling.
  • Treated empty enclosed XML expressions as empty sequences.
  • Normalized atomic content correctly inside XML constructors.
  • Rejected function values inside XML constructor content.
  • Fixed processing-instruction target whitespace and name checks.
  • Tightened checks for malformed XML tag separators and names.
  • Preserved DTD ID and IDREF metadata.
  • Checked document references against IDs, including xml:id.
  • Improved DTD and XML infoset handling.
  • Fixed XML element subtype checks.

Schemas and resource locations

  • Preserved base URIs through XML parsing and schema validation.
  • Fixed document URI handling.
  • Resolved empty URI literals against the complete static base URI.
  • Rejected unsuitable base URIs in resolve-uri.
  • Honored configured resource URI mappings during execution.
  • Inferred XML document root names before schema validation.
  • Combined schema location hints within an XSD 1.0 namespace.
  • Preserved namespace prefixes in schema type definitions.
  • Included the standard XPath functions schema for namespace imports.

JSON, maps, arrays, and collections

  • Improved diagnostics for malformed JSON.
  • Improved diagnostics for malformed XML.
  • Fixed JSON escaping during serialization.
  • Accepted XML Schema scalar spellings in xml-to-json.
  • Fixed maps being called as functions.
  • Made equivalent date, duration, binary, and QName map keys hash consistently.
  • Reworked map storage to use a persistent hash trie.
  • Required integer keys for array lookups.
  • Rejected nonnumeric array lookup keys correctly.
  • Converted node and array lookup keys to values consistently.
  • Checked very large array indexes before narrowing them to machine integers.
  • Fixed collection-handling edge cases.
  • Added regression coverage for null values in directory collections.
  • Allowed collection URI listing without parsing file contents.

Strings and regular expressions

  • Made string(()) return one empty string.
  • Allowed atomic values in XQuery string-join.
  • Fixed substring handling for special positions.
  • Fixed substring handling of Unicode code points.
  • Improved Unicode normalization.
  • Normalized Unicode normalization-form names consistently.
  • Fixed XPath regular-expression character-class subtraction.
  • Fixed XML-name escapes in regular expressions.
  • Fixed Unicode digit and word character classes.
  • Applied XPath whitespace rules for the regex x flag.
  • Fixed case-insensitive regular-expression backreferences.
  • Made end anchors match only at the actual end of input.
  • Normalized source line endings before parsing XQuery.

Numbers, dates, and comparisons

  • Preserved xs:integer types while using compact internal storage.
  • Improved handling of large integer literals and sequence positions.
  • Added checked integer conversions.
  • Printed floating-point values with short, round-trippable decimals.
  • Fixed rounding with extremely large precision arguments.
  • Improved decimal precision when dividing durations.
  • Fixed conversion to day-time durations.
  • Fixed extraction of the day when casting dates to gDay.
  • Fixed IETF date parsing around midnight and timezones.
  • Improved recognition of timezone identifiers and names.
  • Fixed era formatting in dates.
  • Preserved conventional casing in the default language tag.
  • Preserved string subtypes in min and max.
  • Validated sum inputs before aggregation.
  • Corrected distributed min, max, and sum result types.
  • Fixed boolean conversion of untyped atomic values.
  • Fixed boolean evaluation in local predicates.
  • Converted switch operands to values consistently.
  • Fixed checks on the number of values used by switch.

Output and error reporting

  • Fixed several XQuery serialization edge cases.
  • Accounted for item separators when validating XML output.
  • Resolved serialization parameter names using the default namespace.
  • Validated output declarations in query prologs.
  • Improved errors for invalid casts.
  • Distinguished malformed QNames from namespace-resolution errors.
  • Reported unsupported datatypes with the correct error code.
  • Reported duplicate default namespaces correctly.
  • Checked duplicate decimal-format declarations.
  • Checked invalid decimal-format zero digits.
  • Validated function declaration namespaces.
  • Improved source locations for variable and grouping bindings.
  • Fixed the iterator-tree display so it shows the complete tree.

Distributed execution and performance

  • Fixed a join optimization that could leave required input fields out.
  • Fixed conversion of empty distributed item collections to DataFrames.
  • Fixed DataFrame object accumulation losing fields.
  • Kept already-distributed input distributed in parallelize.
  • Preserved input order in parallelize.
  • Validated partition counts in parallelize.
  • Fixed Spark serialization of XML schema annotations.
  • Shared Spark serialization registrations consistently.
  • Fixed class-loader handling for DataFrame columns.
  • Reduced overhead when calling user-defined functions directly.
  • Separated evaluation state from reusable execution plans.
  • Stopped creating a Spark session merely by constructing a Rumble instance.
  • Stopped registering the Delta catalog implicitly.

Command line, maintenance, and release changes

  • Fixed packaging of the bundled command-line JAR so service registrations are merged correctly.
  • Added guidance when the command line is launched without arguments.
  • Made --version print the version correctly.
  • Removed the in-repository documentation as part of release preparation.
  • Added more regression tests across parsing, typing, XML, and Spark.
  • Made XQuery conformance-test failures fail the automated checks.
  • Expanded automated composability checks.
  • Split command-line checks into output-format and runtime groups.
  • Added merge-queue support.
  • Simplified shared parsing and module-translation code.