Krishiv 0.2.0

Install

pip install "krishiv[pyarrow]"   # Python 3.11+, or 3.14t
cargo add krishiv@0.2.0           # Rust

Older x86-64 CPUs without AVX2: pip install krishiv-compat. Documentation: https://krishiv.ai/docs

The first release is the embedded core: the engine runs in your process, from Python or Rust.

Highlights

• Embedded Spark-compatible engine. krishiv.session() gives a PySpark-style session in your process, from Python or
  Rust, with no JVM; Spark-observable behaviour is checked against real Spark runs.
• Memory-bounded and fast. A vectorized Arrow executor that spills aggregation, joins, sorts and window frames to disk
  instead of failing, with runtime filters, Parquet row-group and dictionary pruning, and a cost-based join order.
• Broad SQL. Windows (ROWS/RANGE), GROUPING SETS, lateral joins, recursive CTEs, SQL scripting, CHAR/VARCHAR,
  collations, VARIANT, calendar intervals, GEOMETRY/GEOGRAPHY, PIVOT, TABLESAMPLE, SQL functions and IDENTIFIER(..).
• Python UDFs in process. @udf, scalar pandas_udf and arrow_udf run on a thread pool, in parallel on free-threaded 3.14t.
• See what happened. EXPLAIN [ANALYZE|COST|VERBOSE], a versioned JSON profile with per-scan I/O, a local web console,
  Prometheus metrics, OTLP spans and JSON logs, and errors that show the SQL span that raised them.
• Zero-copy interop with pyarrow, Polars and DuckDB through the Arrow PyCapsule interface.

Added

• In-process batch SQL and DataFrames over Arrow: krishiv.session() returns a PySpark-style SparkSession
  (sql, read.parquet, temp views, select/filter/groupBy/agg/orderBy/withColumn, collect, show,
  toArrow, toPandas, count).
• Zero-copy Arrow interop through the Arrow PyCapsule interface (__arrow_c_stream__): pyarrow, Polars and DuckDB
  read a DataFrame's result without copying buffers.
• EXPLAIN [EXTENDED | FORMATTED | CODEGEN | COST] with Spark's section layout, EXPLAIN ANALYZE (per-operator rows,
  time, peak memory and spill bytes, as text and as a versioned JSON profile) and EXPLAIN VERBOSE (optimizer steps).
• A vectorized, Arrow-native executor with memory-bounded spilling hash aggregation, hash joins (including grace
  partitioning), sorts and Parquet scans with runtime filters.
• krishiv sql, a command line SQL shell (script mode, table/CSV/JSON output, --timing).
• Python wheels for Python 3.11+ (abi3) and free-threaded 3.14t, for manylinux x86_64 and aarch64, musllinux
  x86_64 and macOS arm64, built for x86-64-v3 and neoverse-n1, with a krishiv-compat wheel for older x86-64 CPUs.
  The import checks the CPU and says which feature is missing instead of crashing.
• Spark error classes and the configuration registry, so errors read as Spark's do.
• Errors raised at a span of SQL show it: krishiv sql and the text of Python exceptions from spark.sql follow the
  message with an == SQL (line, position) == block, the source line and a caret under the span (a syntax error
  instead shows Spark's ParseException layout with its 0-based position).
• Spark SQL coverage: window functions with ROWS and RANGE frames (day-time and year-month interval bounds
  too), GROUPING SETS/CUBE/ROLLUP, lateral joins, generators in FROM, recursive CTEs, and the JSON, CSV and
  XML functions with Spark's option names (the legacy FastDateFormat patterns included).
• Reads over https:// and hf://, and credential refresh and per-location scopes for object stores.
• Telemetry: OTLP export of query spans and engine metrics behind the telemetry-otlp feature.
• Optimizer: eager aggregation below joins.
• Results checked against real Spark runs: division and remainder evaluate their operands in Spark's order, atan2 is
  bit-identical.
• Python UDFs: @udf, scalar pandas_udf and arrow_udf run in process on a thread pool.
• Collations (UTF8_LCASE, _RTRIM, COLLATE, collation-key comparison and operators, Parquet collation metadata).
• VARIANT in Parquet at any depth; GEOMETRY and GEOGRAPHY types with WKT and SRS functions; CALENDAR INTERVAL;
  CHAR(n)/VARCHAR(n); MSCK, partition commands and column statistics; SQL scripting with exit handlers.
• Deterministic row order (krishiv.deterministicOrder), krishiv.optimizer.joinCost=bytes, plan-time distinct-value
  samples of Parquet columns, and inferred join filters.
• A local web console (krishiv.ui.enabled), a bearer-token guard for /metrics, per-scan I/O fields in the profile, JSON
  logs, and a --share profile that drops identity and trace ids.

Known limits

• Temporary views over in-memory DataFrames and SQL expression strings (selectExpr, expr) are not supported
  yet; read data from files for views.
• Spark Connect, streaming, incremental views and distributed execution are not part of this release.
• The APIs are not stable.