Add docs on how to use Spark vortex datasource

README.md

@@ -6,7 +6,7 @@
 [![CodSpeed Badge](https://img.shields.io/endpoint?url=https://codspeed.io/badge.json)](https://codspeed.io/vortex-data/vortex)
 [![Crates.io](https://img.shields.io/crates/v/vortex.svg)](https://crates.io/crates/vortex)
 [![PyPI - Version](https://img.shields.io/pypi/v/vortex-data)](https://pypi.org/project/vortex-data/)
-[![Maven - Version](https://img.shields.io/maven-central/v/dev.vortex/vortex-spark)](https://central.sonatype.com/artifact/dev.vortex/vortex-spark)
+[![Maven - Version](https://img.shields.io/maven-central/v/dev.vortex/vortex-spark_2.13)](https://central.sonatype.com/artifact/dev.vortex/vortex-spark_2.13)
 [![codecov](https://codecov.io/github/vortex-data/vortex/graph/badge.svg)](https://codecov.io/github/vortex-data/vortex)
 [![Cite](https://img.shields.io/badge/cite-CITATION.cff-blue)](CITATION.cff)
 

docs/user-guide/spark.md

@@ -1,34 +1,71 @@
 # Spark
 
 Vortex provides a Spark DataSource V2 connector for reading and writing Vortex files. The
-connector is published to Maven Central as `dev.vortex:vortex-spark`.
+connector is published to Maven Central in two flavors:
 
-## Installation
+- `dev.vortex:vortex-spark_2.13` for Spark 4.x (Scala 2.13)
+- `dev.vortex:vortex-spark_2.12` for Spark 3.5.x (Scala 2.12)
 
-Add the dependency to your build. The connector is built against Spark 4.x with Scala 2.13.
+Use the `all` classifier JAR (e.g. `vortex-spark_2.13-0.78.0-all.jar`). It is self-contained:
+it bundles the Vortex JNI bindings, native libraries for Linux (x86_64 and aarch64) and macOS
+(aarch64), and relocates its Arrow, Guava, and Jackson dependencies to avoid classpath
+conflicts with Spark. The thin (unclassified) JAR does not work on its own because it
+references relocated classes that only ship in the `all` JAR.
+
+## Getting Vortex into Spark
+
+For `spark-shell`, `spark-submit`, or `pyspark`, pass the `all` JAR with `--jars`. Spark
+accepts either a local path or a URL, so you can point directly at Maven Central:
+
+```shell
+spark-shell --jars https://repo1.maven.org/maven2/dev/vortex/vortex-spark_2.13/0.78.0/vortex-spark_2.13-0.78.0-all.jar
+```
+
+Or equivalently when building a session programmatically, e.g. in PySpark:
+
+```python
+spark = (
+    SparkSession.builder
+    .config("spark.jars", "/path/to/vortex-spark_2.13-0.78.0-all.jar")
+    .getOrCreate()
+)
+```
+
+```{note}
+`--packages dev.vortex:vortex-spark_2.13:0.78.0` does not work: `--packages` cannot select
+the `all` classifier and resolves the thin JAR, which fails at runtime with
+`NoClassDefFoundError: dev/vortex/relocated/...`.
+```
+
+Once the JAR is on the classpath, the connector registers itself automatically under the
+format name `vortex` — no session configuration is required.
+
+## Installation as a Build Dependency
+
+To depend on the connector from a JVM project, add the `all` classifier to the dependency:
 
 ````{tab} Gradle (Kotlin)
 ```kotlin
-implementation("dev.vortex:vortex-spark:<version>")
+implementation("dev.vortex:vortex-spark_2.13:0.78.0:all")
 ```
 ````
 
 ````{tab} Maven
 ```xml
 <dependency>
     <groupId>dev.vortex</groupId>
-    <artifactId>vortex-spark</artifactId>
-    <version>${vortex.version}</version>
+    <artifactId>vortex-spark_2.13</artifactId>
+    <version>0.78.0</version>
+    <classifier>all</classifier>
 </dependency>
 ```
 ````
 
-The connector ships as a shadow JAR that relocates its Arrow, Guava, and Protobuf dependencies
-to avoid classpath conflicts with Spark.
-
 ## Reading Vortex Files
 
-Use the `vortex` format to read a single file or a directory of Vortex files:
+Paths may be local filesystem paths (`/path/to/data`) or URLs (`file:///path/to/data`,
+`s3://bucket/path/to/data`). Use the `vortex` format to read a single file or a directory of
+Vortex files:
 
 ```java
 Dataset<Row> df = spark.read()
@@ -65,6 +102,72 @@ Each Spark partition produces one output file named `part-{partitionId}-{taskId}
 The connector supports all standard Spark save modes: `Overwrite`, `Append`, `Ignore`, and
 `ErrorIfExists`.
 
+## Spark SQL
+
+The connector can also be used from pure SQL. To query existing Vortex files, register them
+as a temporary view:
+
+```sql
+CREATE TEMPORARY VIEW people
+USING vortex
+OPTIONS (path '/path/to/data');
+
+SELECT name, age FROM people WHERE age > 30;
+```
+
+Tables can be created with `USING vortex`, then written to and read back with plain SQL.
+With a `LOCATION` clause the table is external, backed by the files at that path; without
+one the table is managed, and Spark stores its data under the warehouse directory (and
+deletes it on `DROP TABLE`):
+
+```sql
+CREATE TABLE student (id INT, name STRING, age INT)
+USING vortex;
+
+INSERT INTO student VALUES (1, 'Alice', 20), (2, 'Bob', 21);
+
+SELECT * FROM student;
+```
+
+`CREATE TABLE ... AS SELECT` works the same way:
+
+```sql
+CREATE TABLE adults
+USING vortex
+AS SELECT * FROM people WHERE age >= 18;
+```
+
+```{note}
+On Spark 3.5, `CREATE TABLE ... USING vortex` additionally requires replacing the session
+catalog, because Spark 3.5's built-in catalog cannot read tables backed by a DataSource
+V2-only connector:
+
+    spark.sql.catalog.spark_catalog=dev.vortex.spark.VortexSessionCatalog
+
+The extension delegates everything to the built-in session catalog (including the Hive
+metastore, if configured) and only changes how `vortex` tables are resolved; tables of other
+providers are untouched. It is not needed on Spark 4, though setting it is harmless.
+```
+
+## Direct File Queries
+
+Spark's built-in ``SELECT * FROM format.`path` `` syntax only works for built-in file
+formats, so the connector ships a path-based catalog that provides the equivalent for
+Vortex. Register it in the session configuration under the name `vortex`:
+
+```shell
+spark-sql --conf spark.sql.catalog.vortex=dev.vortex.spark.VortexCatalog
+```
+
+Then query a Vortex file, or a directory of Vortex files, directly by path — no view or
+table required:
+
+```sql
+SELECT * FROM vortex.`/path/to/data`;
+
+INSERT INTO vortex.`/path/to/data` VALUES (1, 'Alice', 20);
+```
+
 ## Supported Types
 
 | Spark Type         | Vortex Type                            |

java/README.md

@@ -3,7 +3,9 @@
 We provide two interfaces for working with Vortex from Java:
 
 - `vortex-java` - a low-level interface JNI for working with Vortex files and arrays on cloud and local storage
-- `vortex-spark` - A Spark connector for working with datasets of Vortex files
+- `vortex-spark` - A Spark connector for working with datasets of Vortex files. See
+  [vortex-spark/README.md](vortex-spark/README.md) for how to load the connector into Spark
+  and query Vortex files from the DataFrame API or Spark SQL.
 
 ## Publishing
 

java/vortex-spark/README.md

@@ -0,0 +1,115 @@
+# vortex-spark
+
+A Spark DataSource V2 connector for reading and writing [Vortex](https://vortex.dev) files.
+It registers itself under the format name `vortex` and supports both the DataFrame API and
+Spark SQL.
+
+Two flavors are published to Maven Central:
+
+| Artifact                      | Spark     | Scala |
+|-------------------------------|-----------|-------|
+| `dev.vortex:vortex-spark_2.13` | Spark 4.x | 2.13  |
+| `dev.vortex:vortex-spark_2.12` | Spark 3.5.x | 2.12 |
+
+Use the `all` classifier JAR (e.g. `vortex-spark_2.13-0.78.0-all.jar`). It is self-contained:
+it bundles the Vortex JNI bindings, native libraries for Linux (x86_64 and aarch64) and macOS
+(aarch64), and relocates its Arrow, Guava, and Jackson dependencies to avoid classpath
+conflicts with Spark. The thin (unclassified) JAR does not work on its own because it
+references relocated classes that only ship in the `all` JAR.
+
+## Getting Vortex into Spark
+
+Pass the `all` JAR to `spark-shell`, `spark-submit`, or `pyspark` with `--jars`. Spark accepts
+either a local path or a URL, so you can point directly at Maven Central:
+
+```shell
+spark-shell --jars https://repo1.maven.org/maven2/dev/vortex/vortex-spark_2.13/0.78.0/vortex-spark_2.13-0.78.0-all.jar
+```
+
+Or configure it on the session builder, e.g. in PySpark:
+
+```python
+spark = (
+    SparkSession.builder
+    .config("spark.jars", "/path/to/vortex-spark_2.13-0.78.0-all.jar")
+    .getOrCreate()
+)
+```
+
+Note that `--packages dev.vortex:vortex-spark_2.13:0.78.0` does not work: `--packages` cannot
+select the `all` classifier and resolves the thin JAR, which fails at runtime with
+`NoClassDefFoundError: dev/vortex/relocated/...`.
+
+To depend on the connector from a JVM project instead, add the `all` classifier to the
+dependency:
+
+```kotlin
+implementation("dev.vortex:vortex-spark_2.13:0.78.0:all")
+```
+
+## Usage
+
+Paths may be local filesystem paths (`/path/to/data`) or URLs (`file:///path/to/data`,
+`s3://bucket/path/to/data`).
+
+### DataFrame API
+
+```java
+// Write
+df.write()
+    .format("vortex")
+    .option("path", "/path/to/output")
+    .mode(SaveMode.Overwrite)
+    .save();
+
+// Read a single file or a directory of .vortex files
+Dataset<Row> df = spark.read()
+    .format("vortex")
+    .option("path", "/path/to/output")
+    .load();
+```
+
+### Spark SQL
+
+```sql
+-- Query existing Vortex files through a temporary view
+CREATE TEMPORARY VIEW people
+USING vortex
+OPTIONS (path '/path/to/data');
+
+SELECT name, age FROM people WHERE age > 30;
+
+-- Create a table and write to it. With a LOCATION clause the table is external,
+-- backed by the files at that path; without one it is managed by Spark.
+CREATE TABLE student (id INT, name STRING, age INT)
+USING vortex;
+
+INSERT INTO student VALUES (1, 'Alice', 20), (2, 'Bob', 21);
+
+SELECT * FROM student;
+```
+
+On Spark 3.5, `CREATE TABLE ... USING vortex` additionally requires replacing the session
+catalog with `spark.sql.catalog.spark_catalog=dev.vortex.spark.VortexSessionCatalog`,
+because Spark 3.5's built-in catalog cannot read tables backed by a DataSource V2-only
+connector. The extension delegates everything else to the built-in session catalog and
+leaves tables of other providers untouched; it is not needed on Spark 4.
+
+### Direct file queries
+
+Spark's built-in ``SELECT * FROM format.`path` `` syntax only works for built-in file
+formats, so the connector ships a path-based catalog that provides the equivalent for
+Vortex. Register it under the name `vortex` with this session config:
+
+```
+spark.sql.catalog.vortex=dev.vortex.spark.VortexCatalog
+```
+
+Then query (or insert into) Vortex files directly by path:
+
+```sql
+SELECT * FROM vortex.`/path/to/data`;
+```
+
+See the [Spark user guide](https://docs.vortex.dev/user-guide/spark.html) for the full
+documentation, including supported types, write options, and S3 configuration.