[SPARK-57787][CONNECT] Reuse a persistent local Spark Connect server for faster local startup #56907

nchammas · 2026-07-29T16:32:41Z

Just curious: Since we are already pointing the user to the sbin scripts, why add another mechanism for stopping the server? It's another public interface we have to maintain.

Also, if PySpark starts a persistent Connect server via local_server, can the user still find and stop it normally using the sbin scripts? Or do they have to use local_server?

It's not really a second stack, the managed server is a normal spark-daemon.sh daemon started through sbin/start-connect-server.sh.
But it runs under a per-user SPARK_PID_DIR/SPARK_IDENT_STRING so it can't collide with a server you start by hand, which also means a plain sbin/stop-connect-server.sh doesn't find its pid file. --stop just signals the recorded pid and cleans up the discovery file. Added a paragraph to the docs explaining this.

Only by pointing the script at the managed locations (SPARK_PID_DIR=<runtime dir> SPARK_IDENT_STRING=local-connect sbin/stop-connect-server.sh), which isn't ergonomic. Plain kill <pid> works too -- the next run notices the dead server and starts a fresh one. Covered in the new docs paragraph.

Hmm, OK thanks for explaining.

I'm thinking ahead to a world where users can run something like spark connect {start | stop | status}. I would want that to be the only thing they have to learn to manage a Connect server, including when/if PySpark auto-starts one. i.e. User runs PySpark with the new config added here, but can also see and manage the auto-started Connect server using spark connect ....

I feel like that would be ideal, but that might also mean that the discovery mechanism would need to live in Scala, not here. I'm not sure. Perhaps that can be a future change.

I just want to call this out here and now in case it makes sense to explicitly label the local_server interface as experimental, which would allow us to change it or fold it into Scala down the line.

Makes sense. Marked this as experimental in the docs and the module docstring - specifically the --stop command and the discovery file location/format, which are the parts a spark connect CLI would replace. I'd keep the opt-in config name itself stable since that's what ends up in users' scripts. One note on the Scala point: the discovery write path would move server-side, but the client-side probe (is the recorded server alive, right version, port open) stays a client concern either way, so this shouldn't paint us into a corner.

-Original file line number
+Diff line change
@@ Expand Up / @@ -1178,6 +1178,7 @@ def __hash__(self): @@
             "pyspark.sql.tests.connect.test_connect_readwriter",
             "pyspark.sql.tests.connect.test_connect_retry",
             "pyspark.sql.tests.connect.test_connect_session",
+            "pyspark.sql.tests.connect.test_connect_local_server",
             "pyspark.sql.tests.connect.test_connect_stat",
             "pyspark.sql.tests.connect.test_parity_geographytype",
             "pyspark.sql.tests.connect.test_parity_geometrytype",
@@ Expand Down @@

-Original file line number
+Diff line change
@@ Expand Up @@
     </div>
     </div>
+    ## Faster local iteration with a persistent Connect server
+    When you develop or test locally with
+    ```python
+    from pyspark.sql import SparkSession
+    spark = SparkSession.builder.remote("local[*]").getOrCreate()
+    ```
+    PySpark boots a fresh in-process Spark Connect server that lives only as long as that Python
+    process. Every `python script.py` invocation (or every new test worker process) therefore re-pays
+    the one-time startup cost -- JVM warmup, `SparkContext` construction, and Connect server boot --
+    which can take a few seconds and makes a quick edit/run loop feel slow.
+    To amortize that cost across runs, start one persistent local Spark Connect server and point
+    every run at it:
+    ```bash
+    # Start once; it stays up across runs. (--master is optional; it defaults to local[*].)
+    $SPARK_HOME/sbin/start-connect-server.sh --master "local[*]"
+    # Every run reconnects instead of booting a new server.
+    python -c 'from pyspark.sql import SparkSession; SparkSession.builder.remote("sc://localhost:15002").getOrCreate()'
+    # Stop it when you are done.
+    $SPARK_HOME/sbin/stop-connect-server.sh
+    ```
+    Alternatively, on POSIX systems PySpark can manage this persistent server for you. With
+    `SPARK_LOCAL_CONNECT_REUSE=1` set (or `spark.local.connect.reuse=true` on the builder),
+    `SparkSession.builder.remote("local[*]").getOrCreate()` starts a persistent server through
+    `sbin/start-connect-server.sh` on the first run and reconnects to it on later runs, so scripts
+    keep the plain `local[*]` URL:
+    ```bash
+    export SPARK_LOCAL_CONNECT_REUSE=1
+    # The first run starts the server; later runs reconnect to it.
+    python -c 'from pyspark.sql import SparkSession; SparkSession.builder.remote("local[*]").getOrCreate()'
+    # Stop the managed server when you are done.
+    python -m pyspark.sql.connect.local_server --stop
+    ```
+    The managed server is an ordinary `spark-daemon.sh` daemon, but it runs with a per-user pid
+    directory and ident string so it cannot collide with a server you started by hand -- which also
+    means a plain `sbin/stop-connect-server.sh` does not find it. The `--stop` command signals the
+    recorded server and cleans up the discovery file; killing the server's pid directly also works,
+    and the next run notices the dead server and starts a fresh one.
+    This managed-server workflow is experimental. The `--stop` command and the discovery file
+    location and format may change in a future release, for example if local server management is
+    folded into a unified `spark connect` CLI.
+    The connection details (host, port, auth token, pid, Spark version) are recorded in a discovery
+    file in a private per-user directory; set `SPARK_LOCAL_CONNECT_DISCOVERY` to override its
+    location. A run reconnects only to a server whose Spark version matches. After upgrading Spark,
+    the server from the previous version cannot be reused and the next run fails with an error asking
+    you to stop it; run the `--stop` command above and rerun to start a fresh server.
+    Each run connects as its own Connect session, so session-local state -- temp views, runtime SQL
+    configurations, and session artifacts -- is fresh on every run and never leaks between runs. State
+    backed by the shared `SparkContext` (the persistent catalog/warehouse, global temp views, and
+    cached datasets) *is* shared across runs, so namespace per-run databases or clear that state
+    yourself if your runs must be fully isolated.
     ## Use Spark Connect in standalone applications
     <div class="codetabs">
@@ Expand Down Expand Up @@
     </div>
     For more information on application development with Spark Connect as well as extending Spark Connect
-    with custom functionality, see [Application Development with Spark Connect](app-dev-spark-connect.html).
+    with custom functionality, see [Application Development with Spark Connect](app-dev-spark-connect.html).
     # Client application authentication
     While Spark Connect does not have built-in authentication, it is designed to
@@ Expand Down @@

-Original file line number
+Diff line change
@@ Expand Up / @@ -345,7 +345,9 @@ def run(self): @@
                 "pyspark.sbin": [
                     "spark-config.sh",
                     "spark-daemon.sh",
+                    "start-connect-server.sh",
                     "start-history-server.sh",
+                    "stop-connect-server.sh",
                     "stop-history-server.sh",
                 ],
                 "pyspark.python.lib": ["*.zip"],
@@ Expand Down @@

-Original file line number
+Diff line change
@@ Expand Up / @@ -546,6 +546,16 @@ @@
           "<arg1> and <arg2> should be of the same length, got <arg1_length> and <arg2_length>."
         ]
       },
+      "LOCAL_CONNECT_RUNTIME_DIR_UNAVAILABLE": {
+        "message": [
+          "Cannot claim the per-user runtime directory <path> (was it created by another user?); remove it or point SPARK_LOCAL_CONNECT_DISCOVERY at a path you own."
+        ]
+      },
+      "LOCAL_CONNECT_SERVER_START_FAILED": {
+        "message": [
+          "Failed to start a persistent local Spark Connect server: <reason>."
+        ]
+      },
       "LOCAL_RELATION_SIZE_LIMIT_EXCEEDED": {
         "message": [
           "Local relation size (<actualSize> bytes) exceeds the limit (<sizeLimit> bytes)."
@@ Expand Down @@

Navigation Menu

Uh oh!

[SPARK-57787][CONNECT] Reuse a persistent local Spark Connect server for faster local startup #56907

Uh oh!

Diff view

Diff view

There are no files selected for viewing

Uh oh!

Uh oh!

nchammas Jul 29, 2026

Uh oh!

nchammas Jul 29, 2026

Uh oh!

ericm-db Jul 29, 2026 •

edited

Loading

Uh oh!

ericm-db Jul 29, 2026

Uh oh!

nchammas Jul 29, 2026

Uh oh!

ericm-db Jul 29, 2026

Uh oh!

Uh oh!

Uh oh!

[SPARK-57787][CONNECT] Reuse a persistent local Spark Connect server for faster local startup #56907

Uh oh!

[SPARK-57787][CONNECT] Reuse a persistent local Spark Connect server for faster local startup #56907

Uh oh!

Uh oh!

Diff view

Diff view

There are no files selected for viewing

Uh oh!

Uh oh!

nchammas Jul 29, 2026

Choose a reason for hiding this comment

Uh oh!

nchammas Jul 29, 2026

Choose a reason for hiding this comment

Uh oh!

ericm-db Jul 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ericm-db Jul 29, 2026

Choose a reason for hiding this comment

Uh oh!

nchammas Jul 29, 2026

Choose a reason for hiding this comment

Uh oh!

ericm-db Jul 29, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ericm-db Jul 29, 2026 •

edited

Loading