[SPARK-58227][CONNECT] Support Spark Connect in the spark-sql CLI (spark-sql --remote)

[dhruv-pratap]

What changes were proposed in this pull request?

This adds Spark Connect support to the spark-sql CLI: spark-sql --remote sc://...
starts a SQL shell backed by a remote Spark Connect session, mirroring the existing
spark-shell --remote (SPARK-48936).

A new SparkConnectSQLCLIDriver is added in sql/connect/client/jvm, selected when
--remote / spark.api.mode=connect is set (the same dispatch spark-shell uses). It
builds a remote SparkSession against the sc:// URL (as ConnectRepl does), executes
statements via spark.sql(...), prints tab-separated output client-side, and supports
-e / -f and an interactive loop.

The existing SparkSQLCLIDriver (Catalyst / hive-thriftserver-coupled) is unchanged and
remains the default; there is no sql/hive-thriftserver dependency and no change to
existing public API. This completes the --remote entry-point set under the SPARK-49194
umbrella (spark-shell / spark-submit / pyspark already have it); spark-sql was the
last interactive entry point without it.

Scope: an initial, focused implementation. Follow-ups (to be tracked as separate JIRAs):
full hiveResultString output parity (types / nulls / complex / timestamps), remaining
CLI flags (--database, init files, ...), prompt / USE db semantics, and a
quote/comment-aware statement splitter (a robust one exists as StringUtils.splitSemiColon
in catalyst but isn't on the Connect client's classpath -- relocating it to sql-api for
shared reuse is a candidate; guidance welcome).

Design doc: https://docs.google.com/document/d/14iOZAZFt5BN9whbbjCzZX1mVuhaKq2LEkfP6Ts4prHo/edit
dev@ discussion: https://lists.apache.org/thread/cw897hvjqlmrtdo7byr10b86wtls819l

Why are the changes needed?

spark-sql is the last interactive entry point without Spark Connect support, and the only
one still requiring a local Catalyst driver and the Hive Thrift Server module. This lets the
SQL CLI run against remote, service-hosted compute, consistent with the other entry points.

Does this PR introduce any user-facing change?

Yes -- a new spark-sql --remote sc://... mode. No change to existing spark-sql behavior;
the classic local driver remains the default.

How was this patch tested?

• Unit (server-free, ConnectFunSuite): SparkConnectSQLCLIDriverSuite -- covers the
  -e/-f split from the Connect client args, statement splitting, and row formatting.
• Integration / E2E (SparkConnectSQLCLIDriverE2ESuite, on RemoteSparkSession): boots a
  local Spark Connect server and drives the real remote path -- spark-sql --remote -e,
  multiple ;-separated statements, and an -f script -- asserting rendered output.
• CI green: compile, unit, and the E2E integration test on JDK 17/21; scalastyle / scalafmt /
  license / checkstyle all pass.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Opus 4.8 (Claude Code)