Include spark submit canonical logs in failure exceptions

[amoghrajesh]

Why?

Airflow only surfaced spark-submit's exit code when the process failed, discarding the JVM's actual stdout/stderr (missing jar, bad main class, Kerberos errors, etc.). Humans and Retry policies evaluating these exceptions had nothing but an exit code to reason from.

Some cases I encountered are when Spark's SparkSubmitHook.submit() raises AirflowException on three
failure paths:

• a nonzero spark-submit exit code,
• a Kubernetes exit-code mismatch, and
• a standalone-cluster submit that produced no driver id.

In all three cases the exception message contained only the masked command line and an exit code -- never the spark-submit process's own stdout/stderr, even though that output already streams through _process_spark_submit_log() line by line and often contains the real cause (missing jar, bad main class, Kerberos failure, etc.).

This makes root-causing failures harder for humans, and gives automated/LLM-based retry policies nothing to work with beyond an exit code cos they can't distinguish a transient submit infra hiccup from a permanent misconfiguration.

What

This PR adds a small rolling buffer of the last 20 lines of spark-submit's own output (_last_submit_log_lines, populated in
_process_spark_submit_log) and appends it, masked the same way the command line already is, to all three submit-failure exceptions via a new _format_submit_log_tail() helper. Example:

            "Cannot execute: spark-submit --master spark://spark-master:7077 --deploy-mode cluster "
            "/opt/spark/apps/does-not-exist.jar. Error code is: 1.\n"
            "Last spark-submit output:\n"
            "WARNING: Using incubator modules: jdk.incubator.vector\n"
            "26/07/21 07:03:03 WARN NativeCodeLoader: Unable to load native-hadoop library for your "
            "platform... using builtin-java classes where applicable\n"
            'Exception in thread "main" org.apache.spark.SparkException: Failed to get main class in '
            "JAR with error 'File file:/opt/spark/apps/does-not-exist.jar does not exist'.  Please "
            "specify one with --class.\n"
            "\tat org.apache.spark.deploy.SparkSubmit.error(SparkSubmit.scala:1065)\n"
            "\tat org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:547)\n"
            "\tat org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:95)"

---

Was generative AI tooling used to co-author this PR?

• Yes (please specify the tool below)

---

• Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
• For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
• When adding dependency, check compliance with the ASF 3rd Party License Policy.
• For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

[gopidesupavan]

This is useful as a baseline improvement over the current exit-code-only exception, but I’m not sure the last 20 lines will consistently include the root cause. Spark often logs the meaningful failure earlier and then ends with generic shutdown/final-status lines.

[shahar1]

Static checks currently fail

[Lee-W]

rebase for the main for the CI fix