[SPARK-33831][UI] Update to jetty 9.4.34

[srowen]

What changes were proposed in this pull request?

Update Jetty to 9.4.34

Why are the changes needed?

Picks up fixes and improvements, including a possible CVE fix.

https://github.com/eclipse/jetty.project/releases/tag/jetty-9.4.33.v20201020
https://github.com/eclipse/jetty.project/releases/tag/jetty-9.4.34.v20201102

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Existing tests.

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37569/

[SparkQA]

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37569/

[SparkQA]

Test build #132966 has finished for PR 30828 at commit 188e81b.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[maropu]

retest this please

[dongjoon-hyun]

Thanks, @srowen . The behavior of Jetty seems to change. This fails at

      // Test Jetty's built-in redirect to add the trailing slash to the context path.
      TestUtils.withHttpConnection(new URL(s"$serverAddr/ctx1")) { conn =>
        assert(conn.getResponseCode() === HttpServletResponse.SC_FOUND)
        val location = Option(conn.getHeaderFields().get("Location"))
          .map(_.get(0)).orNull
        assert(location === s"$proxyRoot/ctx1/")
      }

Could you check the root cause and adjust some configuration or the UT UiSuite.redirect with proxy server support? It would be great if we have the same behavior, but if it's inevitable, shall we add some comment on the migration guide?

[info] - redirect with proxy server support *** FAILED *** (32 milliseconds)
[info]   404 did not equal 302 (UISuite.scala:367)

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37572/

[srowen]

Weird, doesn't happen on the previous 9.4.34 version. I'll just go with that for now, because either:

• It's a bug in the Spark Jetty code, which I'd punt on this moment as I just want to backport fixes back to 2.x, and want to minimize change
• Or, maybe it's an unintentional change that a slightly later Jetty will fix anyway

[SparkQA]

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37572/

[dongjoon-hyun]

Thank you for updates, @srowen .
+1, LGTM (Pending CIs).

[dongjoon-hyun]

cc @HyukjinKwon since he is the release manager of Apache Spark 3.1.0.

This issue is targeting all branches.

[HyukjinKwon]

LGTM too. Thanks @dongjoon-hyun for cc'ing me.

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37577/

[SparkQA]

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37577/

[dongjoon-hyun]

Merged to master/3.1/3.0/2.4. Thanks.

[SparkQA]

Test build #132976 has finished for PR 30828 at commit f5b67b1.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #132969 has finished for PR 30828 at commit 188e81b.

• This patch fails from timeout after a configured wait of 500m.
• This patch merges cleanly.
• This patch adds no public classes.