[SPARK-40680][CONNECT][BUILD] Avoid hardcoded versions in SBT build

[HyukjinKwon]

What changes were proposed in this pull request?

This PR proposes to read the corresponding versions from POM in SBT build instead of hardcoding the dependency versions.

In addition, this PR removes Scala system libraries in the dependency.

Why are the changes needed?

To avoid version duplication.

Does this PR introduce any user-facing change?

No, dev-only.

How was this patch tested?

CI in this PR should test it out.

[HyukjinKwon]

cc @grundprinzip @amaliujia @LuciferYang FYI

[HyukjinKwon]

Build: https://github.com/HyukjinKwon/spark/actions/runs/3197768433/jobs/5221428135

[dongjoon-hyun]

Could you re-trigger GitHub Action, @HyukjinKwon ?

[HyukjinKwon]

(the build passed - there seems a problem with linking the build to this PR)

[amaliujia]

thanks the change itself LGTM.

[amaliujia]

Seeming the same issue on my PR: so probably an infra issue?

[amaliujia]

Error: Unhandled error: HttpError: Resource not accessible by integration

[dongjoon-hyun]

Not only PR linking, the master branch seems to be broken due to ghcr.io/apache/apache-spark-ci-image.

• https://github.com/apache/spark/actions/runs/3199351281/jobs/5225053351

/usr/bin/docker --config /home/runner/work/_temp/.docker_ac20db86-d07f-459d-9681-cfa6ca3f8a2e pull ghcr.io/apache/apache-spark-ci-image:master-3197794220
  Error response from daemon: manifest unknown
  Error: Docker pull failed with exit code 1

[HyukjinKwon]

Merged to master.

[dongjoon-hyun]

+1, late LGTM.