Update the spark version to the current version

[mikealfare]

resolves #1054

Problem

• we're using an old runtime version of Spark for some tests
• there's an issue in spacy that requires pydanic>=1.10.8 (see FIXED: Pydantic issubclass error for python 3.8 and 3.9 explosion/spaCy#12659)
• there's an issue in databricks.koalas that requires an update in handling numpy updates; however, this library is no longer supported

Solution

• update the version in the tests
• update pydanic<1.10.3 to pydanic>=1.10.8
• add a docstring explaining all of this
• exclude the koalas test

Checklist

• I have read the contributing guide and understand what's expected of me
• I have run this code in development and it appears to resolve the stated issue
• This PR includes tests, or tests are not required/relevant for this PR
• This PR has no interface changes (e.g. macros, cli, logs, json artifacts, config files, adapter interface, etc) or this PR has already received feedback and approval from Product or DX

[github-actions]

Thank you for your pull request! We could not find a changelog entry for this change. For details on how to document a change, see the dbt-spark contributing guide.

[VersusFacit]

Really appreciate the comments.