OPS-5046 support spark 3.4.0 on SPOK

[xuzeng012]

spark 3.4.0 made a change to spark type CharType which requires the constructor to take a parameter length so we are using spark's internal method to take care of all that parsing

[srstrickland]

tests are broken, it seems because the Dockerfile is not properly set up to install jdk8. this needs to be fixed. I recommend testing locally using make test, or for faster iteration, make dev then run tests with tox when you're inside the container.

[srstrickland]

Looks good, but let's make sure! We can add a spark34 section to tox.ini to force our test suite to run on spark 3.4. just copy the spark33 section and modify the pyspark version.

[srstrickland]

Looks like there are several failures with 3.4.0:

tests/integration/test_catalog.py ..............F...............F.       [ 19%]
tests/integration/test_functions.py .....................                [ 31%]
tests/integration/test_instant_testing.py ..                             [ 32%]
tests/integration/test_reader.py ......                                  [ 36%]
tests/integration/test_session.py ....                                   [ 38%]
tests/integration/test_testing.py .........                              [ 44%]
tests/integration/test_writer.py .....FF.....................            [ 61%]
tests/unit/test_instant_testing.py ....                                  [ 63%]
tests/unit/test_reader.py ...........                                    [ 70%]
tests/unit/test_session.py ...............                               [ 79%]
tests/unit/test_testing.py ..........                                    [ 85%]
tests/unit/test_utils.py .................                               [ 95%]
tests/unit/test_writer.py ........                                       [100%]

FAILED tests/integration/test_catalog.py::TestSparklyCatalog::test_rename_table_non_default_db
FAILED tests/integration/test_catalog.py::TestSparklyWithOldCatalog::test_rename_table_non_default_db
FAILED tests/integration/test_writer.py::TestWriteKafka::test_write_kafka_dataframe
FAILED tests/integration/test_writer.py::TestWriteKafka::test_write_kafka_dataframe_error

it would seem that more work is required here to make everything work.