Spark 3.5, 4.0: Fix the test parameters to match the corresponding values

[tomtongue]

Change

The current test parameters in the TestRewritePositionDeleteFiles don't match the actual test values (the details are below). So this PR tries to fix these parameters to match the actual values, and also to parameterize the table format version for the method createTable.

Details

An example of the current test logs (each parameter name like formatVersion, catalogName etc. doesn’t match the corresponding value as below):

TestRewritePositionDeleteFiles > testDatePartition() > formatVersion = testhive, catalogName = org.apache.iceberg.spark.SparkCatalog, implementation = {type=hive, default-namespace=default, cache-enabled=false}, config = {3} STANDARD_ERROR // <= not match
    [Test worker] INFO org.apache.spark.sql.internal.SharedState - Setting hive.metastore.warehouse.dir ('null') to the value of spark.sql.warehouse.dir.
    [Test worker] INFO org.apache.spark.sql.internal.SharedState - Warehouse path is 'file:/path/to/iceberg/spark/v4.0/spark-extensions/spark-warehouse'.
    [Test worker] INFO org.sparkproject.jetty.server.handler.ContextHandler - Started o.s.j.s.ServletContextHandler@708378cc{/SQL,null,AVAILABLE,@Spark}
...

After applying this change (each parameter name matches the corresponding value):

TestRewritePositionDeleteFiles > testDatePartition() > catalogName = testhive, implementation = org.apache.iceberg.spark.SparkCatalog, config = {type=hive, default-namespace=default, cache-enabled=false}, formatVersion = 2 STANDARD_ERROR // <= match
    [Test worker] INFO org.apache.spark.sql.internal.SharedState - Setting hive.metastore.warehouse.dir ('null') to the value of spark.sql.warehouse.dir.

[tomtongue]

I believe it's also possible to simply remove the formatVersion parameter if this test is only for V2.

[ebyhr]

I believe it's also possible to simply remove the formatVersion parameter if this test is only for V2.

+1 to removing the formatVersion from this annotation, since I don't expect any new format versions to be added here given the deletion vectors. We can always parameterize it later if needed.

[nastra]

we should actually keep the format version and upgrade the test to also run with V3. I have some changes locally that I created a while ago to do that

[tomtongue]

Sure, thanks for checking this part.

[nastra]

LGTM, thanks for fixing @tomtongue

[tomtongue]

Thanks for the review and merge @nastra and thanks for checking this change @ebyhr!