chore(provision): make Spark integration test provisioning idempotent

[freedom3219]

Purpose

Make Spark-based integration test fixture provisioning idempotent.

dev/spark/provision.py previously used CREATE TABLE IF NOT EXISTS together with unconditional INSERT INTO statements. Re-running provisioning against the same warehouse caused append/log tables to accumulate duplicate rows, which made the test fixture non-deterministic and broke integration tests.

Brief change log

• Resolve the configured Paimon warehouse path from Spark config
• Clear the existing warehouse contents before provisioning test fixtures
• Keep the existing table definitions and fixture data unchanged
• Preserve the current reader behavior and test expectations while making repeated provisioning deterministic

This approach fits the current repository architecture because the integration tests assert exact fixture contents, while append tables are read as appended data rather than deduplicated business rows. Resetting the whole warehouse before provisioning is the most reliable way to keep the fixture stable across append tables, DV PK tables, and data evolution tables.

Tests

Verified idempotent provisioning by running fixture setup twice against the same warehouse, then running:

• cargo test -p paimon-integration-tests --test read_tables
• cargo test -p paimon-datafusion --test read_tables

API and Format

Documentation

[JingsongLi]

+1