[Feature] Improve version compatibility tests between old and new Paimon versions

[zhang-arvin]

Fixes #3531: Add version compatibility tests to verify:

1. New Paimon version can read old metadata and files
2. Old Paimon version can read new metadata and files

Changes

• Added SnapshotVersionCompatibilityTest for snapshot-level compatibility
• Added SchemaVersionCompatibilityTest for schema-level compatibility

[JingsongLi]

Requirement fit: PIVOT
Implementation: FINDINGS

[P2] Exercise real old-version readers and files

Location: SnapshotVersionCompatibilityTest#testNewSnapshotJsonCompatibleWithOldReader and SchemaVersionCompatibilityTest#testNewSchemaJsonCompatibleWithOldReader

Problem: Both “old reader” tests serialize with the current classes and immediately deserialize with the same current Snapshot.fromJson / JsonSerdeUtil.fromJson. No old Paimon artifact or old reader is loaded. Likewise, the backward tests use hand-written snapshot/schema JSON only; they do not read metadata and data files produced by an old release.

Trigger: A current change emits JSON, manifest, or data-file content that the current reader accepts but a supported prior Paimon release cannot read. These tests still pass because only the current reader is exercised.

Impact: This PR can go green while neither direction requested by #3531 is actually compatible, giving future format changes a false compatibility gate.

Fix: Run a real cross-version matrix in isolated classloaders/processes: have a pinned prior release write a table (including manifests/data files) and read it with HEAD, then have HEAD write a table and read it with that prior release. Checked-in release-generated fixtures can cover the old-writer/current-reader direction, but the current-writer/old-reader direction still needs the old reader artifact.

Claim: #3531 asks for both new→old and old→new metadata/file compatibility.
Evidence: Every parse in these new tests resolves to the current checkout, and no historical dependency, process, fixture archive, manifest, or data file is used.
Concern: The implementation verifies current-version round trips and permissive current parsing, not cross-version compatibility.
Cost: 839 lines of tests add maintenance while leaving the requested regression boundary unprotected.
Pivot: Replace the synthetic same-version cases with a small, version-pinned end-to-end compatibility harness covering both directions.

[zhang-arvin]

Thanks @JingsongLi — [P2] is a fair point and I agree with the direction. As the PR stands, both "old reader" tests round-trip current-version JSON through the current reader, so a change that an older release couldn't parse would still go green: that's a false compatibility gate, exactly the thing #3531 asks to protect.

Plan accepted: (1) old-writer → current-reader direction via checked-in fixtures generated by a pinned prior release (real snapshot/schema JSON plus manifest and data files), so HEAD must parse genuine old-release output; (2) current-writer → old-reader direction via a pinned old-release reader running against HEAD-produced files in an isolated classloader/process — as you note, that direction genuinely needs the old artifact. I'll restructure the two test classes around that harness.

On CI: I root-caused the all-matrix failure — the jobs never reached the tests. Both new test files trip checkstyle NewlineAtEndOfFile, which fails the validate phase of the paimon-core build that every build_test job depends on. Local runs passed only because I built with -Pfast-build, which skips checkstyle. I'll fix the newlines, run checkstyle locally before pushing, and report the real test results from the next CI run.