[SPARK-58480][PYTHON][TEST] Declare DataFrame golden test cases as methods of a test class

[dtenedor]

What changes were proposed in this pull request?

This PR follows up on #57122 and #57639 by replacing the per-case Python snippet files of the DataFrame golden test framework with a single test class per golden file.

The files under python/pyspark/sql/tests/df_golden/scripts/group_by/ are replaced by one module, python/pyspark/sql/tests/df_golden/test_group_by.py, holding one class whose _test_<case> methods each build and return the DataFrame under test:

        class GroupByGoldenTests(DFGoldenTestMixin, ReusedConnectTestCase):
            golden_file = "group_by.test"
            @unordered
            def _test_group_by_count(self, spark):
                """Aggregate with non-empty GroupBy expressions."""
                return spark.table("testData").groupBy(col("a")).agg(count(col("b")))

This is ordinary Python: imports are at the top of the module, cases can share helpers, and spark is a parameter rather than a name injected into an exec() namespace.

Cases are ordinary unit tests: DFGoldenTestMixin (a mixin, following the existing pyspark.testing.goldenutils.GoldenFileTestMixin pattern) registers a real test_<case> method for every _test_<case> method, carrying its docstring over. Each case is reported individually and can be run on its own:

        python/run-tests --testnames \
          "pyspark.sql.tests.df_golden.test_group_by GroupByGoldenTests.test_group_by_count"

Why are the changes needed?

The framework as merged put each case's DataFrame program in its own file, which drew several objections in the review of #57122:

1. File count. One file per case does not scale.
2. The snippets were not real Python. They were exec()'d with spark injected into their namespace, which is why they needed an F821 lint exemption and a RAT exclusion, and why they could not be imported or run outside the framework.
3. Reproducing a failure was hard A whole .test file used to run as one unittest method, so a failure named a case but gave no way to run just that case. Now every case is a test method that can be selected by name, and the case body is an importable method.
4. Structure. Organizing cases as methods of a class is the conventional Python shape, and it lets cases share helpers.

Does this PR introduce any user-facing change?

No. This is test-only: the framework and its test corpus.

How was this patch tested?

This PR is test-only. The migration was verified as follows.

• Golden outputs are unchanged.
• The regeneration writer reproduces the file byte for byte.
• The class and the golden file are in sync.
• Framework unit tests: pyspark.sql.tests.df_golden.test_df_golden_framework passes (51 tests).

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Cursor (Opus 5)

[uros-b]

Thank you @dtenedor! cc @zhengruifeng

[gaogaotiantian]

Generally, I believe this is the correct direction. We should have a relatively complicated mixin and many easy to read/reproduce test cases. I'm glad that we can get rid of a lot of files.

I have some comments about the implementation. It's a framework and I hope the code is cleaner and easier for extension in the future.

I also have some suggestions for next steps, but they are not for this PR.

[gaogaotiantian]

I hope eventually we can get rid of our own file format and use something more generic like toml - which are also super readable by both humans and machines. We won't need to maintain our own parser for each section anymore. But this is not part of this PR.

[dtenedor]

Yes, currently there are a lot of these .out golden files for testing in Apache Spark (example [1]). We could consider changing that, but it would be a bigger change than this PR would do in one go.

[1]

spark/sql/core/src/test/resources/sql-tests/analyzer-results/kllquantiles.sql.out

Line 4 in 3539561

-- !query analysis