[SPARK-45597][PYTHON][SQL] Support creating table using a Python data source in SQL (single wrapper)

[HyukjinKwon]

What changes were proposed in this pull request?

This PR is another approach of #43784 which proposes to support Python Data Source can be with SQL (in favour of #43949), SparkR and all other exiting combinations by wrapping the Python Data Source by DSv2 interface (but yet uses V1Table interface).

The approach is: one Python Data Source wrapper looks up Python data sources

Self-contained working example:

from pyspark.sql.datasource import DataSource, DataSourceReader, InputPartition

class TestDataSourceReader(DataSourceReader):
    def __init__(self, options):
        self.options = options
    def partitions(self):
        return [InputPartition(i) for i in range(3)]
    def read(self, partition):
        yield partition.value, str(partition.value)

class TestDataSource(DataSource):
    @classmethod
    def name(cls):
        return "test"
    def schema(self):
        return "x INT, y STRING"
    def reader(self, schema) -> "DataSourceReader":
        return TestDataSourceReader(self.options)

spark.dataSource.register(TestDataSource)
sql("CREATE TABLE tblA USING test")
sql("SELECT * from tblA").show()

results in:

+---+---+
|  x|  y|
+---+---+
|  0|  0|
|  1|  1|
|  2|  2|
+---+---+

There are limitations and followups to make:

1. Statically loading Python Data Sources is still not supported (SPARK-45917)

Why are the changes needed?

In order for Python Data Source to be able to be used in all other place including SparkR, Scala together.

Does this PR introduce any user-facing change?

Yes. Users can register their Python Data Source, and use them in SQL, SparkR, etc.

How was this patch tested?

Unittests were added, and manually tested.

Was this patch authored or co-authored using generative AI tooling?

No.

Closes #43784

[HyukjinKwon]

cc @allisonwang-db and @cloud-fan

[allisonwang-db]

The approach looks good!

[allisonwang-db]

This one should be fixed?

[HyukjinKwon]

For basic support, I think so. The thing is that we should take a look into session inheritance, testcase, etc. So I leave this as a todo for now.

[allisonwang-db]

Does it get the correct session for spark connect?

[HyukjinKwon]

yes

[allisonwang-db]

Should we support external metadata for this data source? I.e users can create a table using a python datasource with user defined table schema.

[HyukjinKwon]

I believe it already does (?).

[dongjoon-hyun]

Just a question. So, which one is the final decision, this PR or #43784 ? Do we need to collect more opinions?

[HyukjinKwon]

Was investigating pros and cons. For now, this one is more likely the one but waiting @cloud-fan 's sign off :-).

[HyukjinKwon]

Update: we're offline discussing. I will make another POC PR. We will write up the summary in the final PR description.