Unable to use deltalake Schema in write_deltalake

[LoicRaillon]

Environment

Delta-rs version: 0.13.0
Binding: Python 3.11
OS: Windows 10 WSL2 (Ubuntu 22.04.02 LTS)

---

Bug

I want to use a deltalake.schema.Schema instance in the function write_deltalake(schema=...) in order to control the nullable parameter and add metadata. However, a pyarrow.lib.Schema type is expected instead of deltalake._internal.Schema. This is issue is reproducible with Polars and Pandas as shown with the snippet below

import json

import polars as pl
import pandas as pd
from deltalake import write_deltalake
from deltalake.schema import Schema


schema = Schema.from_json(
    json.dumps(
        {
            "type": "struct",
            "fields": [
                {"name": "x", "type": "integer", "nullable": True, "metadata": {"description": "variable x"}},
                {"name": "y", "type": "float", "nullable": False, "metadata": {"description": "variable y"}},
            ],
        }
    )
)

data = {"x": [1, 2], "y": [3.0, 4.0]}

try:
    write_deltalake(
        "data/test_schema",
        data=pl.DataFrame(data),
        name="test",
        schema=schema,
        description="A test",
        mode="overwrite",
    )
except TypeError as e:
    print(f"Polars -> {e}")

try:
    write_deltalake(
        "data/test_schema",
        data=pd.DataFrame(data),
        name="test",
        schema=schema,
        description="A test",
        mode="overwrite",
    )
except TypeError as e:
    print(f"Pandas -> {e}")

Polars -> Argument 'schema' has incorrect type (expected pyarrow.lib.Schema, got deltalake._internal.Schema)
Pandas -> Argument 'schema' has incorrect type (expected pyarrow.lib.Schema, got deltalake._internal.Schema)

[r3stl355]

take

[r3stl355]

I'm not actually looking into this, maybe I will later, just tried the take comment action

[r3stl355]

I wouldn't call this a bug as the write_deltalake definition clearly states it's accepting a pyarrow.Schema. On top of that, deltalake._internal.Schema has a to_pyarrow function which you can use on your Schema before passing to write_deltalake.

[ion-elgreco]

@r3stl355 I think there is a method call to convert delta schema to pyarrow, so perhaps we should allow take both inputs

[r3stl355]

yep @ion-elgreco , deltalake._internal.Schema.to_pyarrow as I mentioned earlier. I can extend the function to take either if you think it's useful

[ion-elgreco]

@r3stl355 ah my bad, read too fast over your post

[r3stl355]

take

[LoicRaillon]

I missed the fact that the nullable and the metadata are stored in the fields attributes of the `pyarrow.Schema. Should I close this issue or is it still pertinent ?

[r3stl355]

I don't know, maybe remove a bug label for now. What do you think @ion-elgreco - should I extend the write_deltalake to accept either of Schema types?

[ion-elgreco]

@r3stl355 yes makes sense to add