Separate "writing a plugin" from "registering an expression" in user guide and pyo3-polars examples

[MarcoGorelli]

Background

The user guide and the pyo3-polars repo teach you how to write a plugin so that you can write

out = df.with_columns(
   pig_latin = pl.col("names").language.pig_latinnify()
)

I think this is mixing together two different concepts:

• registering a custom namespace (here, .language)
• defining a function which works on expressions in Rust (here, pig_latinnify)

They're orthogonal, because:

• you can define a custom namespace without writing a plugin
• you can write a plugin without registering a custom namespace

Why I think this is an issue

Custom namespaces have their downsides:

• they require an extra unused import, which requires an extra # noqa: F401 for linters to not complain about it
• they break static typing

On the second point, some solutions have been proposed:

• polugins. This uses stubgen to generate stub files for Polars, and then adds stubs from a user's plugin. It requires the polugins repo to be kept up-to-date with Polars. Furthermore, the stubgen docs themselves say that stubgen only generates "draft stubs"
• plugins define their own col, as suggested and tried here. Unfortunately, this breaks in expression chaining (unless the user remembers to use the plugin's own col for the entire chain), and also breaks if the user wants to combine multiple plugins

Proposal: de-couple defining plugins from registering expressions

Instead of teaching people to write

import polars as pl
import polars_language  # noqa: F401

out = df.with_columns(
   pig_latin = pl.col("names").language.pig_latinnify()  # type: ignore[attr-defined]
)

teach them how to write

import polars as pl
import polars_language as language

out = df.with_columns(
   pig_latin = language.pig_latinnify("names")
)

Then, if people also want to register them with a custom namespace, they are free to do that. It's very easy. The plugins page in the user guide could even point people there and mention it as a possibility. But I think it shouldn't be the primary way that plugins are taught.

[baggiponte]

Then, if people also want to register them with a custom namespace, they are free to do that. It's very easy. The plugins page in the user guide could even point people there and mention it as a possibility. But I think it shouldn't be the primary way that plugins are taught.

This is a good take. We might "standardise" the plugin structure and recommend to add a "registration" method for those who wish to do so, to make it explicit.

import polars as pl
import polars_language

try:
    polars_language.register(namespace="language")
except pl.exceptions.PolarsInvalidNamespaceError as e:
   ...

out = df.with_columns(
   pig_latin = pl.col("names").language.pig_latinnify()  # type: ignore[attr-defined]
)

[dangotbanned]

Custom namespaces have their downsides:

• they require an extra unused import, which requires an extra # noqa: F401 for linters to not complain about it

• they break static typing

I think these points, particularly the latter, should be documented alongside any mention of custom namespaces. I'm only just discovering this here after searching for why my extension didn't show up in code completion.