Evaluate ditching optional dependencies aggregation in setup.py in favour of fully static metadata

[astrojuanlu]

This doesn't technically obliterate setup.py, but since most of the project metadata is in pyproject.toml now, maybe we could just keep setup.py around for our extras mangling, it's not a big deal.

Originally posted by @astrojuanlu in #161 (comment)

I just discovered an interesting trick attrs is using to merge and combine optional dependencies: https://github.com/python-attrs/attrs/pull/993/files#diff-50c86b7ed8ac2cf95bd48334961bf0530cdc77b5a56f852c5c61b89d735fd711R36-R65

This seems to be possible with any PEP 621 compatible build backend, but now it's not documented broadly (pypa/pip#11296), only for Hatch: https://hatch.pypa.io/latest/config/dependency/#self-referential

Feature groups can self-referentially extend others.

(Source: https://discuss.python.org/t/aliased-or-aggregated-optional-dependencies/20488/2?u=astrojuanlu)

In summary, it might be possible to actually remove setup.py completely, by replacing our dynamic aggregation of optional dependencies by this self-referencing feature directly in pyproject.toml.

Low priority, but it would be nice to have 100 % static metadata and only 1 metadata file.

Related: kedro-org/kedro#2334

[ofek]

Would you be open to PRs? I wouldn't mind creating a few that switch to Hatchling.

[astrojuanlu]

Hi @ofek, thanks for chiming in! Assuming we don't embrace Hatch as a workflow tool yet, what benefits would hatchling bring as a build backend over setuptools?

For extra context, I already migrated most of the project metadata of the Kedro framework to pyproject.toml while keeping setuptools in kedro-org/kedro#2475, and this issue is about kedro-datasets, a plugin that exists in this repository. Since the team already has familiarity with setuptools I didn't want to complicate reviews.

On the other hand, now that you're here, I think kedro-viz might actually benefit from hatchling plugins, since it bundles some JavaScript using hacky workarounds that have some nasty consequences... kedro-org/kedro-viz#1267