Repository navigation
[FR] External Data Support #3191
Description
Activity
Hi @blink1073 thank you very much for putting up this issue.
I think this request is very reasonable. Is there any other requirements for thejupyteruse case?
If I understood correctly the idea is to pick-up a directory and then replicate the file tree under it in the appropriate installation schema, right?Regarding the "deprecated" status of
data_files, my personal opinion is that "it's complicated"...The behaviour of installing files to arbitrary locations as it used to do in the distutils and then in the easy_install era is definitely deprecated. But I don't see the feature itself disappearing in the future (just that users have to understand its limitations and know exactly when to use it).
Other maintainers might disagree, but I do appreciate that there are genuine use cases for it (like jupyter).
Jupyter looks in
<sys_prefix>/share/jupyterand<sys_prefix>/etc/jupyterfor its files. As long as we can target those two directories we are fine.I think the current semantics of
data_filesare what is complicated: it can target absolute paths, and it is very tricky to use for nested files. We use a helper function to make it glob-friendly.I think with a new API we can offer a simpler, safer alternative and eventually remove
data_filessupport altogether.If there's buy-in I can take a crack at implementation.
Reacted by Tim Paine, Anderson Bravalheri and Min RKI am sold on that.
It would be nice to hear the opinion of @jaraco.
For more information, there is a in-depth discussion about why jupyter needs data dirs in the discourse, this is also relevant.
If I understood correctly jupyter needs to be able to share/read/use files from extensions. These files need to be found in the disk in a reliable way, even if the package providing them has not been imported yet. The performance of searching for entry-points on disk also seems to be an issue for jypyter.
@blink1073 Hey! I saw this linked on your slides from the packaging summit.
Hatchling supports a
shared-dataoption for wheels. Would that satisfy Jupyter's use case?Hi @ofek, thanks for pointing that out! The consensus at the summit was that I should draft a PEP to standardize the handling of the
dataandheadersdirectories for build backends. I will useflitandhatchas existing implementation examples.As for Jupyter, we could have used
flitalready (in fact I had a WIP), but many of the consumers ofjupyter-packagingare usingsetuptools-specific mechanisms and extensions.Reacted by Ofek Lev
What's the problem this feature will solve?
Project Jupyter uses
data_filesfor discovery of runtime extensions. This feature was deprecated without a viable alternative.Describe the solution you'd like
The
flitproject recently added support forexternal_data, which is a simplified and constrained version ofdata_files. We could support an equivalent feature insetuptools, with caveats about when it is appropriate to use it.Alternative Solutions
Project Jupyter and other current consumers of
data_filescould instead switch to using aflitbackend, which we have been actively exploring.Additional context
Project Jupyter uses
data_filesto allow libraries to provide shared setting and static assets files that are consumed by Jupyter at runtime. We have explored usingentry_pointsfor this purpose, but it did not scale well.Code of Conduct