Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

File-based archive/extractor data loaders? #1247

Open
Fil opened this issue Apr 16, 2024 · 0 comments
Open

File-based archive/extractor data loaders? #1247

Fil opened this issue Apr 16, 2024 · 0 comments
Labels
enhancement New feature or request question Further information is requested

Comments

@Fil
Copy link
Contributor

Fil commented Apr 16, 2024

In the discussion in #1222 we're thinking of using the filesystem to generate multiple files from a data loader. This could be better for huge files (e.g. video) than having to go through “zipping”; another benefit would be to make it easier for the user, who is used to saving many files, and not so much to load a zip module/extension (although we provide clear examples of how to do that).

An idea could be to use the .list extension (based on the research I did in #207, both .uri and .list seem to be usable, and .list is a better name); the data loader would write files "somewhere"[1], and return a list of (relative) file names.

[1] Would we provide this location (directory name) with an ENV variable? Would we create that directory? Would we leave it up to the script to create it—with risks of conflicts and no proper way to clean up?

@Fil Fil added enhancement New feature or request question Further information is requested labels Apr 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request question Further information is requested
Projects
None yet
Development

No branches or pull requests

1 participant