Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add flag to disable file downloads #505

Merged
merged 3 commits into from
Mar 1, 2023

Conversation

alexfikl
Copy link
Collaborator

@alexfikl alexfikl commented Feb 19, 2023

This is an improvement to #387 that tries to avoid downloading files all together (and all the additional traffic!) when the right conditions are met. For this, it adds:

  • a only_data flag to get_matching_importer_or_downloader
  • a only_data flag to get_matching_importer_by_name (new function in papis.utils)

so that they only call Importer.fetch_files if asked. It updates papis add and papis update to use these new functions.

It probably needs a bit more testing, but it seems pretty clean. @alejandrogallo What do you think?

@alexfikl alexfikl changed the title Downloaders: add flag to disable file downloads Downloaders: add setting to disable file downloads Feb 19, 2023
@alexfikl alexfikl changed the title Downloaders: add setting to disable file downloads Downloaders: add flag to disable file downloads Feb 28, 2023
@alexfikl alexfikl marked this pull request as draft February 28, 2023 19:56
@alexfikl alexfikl force-pushed the disable-downloads branch 2 times, most recently from 82a3901 to ca15281 Compare March 1, 2023 08:17
@alexfikl
Copy link
Collaborator Author

alexfikl commented Mar 1, 2023

Hm, this doesn't actually work as expected because a lot of importers just implement fetch in one go instead of fetch_data and fetch_files separately.

It's probably a good idea to clean that up in some future PRs so that it works nicely for all importers. Downloaders are a bit better at this because they implement get_document_url instead, so it can be skipped.

@alexfikl alexfikl marked this pull request as ready for review March 1, 2023 08:27
@alexfikl alexfikl force-pushed the disable-downloads branch 3 times, most recently from 3828e19 to 06525af Compare March 1, 2023 09:00
@alexfikl alexfikl changed the title Downloaders: add flag to disable file downloads Add flag to disable file downloads Mar 1, 2023
@alexfikl
Copy link
Collaborator Author

alexfikl commented Mar 1, 2023

This is pretty much just a refactor at this point, since it relies on the importers to properly split fetch_data and fetch_files, so going to merge it! 🚀

@alexfikl alexfikl merged commit 732ee03 into papis:master Mar 1, 2023
@alexfikl alexfikl deleted the disable-downloads branch March 1, 2023 18:30
@Twix53791
Copy link
Contributor

I did'nt find a flag to set this option when adding entries ? I missed something? So I added the flag myself. papis add --only-data will disable file download. --force-download takes priority over --only-data.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants