[MRG] provide an initial plugin architecture for sourmash that supports new signature saving & loading mechanisms #2428
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov Report
@@ Coverage Diff @@
## latest #2428 +/- ##
==========================================
+ Coverage 84.08% 92.26% +8.17%
==========================================
Files 130 103 -27
Lines 15044 11557 -3487
Branches 2403 2416 +13
==========================================
- Hits 12650 10663 -1987
+ Misses 2098 597 -1501
- Partials 296 297 +1
Flags with carried forward coverage won't be shown. Click here to find out more.
📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more |
ctb
changed the title
ctb
changed the title
|
Ready for review @sourmash-bio/devs! The underlying code is pretty simple and (hopefully) not that ugly - comments welcome! A high level look at things to see what could be improved in terms of documentation, templates, etc. would be very welcome, too. |
bluegenes
approved these changes
Jan 7, 2023
There was a problem hiding this comment.
lgtm, and I got the load_urls plugin working with this PR!
I think the documentation could benefit from a "Using existing plugins" section above the templates and examples. Just a simple explanation, e.g. install the plugin wherever you have sourmash installed (e.g. preferably in a conda env) and then try running the command. This may seem obvious to anyone familiar with plugins, but could be helpful for the rest of us.
|
oops, I forgot approval immediately merges now - will use |
|
Oooh, excellent point! I'll add that in #2438! |
|
jinx! and in this case I enabled automatic merge, it doesn't happen by default. |
ctb
added a commit
that referenced
this pull request
Mar 1, 2023
Add command-line plug-in interface, per #1353. This PR builds on #2428 to provide support for adding new sourmash subcommands, under `sourmash scripts <cmd>`. The CLI plugin interface looks like this on the client side - ``` class cmd_somewhat: command = 'somewhat' description = "does a thing" def __init__(self, subparser): super().__init__(p) # add arguments etc here debug_literal('RUNNING cmd_somewhat.__init__') subparser.add_argument('foo', type=int) def main(self, args): super().main(args) # what we actually run. print('RUNNING cmd', self, args) print('XYZ somewhat', args.foo) print(f'2*{args.foo} is {2*args.foo}') ``` Adding this plugin enables: ``` % sourmash scripts -h == This is sourmash version 4.6.1. == == Please cite Brown and Irber (2016), doi:10.21105/joss.00027. == usage: scripts [-h] optional arguments: -h, --help show this help message and exit extension commands: sourmash scripts somewhat --help - do a thing ``` and ``` % sourmash scripts somewhat 5 == This is sourmash version 4.6.1. == == Please cite Brown and Irber (2016), doi:10.21105/joss.00027. == ... 2*5 is 10 ``` which is prety cool 🎉 ## Alternatives to `script` could be `sourmash ext` or `sourmash plugin`?
Merged
ctb
added a commit
that referenced
this pull request
Mar 3, 2023
# sourmash release 4.7.0 Major new features: * provide an initial plugin architecture for sourmash that supports new signature saving & loading mechanisms (#2428) * add plugin support for new command-line subcommands (#2438) * debias all containment values (#2243) Minor new features: * Use RankLineageInfo to simplify reading lineages (#2467) * store taxids in lineageDB (#2466) * Use new tax classes for taxonomic summarization (#2443) * add tax summarization dataclasses for safety and flexibility (#2439) * add `--scaled` to sourmash compare (#2414) * replace `lca_utils.LineagePair` with `tax_utils.LineagePair` (#2441) * Add new classes for lineage manipulation (#2437) Cleanup and documentation updates: * ReadTheDocs updates (#2445) * update `sourmash compare` command-line docs (#2400) Developer updates: * fix python tests by bumping tox and pip cache versions (#2424) * Update sphinx requirement from <6,>=4.4.0 to >=4.4.0,<7 (#2430) * Build: replace milksnake with maturin (#2393) * importlib_metadata is a dependency on old Python versions (#2484) * Release docs: use two separate sed commands (#2483) * minor fixes to release behavior (#2479) * Use screed and maturin from nixpkgs in `flake.nix` (#2481) * update release procedure after v4.6.0 and v4.6.1 (#2386) * Update makefile and docs (#2432) Dependabot updates: * Bump once_cell from 1.17.0 to 1.17.1 (#2488) * Bump ouroboros from 0.15.5 to 0.15.6 (#2487) * Bump memmap2 from 0.5.8 to 0.5.9 (#2486) * Bump supercharge/redis-github-action from 1.4.0 to 1.5.0 (#2485) * Bump proptest from 1.0.0 to 1.1.0 (#2460) * Bump web-sys from 0.3.60 to 0.3.61 (#2461) * Bump serde_json from 1.0.91 to 1.0.93 (#2471) * Bump wasm-bindgen-test from 0.3.33 to 0.3.34 (#2463) * Bump cachix/install-nix-action from 18 to 19 (#2459) * Bump wasm-bindgen from 0.2.83 to 0.2.84 (#2464) * Bump typed-builder from 0.11.0 to 0.12.0 (#2451) * Bump bumpalo from 3.9.1 to 3.12.0 (#2450) * Bump pypa/cibuildwheel from 2.11.4 to 2.12.0 (#2447) * Bump bzip2 from 0.4.3 to 0.4.4 (#2444) * Bump once_cell from 1.14.0 to 1.17.0 (#2429) * Bump serde from 1.0.151 to 1.0.152 (#2423) * Bump pypa/cibuildwheel from 2.11.3 to 2.11.4 (#2422) * Bump serde_json from 1.0.89 to 1.0.91 (#2418) * Bump serde from 1.0.150 to 1.0.151 (#2419) * Bump thiserror from 1.0.37 to 1.0.38 (#2417) * Bump finch from 0.4.3 to 0.5.0 (#2416) * Bump rayon from 1.6.0 to 1.6.1 (#2404) * Bump serde from 1.0.149 to 1.0.150 (#2403) * Bump pypa/cibuildwheel from 2.11.2 to 2.11.3 (#2402) * Bump serde from 1.0.148 to 1.0.149 (#2397) * Bump capnp from 0.14.5 to 0.14.11 (#2396)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Implement support for
load_fromandsave_toplugins viaimportlib.metadataentry points.This supports a few of the plugins suggested in #1353
I am nominating this as an experimental feature that is not under semantic versioning/not public yet.
Documentation page here, in dev_plugins.html.
A template repo for new plugins is at https://github.com/sourmash-bio/sourmash_plugin_template.
Implementation/this PR
This PR refactors the
_load_databaseloading andSaveSignaturesToLocationsaving code to build a prioritized list of functions to try in order, and then adds hooks in via the newsourmash.pluginsmodule that insert additional loading/saving functions into that list.This PR also moves the current saving/loading functions out of
sourmash.sourmash_argsinto thesourmash.save_loadsubmodule, and simplifies the code a bit.Example plugins:
.avrosigto write.Specific TODOs:
For later: