Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[MRG] refactor ZipFileLinearCollection and SaveSignatures_ZipFile to use ZipStorage #1598

Merged
merged 15 commits into from
Oct 26, 2021

Conversation

ctb
Copy link
Contributor

@ctb ctb commented Jun 16, 2021

This refactors ZipFileLinearCollection (reading) and SaveSignatures_ZipFile (creation) to use ZipStorage underneath, per #1441.

Fixes #1441.

In detail, this PR:

  • in index.py, replaces most direct access to files via ZipFileLinearIndex.zf (a zipfile.ZipFile object) with access via sbt_storage.ZipStorage;
  • in sourmash_args.py, changes SaveSignatures_ZipFile to use ZipStorage underneath;
  • we also extend SaveSignatures_ZipFile to append in situations where there is an existing zipfile + manifest;
  • updates sbt_storage.py:ZipStorage to play more nicely with others - basically, a few new situations are encountered so we upgrade code accordingly;
  • fixes a bug in manifest.py where with_abundance was incorrectly specified to be an integer - turns out that int works on a boolean False, but not a string 'False' ;)
  • updates and adds tests for new behaviors.

Notes

This is a mostly-clean PR, with one exception - in ZipFileLinearIndex, we have to break Storage encapsulation and go directly to the zipfile to get a list of all files, in cases where there is no manifest. (Listing all files is not an option supported through the current Storage interface.) There are two options for fixing this -

  1. provide a way to list all (relevant) files in a Storage. It's not clear to me that this is a good idea, because a Storage may contain many files and many signatures.
  2. require manifests when using zipfile collections. This seems fine to me, but would (I think) require a 5.0 release.

Ultimately I think we will probably want to implement both, but not today :)

(In this PR, we do require manifests in order to append signatures to an existing zipfile, but they are not required to read from an existing zipfile.)

TODO:

  • test on some wonky, hand-constructed .zip files

@ctb
Copy link
Contributor Author

ctb commented Jun 16, 2021

@luizirber this is another PR you can look at; it doesn't have a lot of extraneous picklist and manifest stuff in it.

@codecov
Copy link

codecov bot commented Jun 16, 2021

Codecov Report

Merging #1598 (9c530a5) into add/picklist_zf_manifests (39abe57) will increase coverage by 0.54%.
The diff coverage is 93.33%.

Impacted file tree graph

@@                      Coverage Diff                      @@
##           add/picklist_zf_manifests    #1598      +/-   ##
=============================================================
+ Coverage                      87.53%   88.08%   +0.54%     
=============================================================
  Files                             77       77              
  Lines                           6933     6942       +9     
  Branches                        1251     1250       -1     
=============================================================
+ Hits                            6069     6115      +46     
+ Misses                           637      600      -37     
  Partials                         227      227              
Flag Coverage Δ
python 88.08% <93.33%> (+0.54%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
src/sourmash/sourmash_args.py 92.41% <88.00%> (-1.31%) ⬇️
src/sourmash/index.py 88.92% <100.00%> (+6.88%) ⬆️
src/sourmash/sbt_storage.py 90.86% <100.00%> (+0.96%) ⬆️
src/sourmash/signature.py 92.38% <0.00%> (+0.95%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 39abe57...9c530a5. Read the comment docs.

@codecov-commenter
Copy link

codecov-commenter commented Jun 18, 2021

Codecov Report

Merging #1598 (4bfe8a9) into latest (5accf22) will increase coverage by 6.68%.
The diff coverage is 91.93%.

Impacted file tree graph

@@            Coverage Diff             @@
##           latest    #1598      +/-   ##
==========================================
+ Coverage   83.42%   90.11%   +6.68%     
==========================================
  Files         113       87      -26     
  Lines       12107     8413    -3694     
  Branches     1555     1557       +2     
==========================================
- Hits        10100     7581    -2519     
+ Misses       1749      574    -1175     
  Partials      258      258              
Flag Coverage Δ
python 90.11% <91.93%> (+<0.01%) ⬆️
rust ?

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
src/sourmash/sbt_storage.py 90.95% <66.66%> (+0.58%) ⬆️
src/sourmash/sourmash_args.py 92.70% <90.00%> (-1.12%) ⬇️
src/sourmash/index.py 95.95% <100.00%> (+0.41%) ⬆️
src/sourmash/manifest.py 90.26% <100.00%> (ø)
src/core/src/cmd.rs
src/core/src/lib.rs
src/core/src/signature.rs
src/core/tests/minhash.rs
src/core/src/index/mod.rs
... and 24 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5accf22...4bfe8a9. Read the comment docs.

Base automatically changed from add/picklist_zf_manifests to latest June 24, 2021 18:07
ctb added 2 commits July 5, 2021 06:13
commit 3268907
Author: C. Titus Brown <titus@idyll.org>
Date:   Fri Jul 2 05:38:56 2021 -0700

    remove leftover merge code

commit 3b53de9
Merge: 0f7dc81 21f5e63
Author: C. Titus Brown <titus@idyll.org>
Date:   Fri Jul 2 05:35:54 2021 -0700

    Merge branch 'latest' of https://github.com/sourmash-bio/sourmash into add/zipfile_use_storage

commit 0f7dc81
Author: C. Titus Brown <titus@idyll.org>
Date:   Fri Jun 25 10:45:34 2021 -0700

    fix error message

commit fc0c6fe
Merge: 65646fb a5a52b1
Author: C. Titus Brown <titus@idyll.org>
Date:   Fri Jun 25 10:17:33 2021 -0700

    Merge branch 'latest' of https://github.com/sourmash-bio/sourmash into add/zipfile_use_storage

commit 65646fb
Author: C. Titus Brown <titus@idyll.org>
Date:   Thu Jun 24 11:35:55 2021 -0700

    fix merge

commit dec537a
Merge: c039fd6 9dbd8b5
Author: C. Titus Brown <titus@idyll.org>
Date:   Thu Jun 24 11:31:47 2021 -0700

    Merge branch 'latest' of https://github.com/sourmash-bio/sourmash into add/zipfile_use_storage

commit c039fd6
Merge: 89fad20 8cc96cd
Author: C. Titus Brown <titus@idyll.org>
Date:   Wed Jun 23 17:48:50 2021 -0700

    Merge branch 'add/picklist_zf_manifests' into add/zipfile_use_storage

commit 8cc96cd
Author: C. Titus Brown <titus@idyll.org>
Date:   Wed Jun 23 15:43:19 2021 -0700

    fix tests for a CLEAN test-data/prot/ directory

commit 5e49336
Author: C. Titus Brown <titus@idyll.org>
Date:   Wed Jun 23 10:32:49 2021 -0700

    update docstring

commit 2438d90
Merge: 873592d 0ff54e7
Author: C. Titus Brown <titus@idyll.org>
Date:   Wed Jun 23 10:34:23 2021 -0700

    Merge branch 'latest' of https://github.com/sourmash-bio/sourmash into add/picklist_zf_manifests

commit 89fad20
Merge: c6a8ad7 0ff54e7
Author: C. Titus Brown <titus@idyll.org>
Date:   Wed Jun 23 10:33:13 2021 -0700

    Merge branch 'latest' of https://github.com/sourmash-bio/sourmash into add/zipfile_use_storage

commit c6a8ad7
Author: C. Titus Brown <titus@idyll.org>
Date:   Wed Jun 23 10:32:49 2021 -0700

    update docstring

commit bd753d2
Author: C. Titus Brown <titus@idyll.org>
Date:   Wed Jun 23 09:04:56 2021 -0700

    fix a few more things

commit 86ac7ad
Merge: 41438a6 873592d
Author: C. Titus Brown <titus@idyll.org>
Date:   Wed Jun 23 09:03:35 2021 -0700

    Merge branch 'add/picklist_zf_manifests' into add/zipfile_use_storage

commit 873592d
Merge: b6d5547 1992de9
Author: C. Titus Brown <titus@idyll.org>
Date:   Wed Jun 23 08:48:31 2021 -0700

    Merge branch 'latest' of https://github.com/sourmash-bio/sourmash into add/picklist_zf_manifests

commit b6d5547
Author: C. Titus Brown <titus@idyll.org>
Date:   Wed Jun 23 08:47:02 2021 -0700

    add test for multiple selects

commit 8ebac0d
Author: C. Titus Brown <titus@idyll.org>
Date:   Wed Jun 23 08:37:25 2021 -0700

    remove print statements

commit 701878b
Author: C. Titus Brown <titus@idyll.org>
Date:   Wed Jun 23 08:36:45 2021 -0700

    update test files to have manifest, update tests

commit faad6ee
Author: C. Titus Brown <titus@idyll.org>
Date:   Wed Jun 23 08:17:40 2021 -0700

    don't test manifest content

commit 44aba07
Author: C. Titus Brown <titus@idyll.org>
Date:   Wed Jun 23 08:08:58 2021 -0700

    more refactor zipfile select

commit 61ce0f2
Author: C. Titus Brown <titus@idyll.org>
Date:   Wed Jun 23 07:37:53 2021 -0700

    refactor zipfile select

commit 5879ff2
Author: C. Titus Brown <titus@idyll.org>
Date:   Wed Jun 23 07:37:28 2021 -0700

    check compatibility in MinHash.intersection_and_union

commit e1c44a6
Merge: 6c1f9da d473199
Author: C. Titus Brown <titus@idyll.org>
Date:   Wed Jun 23 07:14:38 2021 -0700

    Merge branch 'latest' of http://github.com/sourmash-bio/sourmash into add/picklist_zf_manifests

commit 6c1f9da
Author: C. Titus Brown <titus@idyll.org>
Date:   Wed Jun 23 07:12:58 2021 -0700

    more manifest testing for zipfiles

commit 1b2cf73
Author: C. Titus Brown <titus@idyll.org>
Date:   Wed Jun 23 07:04:47 2021 -0700

    add use_manifest fixture, refactor manifest loading

commit 38ec792
Author: C. Titus Brown <titus@idyll.org>
Date:   Wed Jun 23 06:50:13 2021 -0700

    add sig manifest tests for other file types

commit 6905d40
Author: C. Titus Brown <titus@idyll.org>
Date:   Wed Jun 23 06:42:51 2021 -0700

    update sig manifest to error when manifests cannot be generated

commit fa47667
Author: C. Titus Brown <titus@idyll.org>
Date:   Wed Jun 23 06:19:51 2021 -0700

    rename signatures_with_internal to _signatures_with_internal

commit 096b141
Author: C. Titus Brown <titus@idyll.org>
Date:   Wed Jun 23 05:46:32 2021 -0700

    add manifests to default zip collection output

commit 99199ee
Author: C. Titus Brown <titus@idyll.org>
Date:   Wed Jun 23 05:32:52 2021 -0700

    move manifest stuff to manifest.py

commit 0adee52
Author: C. Titus Brown <titus@idyll.org>
Date:   Wed Jun 23 05:28:54 2021 -0700

    remove print

commit 83e387e
Merge: fe83b68 9bb6a9b
Author: C. Titus Brown <titus@idyll.org>
Date:   Tue Jun 22 18:15:48 2021 -0700

    Merge branch 'add/picklist_manifests_sbt' into add/picklist_zf_manifests

commit fe83b68
Author: C. Titus Brown <titus@idyll.org>
Date:   Tue Jun 22 18:07:23 2021 -0700

    revert collection to multiindex

commit 60a6eec
Author: C. Titus Brown <titus@idyll.org>
Date:   Tue Jun 22 14:08:34 2021 -0700

    change LoadedCollection back over to MultiIndex; remove LazyMultiIndex

commit 9bb6a9b
Author: C. Titus Brown <titus@idyll.org>
Date:   Tue Jun 22 12:53:40 2021 -0700

    fix header writing

commit 7486871
Merge: 4221fc9 c3f1a3d
Author: C. Titus Brown <titus@idyll.org>
Date:   Tue Jun 22 12:53:03 2021 -0700

    Merge branch 'add/picklist_zf_manifests' into add/picklist_manifests_sbt

commit 41438a6
Author: C. Titus Brown <titus@idyll.org>
Date:   Tue Jun 22 12:52:32 2021 -0700

    fix header writing

commit 4026855
Merge: 6b18439 c3f1a3d
Author: C. Titus Brown <titus@idyll.org>
Date:   Tue Jun 22 12:48:33 2021 -0700

    Merge branch 'add/picklist_zf_manifests' into add/zipfile_use_storage

commit c3f1a3d
Author: C. Titus Brown <titus@idyll.org>
Date:   Tue Jun 22 12:42:38 2021 -0700

    reverse order of adding to seen set

commit 71b81ed
Author: C. Titus Brown <titus@idyll.org>
Date:   Tue Jun 22 12:41:33 2021 -0700

    add docstring

commit ed5fb7a
Author: C. Titus Brown <titus@idyll.org>
Date:   Tue Jun 22 12:40:43 2021 -0700

    rename matches_siginfo to matches_manifest_row

commit 2756e7d
Author: C. Titus Brown <titus@idyll.org>
Date:   Tue Jun 22 12:28:37 2021 -0700

    add save/load test

commit ba2e53c
Merge: c243b0e c04f137
Author: C. Titus Brown <titus@idyll.org>
Date:   Tue Jun 22 11:12:07 2021 -0700

    Merge branch 'latest' of github.com:dib-lab/sourmash into add/picklist_zf_manifests

commit c243b0e
Author: C. Titus Brown <titus@idyll.org>
Date:   Tue Jun 22 11:10:20 2021 -0700

    add manifest tests

commit e301645
Author: C. Titus Brown <titus@idyll.org>
Date:   Tue Jun 22 10:07:35 2021 -0700

    add a test for sig manifest

commit e315c90
Merge: d95813e 0814bcc
Author: C. Titus Brown <titus@idyll.org>
Date:   Tue Jun 22 09:58:29 2021 -0700

    Merge branch 'latest' of github.com:dib-lab/sourmash into add/picklist_zf_manifests

commit d95813e
Author: C. Titus Brown <titus@idyll.org>
Date:   Sat Jun 19 13:52:50 2021 -0700

    add manifest versions

commit 4221fc9
Merge: d4a9a2e 31018df
Author: C. Titus Brown <titus@idyll.org>
Date:   Fri Jun 18 05:47:28 2021 -0700

    Merge branch 'add/picklist_zf_manifests' into add/picklist_manifests_sbt

commit 6b18439
Merge: 9ff0eab 31018df
Author: C. Titus Brown <titus@idyll.org>
Date:   Fri Jun 18 05:46:39 2021 -0700

    Merge branch 'add/picklist_zf_manifests' into add/zipfile_use_storage

commit 31018df
Merge: 9e46ff8 74de59a
Author: C. Titus Brown <titus@idyll.org>
Date:   Fri Jun 18 05:44:38 2021 -0700

    Merge branch 'latest' of github.com:dib-lab/sourmash into add/picklist_zf_manifests

commit 9ff0eab
Merge: 9c530a5 9e46ff8
Author: C. Titus Brown <titus@idyll.org>
Date:   Thu Jun 17 14:02:08 2021 -0700

    Merge branch 'add/picklist_zf_manifests' into add/zipfile_use_storage

commit d4a9a2e
Author: C. Titus Brown <titus@idyll.org>
Date:   Thu Jun 17 14:01:48 2021 -0700

    fix test for manifests

commit 2da0085
Merge: a7e153a 9e46ff8
Author: C. Titus Brown <titus@idyll.org>
Date:   Thu Jun 17 14:01:27 2021 -0700

    Merge branch 'add/picklist_zf_manifests' into add/picklist_manifests_sbt

commit 9e46ff8
Author: C. Titus Brown <titus@idyll.org>
Date:   Thu Jun 17 13:55:21 2021 -0700

    cleanup of comments etc.

commit e1e367a
Author: C. Titus Brown <titus@idyll.org>
Date:   Thu Jun 17 13:49:21 2021 -0700

    remove @ctb comments

commit 5cad5ff
Merge: 54ea3f9 8812142
Author: C. Titus Brown <titus@idyll.org>
Date:   Thu Jun 17 12:22:46 2021 -0700

    Merge branch 'add/picklist_selectors' into add/picklist_zf_manifests

commit 8812142
Author: C. Titus Brown <titus@idyll.org>
Date:   Thu Jun 17 12:20:45 2021 -0700

    further attempt to fix test

commit 54ea3f9
Author: C. Titus Brown <titus@idyll.org>
Date:   Thu Jun 17 12:17:27 2021 -0700

    only match picklist at end of 'select'

commit 122d043
Merge: f697ec4 de6f3c4
Author: C. Titus Brown <titus@idyll.org>
Date:   Thu Jun 17 11:40:39 2021 -0700

    Merge branch 'add/picklist_selectors' into add/picklist_zf_manifests

commit de6f3c4
Author: C. Titus Brown <titus@idyll.org>
Date:   Thu Jun 17 11:38:35 2021 -0700

    remove order dependence from test

commit f697ec4
Author: C. Titus Brown <titus@idyll.org>
Date:   Thu Jun 17 09:31:18 2021 -0700

    fix coltypes

commit 7937292
Merge: bba101c 4d156e9
Author: C. Titus Brown <titus@idyll.org>
Date:   Thu Jun 17 09:24:38 2021 -0700

    Merge branch 'add/picklist_selectors' into add/picklist_zf_manifests

commit 4d156e9
Author: C. Titus Brown <titus@idyll.org>
Date:   Thu Jun 17 09:13:52 2021 -0700

    add docs

commit ab286cf
Author: C. Titus Brown <titus@idyll.org>
Date:   Thu Jun 17 08:50:57 2021 -0700

    remove debugging print

commit c965648
Author: C. Titus Brown <titus@idyll.org>
Date:   Thu Jun 17 08:44:32 2021 -0700

    add a test for using prefetch CSV as picklist

commit ca6ea4f
Author: C. Titus Brown <titus@idyll.org>
Date:   Thu Jun 17 08:34:08 2021 -0700

    add picklist test that checks indexing-and-then-search == index

commit bba101c
Merge: 39abe57 ba5c8bc
Author: C. Titus Brown <titus@idyll.org>
Date:   Thu Jun 17 08:13:05 2021 -0700

    Merge branch 'add/picklist_selectors' into add/picklist_zf_manifests

commit ba5c8bc
Author: C. Titus Brown <titus@idyll.org>
Date:   Thu Jun 17 07:47:13 2021 -0700

    block multiple picklists on SBTs and LCAs, for now

commit a074127
Author: C. Titus Brown <titus@idyll.org>
Date:   Thu Jun 17 07:38:45 2021 -0700

    add picklists to lca index

commit a0335a3
Author: C. Titus Brown <titus@idyll.org>
Date:   Thu Jun 17 07:32:10 2021 -0700

    add picklists to sourmash compare

commit c0e5781
Author: C. Titus Brown <titus@idyll.org>
Date:   Thu Jun 17 07:25:59 2021 -0700

    add picklists to prefetch

commit 7a30b20
Author: C. Titus Brown <titus@idyll.org>
Date:   Thu Jun 17 07:12:17 2021 -0700

    add picklists and tests for search, gather, index

commit ced72d2
Author: C. Titus Brown <titus@idyll.org>
Date:   Thu Jun 17 06:25:01 2021 -0700

    add picklist args throughout, eek.

commit 984a557
Author: C. Titus Brown <titus@idyll.org>
Date:   Thu Jun 17 06:14:23 2021 -0700

    fix space

commit fddf141
Author: C. Titus Brown <titus@idyll.org>
Date:   Thu Jun 17 06:14:09 2021 -0700

    move picklist reporting into sourmash_args

commit b3c6bb9
Author: C. Titus Brown <titus@idyll.org>
Date:   Thu Jun 17 06:09:40 2021 -0700

    move picklist.py from sourmash.sig into sourmash

commit 21ce4b7
Author: C. Titus Brown <titus@idyll.org>
Date:   Wed Jun 16 15:41:51 2021 -0700

    fix tests for new SignaturePicklist

commit b8f4bb8
Merge: 8e5fb8d b787b75
Author: C. Titus Brown <titus@idyll.org>
Date:   Wed Jun 16 15:40:10 2021 -0700

    Merge branch 'latest' of github.com:dib-lab/sourmash into add/picklist_selectors

commit 8e5fb8d
Merge: 5ac4671 04c209c
Author: C. Titus Brown <titus@idyll.org>
Date:   Wed Jun 16 12:05:31 2021 -0700

    Merge branch 'add/picklist' into add/picklist_selectors

commit 04c209c
Author: C. Titus Brown <titus@idyll.org>
Date:   Wed Jun 16 11:27:21 2021 -0700

    remove comment

commit 14b87d4
Author: C. Titus Brown <titus@idyll.org>
Date:   Wed Jun 16 11:18:32 2021 -0700

    trap errors and be nice to users

commit 4f8e20c
Author: C. Titus Brown <titus@idyll.org>
Date:   Wed Jun 16 11:16:49 2021 -0700

    cover untested code with tests

commit 8f65f22
Author: C. Titus Brown <titus@idyll.org>
Date:   Wed Jun 16 10:17:50 2021 -0700

    test with --md5 selector

commit 9d60e32
Author: C. Titus Brown <titus@idyll.org>
Date:   Wed Jun 16 10:15:06 2021 -0700

    documentation

commit 3d23d87
Author: C. Titus Brown <titus@idyll.org>
Date:   Wed Jun 16 09:39:59 2021 -0700

    add --picklist-require-all &c

commit 14a88a7
Author: C. Titus Brown <titus@idyll.org>
Date:   Wed Jun 16 09:34:43 2021 -0700

    verify output

commit 207a813
Author: C. Titus Brown <titus@idyll.org>
Date:   Wed Jun 16 09:31:22 2021 -0700

    more picklist tests

commit 9b50748
Author: C. Titus Brown <titus@idyll.org>
Date:   Wed Jun 16 09:19:40 2021 -0700

    fix tests :)

commit aaa4548
Author: C. Titus Brown <titus@idyll.org>
Date:   Wed Jun 16 09:18:08 2021 -0700

    update comments, constructor, etc.

commit a7e153a
Author: C. Titus Brown <titus@idyll.org>
Date:   Wed Jun 16 07:22:39 2021 -0700

    fix tests

commit 9c530a5
Author: C. Titus Brown <titus@idyll.org>
Date:   Wed Jun 16 07:12:46 2021 -0700

    add comment about Storage encapsulation

commit 48fd900
Author: C. Titus Brown <titus@idyll.org>
Date:   Wed Jun 16 07:07:17 2021 -0700

    all tests pass, w00t

commit d6a48c1
Author: C. Titus Brown <titus@idyll.org>
Date:   Wed Jun 16 07:01:34 2021 -0700

    refactor ZipFileLinearIndex to use ZipStorage underneath

commit 5a185bb
Author: C. Titus Brown <titus@idyll.org>
Date:   Wed Jun 16 06:45:42 2021 -0700

    change internal zipfile writing to use ZipStorage

commit c356842
Author: C. Titus Brown <titus@idyll.org>
Date:   Wed Jun 16 06:12:24 2021 -0700

    done, I think?

commit 75dc079
Merge: 1dd8170 39abe57
Author: C. Titus Brown <titus@idyll.org>
Date:   Wed Jun 16 05:25:38 2021 -0700

    Merge branch 'add/picklist_zf_manifests' into add/picklist_manifests_sbt

commit 39abe57
Author: C. Titus Brown <titus@idyll.org>
Date:   Wed Jun 16 05:23:42 2021 -0700

    CSV output function

commit 1dd8170
Author: C. Titus Brown <titus@idyll.org>
Date:   Wed Jun 16 05:20:35 2021 -0700

    add manifests to SBTs

commit 72d8497
Author: C. Titus Brown <titus@idyll.org>
Date:   Wed Jun 16 04:23:48 2021 -0700

    move manifest stuff into manifest class

commit a4057e6
Author: C. Titus Brown <titus@idyll.org>
Date:   Tue Jun 15 10:33:09 2021 -0700

    create LazyMultiIndex

commit 730a717
Author: C. Titus Brown <titus@idyll.org>
Date:   Tue Jun 15 09:40:45 2021 -0700

    more cleanup and docs

commit 230c793
Author: C. Titus Brown <titus@idyll.org>
Date:   Tue Jun 15 09:28:09 2021 -0700

    cleanup and simplification of ZipFile stuff

commit 8a8c3b2
Author: C. Titus Brown <titus@idyll.org>
Date:   Tue Jun 15 08:54:23 2021 -0700

    shift signature metadata matching from manifests over to picklist

commit ab0fc0e
Author: C. Titus Brown <titus@idyll.org>
Date:   Tue Jun 15 08:45:06 2021 -0700

    misc cleanup

commit c3b6fc0
Author: C. Titus Brown <titus@idyll.org>
Date:   Tue Jun 15 08:36:08 2021 -0700

    more cleanup

commit 509eb45
Author: C. Titus Brown <titus@idyll.org>
Date:   Tue Jun 15 08:23:41 2021 -0700

    remove MultiIndex

commit af5eb86
Author: C. Titus Brown <titus@idyll.org>
Date:   Tue Jun 15 08:09:53 2021 -0700

    fix test names for new LoadedCollection

commit c6cb1af
Author: C. Titus Brown <titus@idyll.org>
Date:   Tue Jun 15 08:08:46 2021 -0700

    fix all the tests

commit 915f847
Author: C. Titus Brown <titus@idyll.org>
Date:   Tue Jun 15 07:48:35 2021 -0700

    cleanup/simplification of LoadedCollection

commit be9ef77
Author: C. Titus Brown <titus@idyll.org>
Date:   Tue Jun 15 07:41:48 2021 -0700

    create LoadedCollection to replace MultiIndex non-lazy loading

commit 3c0c9cf
Author: C. Titus Brown <titus@idyll.org>
Date:   Tue Jun 15 07:17:08 2021 -0700

    try making manifests obligatory for MultiIndex

commit 23c1531
Merge: 67a9be1 5ac4671
Author: C. Titus Brown <titus@idyll.org>
Date:   Mon Jun 14 13:35:20 2021 -0700

    Merge branch 'add/picklist_selectors' into add/picklist_zf_manifests

commit 5ac4671
Merge: a88b66d 031522c
Author: C. Titus Brown <titus@idyll.org>
Date:   Mon Jun 14 13:35:04 2021 -0700

    Merge branch 'add/picklist' into add/picklist_selectors

commit 031522c
Merge: 3c05f95 ff75ec0
Author: C. Titus Brown <titus@idyll.org>
Date:   Mon Jun 14 13:34:37 2021 -0700

    Merge branch 'latest' of github.com:dib-lab/sourmash into add/picklist

commit 67a9be1
Author: C. Titus Brown <titus@idyll.org>
Date:   Mon Jun 14 11:46:07 2021 -0700

    more comment

commit 1d7e0cf
Author: C. Titus Brown <titus@idyll.org>
Date:   Mon Jun 14 11:41:22 2021 -0700

    update comment about picklist.found

commit 2f2269b
Author: C. Titus Brown <titus@idyll.org>
Date:   Mon Jun 14 11:33:56 2021 -0700

    work through manifests for MultiIndex

commit cb8e28d
Author: C. Titus Brown <titus@idyll.org>
Date:   Mon Jun 14 06:34:34 2021 -0700

    get started adding manifests to MultiIndex

commit 01d33fc
Author: C. Titus Brown <titus@idyll.org>
Date:   Mon Jun 14 06:21:05 2021 -0700

    provide 'select' more generically on manifests

commit 17b9576
Author: C. Titus Brown <titus@idyll.org>
Date:   Mon Jun 14 06:12:39 2021 -0700

    build out a manifest class a bit

commit b2547f3
Author: C. Titus Brown <titus@idyll.org>
Date:   Sun Jun 13 20:20:28 2021 -0700

    add missing manifest CLI file

commit 14a5ee1
Author: C. Titus Brown <titus@idyll.org>
Date:   Sun Jun 13 08:59:00 2021 -0700

    hacky but functional manifest support

commit 6593a42
Author: C. Titus Brown <titus@idyll.org>
Date:   Sun Jun 13 08:37:05 2021 -0700

    try out manifests

commit e205e64
Author: C. Titus Brown <titus@idyll.org>
Date:   Sun Jun 13 07:39:47 2021 -0700

    special case md5 prefixes, for prefetch

commit b57b2b3
Author: C. Titus Brown <titus@idyll.org>
Date:   Sun Jun 13 07:23:46 2021 -0700

    support special picklist interactions with zipfile collections

commit a88b66d
Author: C. Titus Brown <titus@idyll.org>
Date:   Sun Jun 13 06:32:38 2021 -0700

    factor out picklist checks to 'passes_all_picklists' fn

commit 54407a3
Author: C. Titus Brown <titus@idyll.org>
Date:   Sat Jun 12 11:16:02 2021 -0700

    test 'Index.find' on picklists for SBTs and LCAs

commit 03cc61b
Merge: de6fc06 3c05f95
Author: C. Titus Brown <titus@idyll.org>
Date:   Sat Jun 12 10:48:59 2021 -0700

    Merge branch 'add/picklist' into add/picklist_selectors

commit 3c05f95
Author: C. Titus Brown <titus@idyll.org>
Date:   Sat Jun 12 10:48:15 2021 -0700

    split column_type out of SignaturePicklist a bit

commit 1bdf88e
Author: C. Titus Brown <titus@idyll.org>
Date:   Sat Jun 12 10:28:23 2021 -0700

    split pickfile out a little bit

commit de6fc06
Author: C. Titus Brown <titus@idyll.org>
Date:   Sat Jun 12 10:48:30 2021 -0700

    picklist tests for .signatures() methods on Index classes

commit def1933
Author: C. Titus Brown <titus@idyll.org>
Date:   Sat Jun 12 10:48:15 2021 -0700

    split column_type out of SignaturePicklist a bit

commit a817843
Author: C. Titus Brown <titus@idyll.org>
Date:   Sat Jun 12 10:28:23 2021 -0700

    split pickfile out a little bit

commit b1fc982
Author: C. Titus Brown <titus@idyll.org>
Date:   Sat Jun 12 10:01:56 2021 -0700

    add picklists to selectors

commit 74f31f5
Author: C. Titus Brown <titus@idyll.org>
Date:   Sat Jun 12 09:21:58 2021 -0700

    track found etc

commit 505b04f
Author: C. Titus Brown <titus@idyll.org>
Date:   Sat Jun 12 09:11:24 2021 -0700

    basic tests for picklist functionality

commit 3ecfb48
Author: C. Titus Brown <titus@idyll.org>
Date:   Sat Jun 12 08:55:28 2021 -0700

    integrate picklists into sourmash sig extract

commit bb794ec
Author: C. Titus Brown <titus@idyll.org>
Date:   Sat Jun 12 08:38:24 2021 -0700

    initial picklist implementation

commit 3a583a9
Author: C. Titus Brown <titus@idyll.org>
Date:   Sat Jun 12 07:36:48 2021 -0700

    clean up sourmash.sig submodule

commit 66b0599
Author: C. Titus Brown <titus@idyll.org>
Date:   Sat Jun 12 07:23:13 2021 -0700

    cleanup flakes errors

commit 0997834
Author: C. Titus Brown <titus@idyll.org>
Date:   Sat Jun 12 07:17:44 2021 -0700

    various cleanups of sourmash_args
@ctb ctb force-pushed the add/zipfile_use_storage branch from 3268907 to e586e9d Compare July 5, 2021 13:27
@ctb ctb changed the base branch from latest to update/multiindex July 5, 2021 13:29
Base automatically changed from update/multiindex to latest July 5, 2021 20:26
@ctb
Copy link
Contributor Author

ctb commented Oct 18, 2021

note to self: I might use this, or a related PR, to write some simple tests demonstrating FSStorage and ZipStorage as encouragement and partial documentation in support of #1752.

@ctb

This comment has been minimized.

@ctb ctb changed the title [EXP] refactor ZipFileLinearCollection and SaveSignatures_ZipFile to use ZipStorage [MRG] refactor ZipFileLinearCollection and SaveSignatures_ZipFile to use ZipStorage Oct 19, 2021
@ctb
Copy link
Contributor Author

ctb commented Oct 19, 2021

Ready for review @mr-eyes @luizirber!

@mr-eyes
Copy link
Member

mr-eyes commented Oct 20, 2021

Ready for review @mr-eyes @luizirber!

TFTM, I'm on it.

Copy link
Member

@mr-eyes mr-eyes left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@ctb ctb merged commit a1da79e into latest Oct 26, 2021
@ctb ctb deleted the add/zipfile_use_storage branch October 26, 2021 21:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

refactor ZipFileLinearIndex to use ZipStorage
3 participants