Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[python] Less-in-core ingest from H5AD [main] #640

Merged
merged 3 commits into from
Jan 4, 2023

Conversation

johnkerl
Copy link
Member

@johnkerl johnkerl commented Jan 3, 2023

Context: Shortcut story 24294 and #641

See also #627 which is for the main-old branch.

@codecov-commenter
Copy link

codecov-commenter commented Jan 3, 2023

Codecov Report

Base: 66.85% // Head: 66.77% // Decreases project coverage by -0.07% ⚠️

Coverage data is based on head (6b59b71) compared to base (9384934).
Patch coverage: 75.00% of modified lines in pull request are covered.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #640      +/-   ##
==========================================
- Coverage   66.85%   66.77%   -0.08%     
==========================================
  Files          64       64              
  Lines        3382     3362      -20     
==========================================
- Hits         2261     2245      -16     
+ Misses       1121     1117       -4     
Flag Coverage Δ
python 74.76% <75.00%> (-0.06%) ⬇️
r 57.02% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
apis/python/src/tiledbsoma/io.py 65.51% <75.00%> (ø)
apis/python/src/tiledbsoma/util_ann.py 10.77% <75.00%> (-7.41%) ⬇️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

@@ -370,7 +370,6 @@ def from_anndata(

anndata.obs_names_make_unique()
anndata.var_names_make_unique()
anndata = util_ann._decategoricalize(anndata)
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since we got anndata = ad.read_h5ad(h5ad_file_name, "r") with the "r" this entire data structure is no longer reconstructible (below). So we split out the decategoricalizations to where they're needed.

Copy link
Member

@aaronwolen aaronwolen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚀

@johnkerl johnkerl merged commit 3850e9a into main Jan 4, 2023
@johnkerl johnkerl deleted the kerl/ingestor-memory-reduction-main branch January 4, 2023 14:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants