Conversation
…rts working with speleos
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #6 +/- ##
===========================================
+ Coverage 50.69% 70.67% +19.98%
===========================================
Files 71 77 +6
Lines 3470 3618 +148
===========================================
+ Hits 1759 2557 +798
+ Misses 1711 1061 -650
🚀 New features to boost your workflow:
|
…rts working with speleos
There was a problem hiding this comment.
Pull request overview
This PR expands DataBUS’s Neotoma upload/validation pipeline with new aeDNA entity abstractions, improves Excel/CSV ingestion (including FAIRe-style headers and value: constants), and adds extensive validator test coverage to prevent silent truncation and mapping errors.
Changes:
- Added aeDNA entity base + assay/library implementations and a shared validator helper for inserting “flat” aeDNA records.
- Enhanced XLSX parsing to detect FAIRe 3-row headers and better handle stacked taxon headers; improved parameter pulling to support
value:constants. - Hardened several validators (sample row-count guard, chronology scalar broadcast, speleothem monitoring coercion, skipping empty data rows) and added/expanded tests.
Reviewed changes
Copilot reviewed 33 out of 33 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| tests/test_valid_speleothem.py | New tests for speleothem + external speleothem validation behaviors. |
| tests/test_valid_sequence.py | New tests for aeDNA sequence validation and helper extractors/mapping. |
| tests/test_valid_sample.py | Adds tests covering the new row-count mismatch guard. |
| tests/test_valid_sample_age.py | Adds tests ensuring chronology-wide scalars don’t truncate per-sample inserts. |
| tests/test_valid_project.py | New tests for project/grant/contact/keyword extraction + linkage. |
| tests/test_valid_pbmodel.py | New tests for 210Pb model validator insertion/linkage behavior. |
| tests/test_valid_hiatus.py | New tests for hiatus clustering and insert behavior. |
| tests/test_valid_geopolitical_units.py | New tests for hierarchical geopolitical unit resolution and linking. |
| tests/test_valid_dataset_database.py | New tests for dataset-database lookup + insertion using value: constant. |
| tests/test_valid_assays.py | New tests for aeDNA assay/library validators using value: constants. |
| tests/test_insert_final.py | New tests for final submission insert step. |
| tests/test_helpers.py | Adds XLSX parsing regression tests (FAIRe headers, stacked taxon headers) and value: constant behavior tests. |
| src/DataBUS/Response.py | Extends Response with match flags (matched, doublematched). |
| src/DataBUS/Project.py | Refactors query variable names and adds parameter lists for grants/institutions. |
| src/DataBUS/neotomaValidator/valid_speleothem.py | Uses shared boolean coercion for monitoring, correctly handling None. |
| src/DataBUS/neotomaValidator/valid_sample.py | Adds guard to fail loudly on mismatched rowwise column lengths. |
| src/DataBUS/neotomaValidator/valid_sample_age.py | Broadcasts chronology-wide scalars across per-sample rows. |
| src/DataBUS/neotomaValidator/valid_project.py | Improves contact resolution via get_contacts + adds participant-linked message. |
| src/DataBUS/neotomaValidator/valid_data.py | Skips placeholder None entries and avoids inserting rows with value is None. |
| src/DataBUS/neotomaValidator/valid_assays.py | New validator for inserting AeDNAAssay (with assaytype resolution). |
| src/DataBUS/neotomaValidator/valid_aednalibraries.py | New validator for inserting AeDNALibrary linked to dataset + assay. |
| src/DataBUS/neotomaValidator/_aedna_entity_validator.py | New shared “flat aeDNA entity” validator helper. |
| src/DataBUS/neotomaValidator/init.py | Exposes new aeDNA validators at package level. |
| src/DataBUS/neotomaHelpers/utils.py | Adds convert_to_bool and validate_str_values; keeps _convert_bool alias. |
| src/DataBUS/neotomaHelpers/read_csv.py | Adds FAIRe header detection + stacked taxon header handling helpers. |
| src/DataBUS/neotomaHelpers/pull_params.py | Improves missing-column handling to allow value: constants. |
| src/DataBUS/neotomaHelpers/init.py | Exports convert_to_bool from helpers package. |
| src/DataBUS/AeDNAModel.py | Extends model params and clarifies naming for superseding models. |
| src/DataBUS/AeDNALibrary.py | Adds new AeDNALibrary entity. |
| src/DataBUS/AeDNAEntity.py | Adds aeDNA base entity with type validation and generic insert logic. |
| src/DataBUS/AeDNAAssay.py | Adds new AeDNAAssay entity. |
| src/DataBUS/init.py | Exposes new aeDNA classes and additional entities for import. |
| .github/workflows/codecov.yml | Adds Codecov configuration content (but currently placed as a workflow file). |
Suppressed comments (1)
src/DataBUS/neotomaHelpers/pull_params.py:73
- In
_process_value_entry, usingif not clean_valor:treats valid falsy values (e.g., numeric0/0.0after conversion, or booleanFalse) as “missing” and drops them. This will silently erase legitimate zero measurements and false flags. Use an explicit “missing” check (None/empty string/empty list) instead of truthiness (apply this both before and afterconvert_value_by_type).
clean_valor = val_entry["value"]
if not clean_valor:
if "taxonname" not in val_entry:
add_unit_inputs[param_name] = None
return
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request introduces several new classes and utility improvements to the DataBUS package, enhances Excel/CSV parsing, and improves type validation and boolean conversion. The most significant changes are the addition of base and domain-specific entity classes for aeDNA data, improvements to the Excel reading logic to support FAIRe templates, and utility function enhancements for type validation and value conversion.
New aeDNA Entity Classes
AeDNAEntitybase class for flat aeDNA tables, with field validation and database insertion logic (src/DataBUS/AeDNAEntity.py).AeDNAAssayandAeDNALibraryclasses, inheriting fromAeDNAEntity, to represent assay and library records with appropriate parameters and string representations (src/DataBUS/AeDNAAssay.py,src/DataBUS/AeDNALibrary.py). [1] [2]__init__.pyto expose new aeDNA classes for import (src/DataBUS/__init__.py). [1] [2]Excel/CSV Parsing Improvements
read_xlsxto detect and properly parse FAIRe 3-row headers, and refactored header handling for better flexibility (src/DataBUS/neotomaHelpers/read_csv.py). [1] [2] [3]_process_value_entryto handle missing columns more robustly, supporting direct value entries for non-standard headers (src/DataBUS/neotomaHelpers/pull_params.py).Utility Function Enhancements
validate_str_valuesfor string field validation and integrated it into entity initialization (src/DataBUS/neotomaHelpers/utils.py,src/DataBUS/AeDNAEntity.py). [1] [2]_convert_booltoconvert_to_bool, now handling more string variants and preservingNone(src/DataBUS/neotomaHelpers/utils.py,src/DataBUS/neotomaHelpers/__init__.py). [1] [2] [3]Project/Grant/Institution Handling
Projectclass to use clearer query variable names for project, grant, and institution insertions, and added parameter lists for grants and institutions (src/DataBUS/Project.py). [1] [2] [3] [4] [5] [6] [7]Other Improvements
Responseclass to track match flags for name/location searches (src/DataBUS/Response.py). [1] [2]AEDNAMODEL_PARAMSand improved naming inAeDNAModelfor superseding models (src/DataBUS/AeDNAModel.py). [1] [2] [3].github/workflows/codecov.yml).