English | 简体中文

bioextract

Stable, provenance-aware domain access to official biological database snapshots.

bioextract hides resource-specific file layouts, identifier rules, hierarchies, directions, and repeated joins. Callers provide local snapshot files; the library neither downloads resources nor knows which application will consume them.

Architecture

The domain contract is primary. Storage is an execution strategy:

• use official/native direct access when the upstream representation is fit for the supported queries (eggNOG, STRING, and OmniPath); or
• publish one bioextract-owned DuckDB for each materialized logical product, regardless of whether it contains one relation or many;
• keep ordinary filtering, sorting, grouping, and SQL in Polars or DuckDB;
• add convenience methods only when they encode resource-owned ID resolution, relationship traversal, grouping, or unmatched-ID accounting.

Materialized writers use write_duckdb(path) with an explicit destination. Readers use XDatabase.from_duckdb(path), and every connect() call returns a fresh caller-owned read-only DuckDB connection. Writers validate into a staging file and atomically publish only after success. Publication provenance lives in exactly five relations in the DuckDB _bioextract schema; biological relations live in main. Metadata v2 is the only supported publication metadata contract. Parquet is an upstream, internal-transfer, or general interchange format, never a canonical bioextract publication.

Read the Domain Access Architecture before adding a resource, public query method, or storage strategy.

Inspect a publication

Inspect one explicit local publication without selecting a resource-specific reader or scanning biological rows:

import bioextract

publication = bioextract.inspect_publication("out/go.duckdb")

print(publication.resource_name)
print(publication.resource_schema_version)
print(publication.release_version)
print(publication.validation_status)
print(publication.table_counts_verified)  # False by default
for table in publication.tables:
    print(table.table_name, table.table_role, table.row_count)

inspect_publication() is the stable top-level function. The immutable result and supporting record types remain available from bioextract.publication, not as top-level package exports.

Install

pip install bioextract

ChEBI and ChemOnt

FULL OBO supplies the canonical compound, identifier, name, cross-reference, property, and relation schema. SDF only supplements molfile records; optional ChemOnt remains a separate chemont_* graph in the same container:

from bioextract import ChEBIDatabase

result = ChEBIDatabase.from_obo(
    "chebi/database/2026-07-07/raw/chebi.obo",
    chemont_obo="ChemOnt_2_1.obo.zip",
).write_duckdb("out/chebi.duckdb")

print(result.tables)

Open the publication for stable domain extraction or unrestricted native read-only SQL:

database = ChEBIDatabase.from_duckdb("out/chebi.duckdb")
selection = database.select_compounds(
    ["CHEBI:15377", "CHEBI:10743"],
    namespace="chebi",
)

lf_compounds = selection.compounds()
lf_names = selection.names()
lf_relations = selection.relations()
lf_unmatched = selection.unmatched_ids()

with database.connect() as connection:
    prefixes = connection.execute(
        "SELECT DISTINCT source_prefix FROM compound_cross_reference"
    ).fetchall()

External cross-references use the official prefix directly as namespace, such as kegg.compound or hmdb. Public shared IDs are complete CHEBI:<number> CURIEs. Use explicit TSV files only for partial source builds; plain, gzip, zip, and tar inputs are detected internally where applicable.

Rhea

Build one query-ready database from a complete extracted release or archive:

from bioextract import RheaDatabase

result = RheaDatabase.from_files("rhea-release.zip").write_duckdb(
    "out/rhea.duckdb"
)
print(result.tables)

Explicit files accept incomplete or mixed capabilities while retaining the same DuckDB container:

from bioextract import RheaDatabase

RheaDatabase.from_files(
    rdf="rhea.rdf.gz",
    directions="rhea-directions.tsv",
    relationships="rhea-relationships.tsv",
    xrefs="rhea2xrefs.tsv",
    uniprot_sprot="rhea2uniprot_sprot.tsv",
    uniprot_trembl="rhea2uniprot_trembl.tsv.gz",
).write_duckdb("out/rhea.duckdb")

Open a published database and select reactions through any one supported official namespace:

database = RheaDatabase.from_duckdb("out/rhea.duckdb")
selection = database.select_reactions(
    ["CHEBI:15377", "CHEBI:16474"],
    namespace="chebi",
)

lf_matches = selection.matches()
lf_reactions = selection.reactions()
lf_participants = selection.participants()
lf_cross_references = selection.cross_references()
lf_unmatched = selection.unmatched_ids()

select_reactions() and select_groups() are deferred domain query plans; their noun terminals return replayable Polars LazyFrame objects. Call .collect() at the application boundary that needs an eager DataFrame. Participant output retains the exact Rhea ID, master ID, direction, side, and compound fields. ChEBI fields are complete CHEBI:<number> CURIEs and can be equality-joined to a ChEBI publication without prefix construction or casts. directional_role is populated only for LR and RL; undefined and bidirectional reactions retain null rather than inventing a substrate/product orientation.

See the Rhea architecture for direction, hierarchy, table, and provenance contracts.

GO

GO is a multi-relation ontology and is published as one DuckDB:

from bioextract import GODatabase

go = GODatabase.from_obo("go-basic.obo")
df_terms = go.select_terms(subset_id="goslim_generic")
df_cellular_components = go.select_terms(namespace="cellular_component")
selection = go.select_ancestors(
    ["GO:0008150", "GO:1234567"],
    target_subset_id="goslim_generic",
    include_self=True,
)
lf_ancestors = selection.ancestors()
lf_unmatched = selection.unmatched_ids()
result = go.write_duckdb("out/go.duckdb")

Tables include term, term_relation, term_synonym, term_xref, term_alternate_id, term_ancestor, and term_depth.

GO ancestor selection resolves canonical or alternate GO IDs and can project their is_a/part_of ancestors into an OBO subset. Protein membership and enrichment analysis remain downstream application responsibilities.

KEGG

An independent KEGG mapping or BRITE profile is published as one-table DuckDB:

from bioextract import KEGGDatabase

source = KEGGDatabase.from_brite_json("br08901.json")
source.write_duckdb("out/kegg-brite.duckdb")

published = KEGGDatabase.from_duckdb("out/kegg-brite.duckdb")
with published.connect() as connection:
    pathway_count = connection.sql("SELECT count(*) FROM pathway").fetchone()[0]

When multiple KEGG products share a directory, use the smallest useful qualifier, such as kegg-mapping.duckdb or kegg-brite.duckdb.

A compound/reaction/enzyme/module snapshot is a multi-relation metabolic publication:

database = KEGGDatabase.from_metabolic_files("kegg/metabolic/2026-07")
database.write_duckdb("out/kegg.duckdb")

published = KEGGDatabase.from_duckdb("out/kegg.duckdb")
selection = published.select_ids(["CHEBI:15377"], namespace="chebi")

lf_reactions = selection.reactions()
lf_pathways = selection.pathway_memberships()
lf_unmatched = selection.unmatched_ids()

with published.connect() as connection:
    relation = connection.sql(
        """
        SELECT reaction_id, count(*) AS participant_count
        FROM reaction_participant
        GROUP BY reaction_id
        """
    )

The domain API supplies reaction-centered traversal and input lineage. connect() exposes the same validated publication as caller-owned, native read-only DuckDB SQL.

Reactome and WikiPathways

Pathway entities and membership relations are published together:

from bioextract import ReactomeDatabase, WikiPathwaysDatabase

ReactomeDatabase.from_files(
    uniprot_mapping="UniProt2Reactome.txt",
    uniprot_all_levels="UniProt2Reactome_All_Levels.txt",
    pathways="ReactomePathways.txt",
    relations="ReactomePathwaysRelation.txt",
    release_version="96",
).write_duckdb("out/reactome.duckdb")

WikiPathwaysDatabase.from_gmt(
    "wikipathways-20260510-gmt-*.gmt",
    species="Homo sapiens",
).write_duckdb("out/wikipathways.duckdb")

Selection methods such as select_ids() retain unmatched inputs and hide resource-specific mapping joins. Reactome calls default to lowest-level UniProt pathways; request pathway_level="all_levels" explicitly for hierarchy- expanded mappings. They do not calculate enrichment statistics. WikiPathways glob expansion is enabled by default; pass glob=False for a literal path or sequence. The constructor validates one Collection and Version and unique pathway IDs across the complete resolved file set before applying the optional species row filter.

eggNOG

The official SQLite representation is consumed directly during extraction:

from bioextract import EggNOGDatabase

db = EggNOGDatabase.from_sqlite(
    "eggnog.db",
    cog_functions="cog-24.fun.tab",
)
mapping = db.select_ids(["9606.ENSP00000369497"]).mappings()

Selections query plain SQLite directly without requiring a published derivative. Gzip-wrapped SQLite is accepted with a warning and is decompressed only to temporary scratch storage; for repeated use, decompress it once and pass the .db file.

InterPro and Pfam

InterPro mapping and every Pfam term, xref, and protein-term relation available from the configured source files share one DuckDB publication:

from bioextract import InterProDatabase

db = InterProDatabase.from_mapping_files(
    protein_to_interpro="108.0/raw/protein2ipr.dat.gz",
    interpro_xml="108.0/raw/interpro.xml.gz",
)
db.write_duckdb("out/interpro.duckdb")

published = InterProDatabase.from_duckdb("out/interpro.duckdb")
with published.connect() as connection:
    print(connection.sql("SHOW TABLES").fetchall())

UniProt

UniProt idmapping remains a separate lazy source profile and publishes one mapping table in DuckDB:

from bioextract import UniProtDatabase

UniProtDatabase.from_idmapping(
    "idmapping_selected.tab.gz",
    release_version="2026_01",
).write_duckdb(
    "out/uniprot_idmapping.duckdb",
    taxon_ids=["9606", "10090"],
)

mapping = UniProtDatabase.from_duckdb("out/uniprot_idmapping.duckdb")
human = mapping.scan_mapping(taxon_ids=["9606"])
with mapping.connect() as connection:
    print(connection.sql("SELECT count(*) FROM mapping").fetchone())

Reviewed UniProtKB is a multi-relation DuckDB publication:

UniProtDatabase.from_knowledgebase(
    entries="uniprot_sprot.dat.gz",
    canonical_sequences="uniprot_sprot.fasta.gz",
    isoform_sequences="uniprot_sprot_varsplic.fasta.gz",
    release_version="2026_01",
).write_duckdb("out/uniprot.duckdb")

db = UniProtDatabase.from_duckdb("out/uniprot.duckdb")
proteins = db.select_ids(
    ["P04637"],
    namespace="uniprot",
    taxon_ids=["9606"],
).proteins()
with db.connect() as connection:
    relation_count = connection.execute(
        "SELECT count(*) FROM protein"
    ).fetchone()[0]

Constructor arguments declare source roles, while headers and record grammar validate their content. Paths never supply release identity. An all-taxid idmapping export requires allow_all_taxa=True.

STRING

select_ids() and select_groups() encapsulate alias resolution, unmatched IDs, group isolation, and edge mapping:

from bioextract import STRINGDatabase

selection = (
    STRINGDatabase.from_files(
        aliases="9606.protein.aliases.v12.0.txt.gz",
        links="9606.protein.links.v12.0.txt.gz",
    )
    .select_groups(
        {
            "TumorA": ["TP53", "EGFR"],
            "TumorB": ["CDK2", "TP53"],
        }
    )
    .with_min_combined_score(400)
)

lf_mapping = selection.mappings()
lf_unmapped = selection.unmatched_ids()
lf_edges = selection.edges()

combined_score is a STRING confidence score, not an interaction-strength measurement.

STRING alias selections accept ordinary non-pipe alias text or one complete UniProt sp|accession|entry_name / tr|accession|entry_name value. Malformed pipe-bearing aliases raise ValueError. Direct namespace="string" inputs remain exact STRING protein IDs after trimming.

OmniPath

from bioextract import OmniPathDatabase

selection = (
    OmniPathDatabase.from_files(
        enzsub="enzsub.tsv.gz",
        interactions="interactions.tsv.gz",
    )
    .select_ids(["P31749", "AKT1", "BAD"])
    .with_enzsub()
)

lf_enzsub = selection.enzsub()
lf_unmapped = selection.unmatched_ids()

OmniPath protein selections accept a plain protein identifier or the same complete UniProt pipe representation; malformed pipe-bearing caller values are rejected before lookup.

Naming and compatibility

Public resource handles use complete *Database names, including GODatabase, ChEBIDatabase, RheaDatabase, KEGGDatabase, ReactomeDatabase, WikiPathwaysDatabase, EggNOGDatabase, InterProDatabase, UniProtDatabase, STRINGDatabase, and OmniPathDatabase.

Import database handles from the lazy top-level API:

from bioextract import ChEBIDatabase, RheaDatabase

The corresponding resource-subpackage path, such as from bioextract.rhea import RheaDatabase, remains stable. Prefer the top-level form when importing handles from multiple resources.

Catch public operational categories through bioextract.errors:

from bioextract.errors import CapabilityError, IntegrityError

Selection, result, namespace, configuration, and tidy implementation types are returned or consumed by database methods but are not stable package exports. Do not depend on their deep module paths for compatibility.

There are no abbreviated *Db aliases, legacy score-filter names, or directory writers. Use with_min_combined_score() and write_duckdb() directly.

Table names, view names, and generated columns use singular snake_case. Official two-dimensional source headers are retained unless a minimal deterministic mapping is required to make them queryable. Any such mapping is recorded in embedded provenance.

The versioned CephFS convention is tidy/data.duckdb. Callers may use other filenames; a filename is never schema identity or a compatibility identifier. Machine identity comes from embedded metadata.

Development

• Documentation is indexed in docs/README.md.
• Test layers and fixture ownership are defined in docs/testing/README.md.
• pdm run format
• pdm run lint
• pdm run typecheck
• pdm run test-unit
• pdm run test-contract
• pdm run test-integration
• pdm run test
• pdm run test-smoke runs only explicitly configured host publications.
• pdm run precommit applies formatting and lint fixes, then runs strict typing and the complete hermetic suite.

Hermetic tests limit DuckDB, Polars, and Rayon-backed work to four threads by default. Set BIOEXTRACT_TEST_THREADS=1 when sharing a constrained host.

For publication builds, set POLARS_MAX_THREADS before importing Polars or bioextract. It bounds both Polars execution and bioextract-owned DuckDB publication connections.

Release

• .github/workflows/py-ci.yml runs test-and-build checks.
• .github/workflows/publish.yml publishes canonical PEP 440 tags.
• PyPI trusted publishing is expected for the pypi environment.