Repository navigation
Added
-
Search(online_fallback=True): ask PubChem and CACTUS what no offline
database holds. Off by default, so a run still opens no socket. When on,
a query that produced no candidate from any offline source, and only such
a query, is asked of PubChem's PUG-REST service and the NCI/CADD
resolver (CACTUS). Their answers are pooled, clustered and scored like
offline ones, and each service counts as one vote inn_source_support:df = Search("cas", online_fallback=True).search(["50-78-2", "1912-24-9"]) df["source"] # "PubChemID", ..., "PubChem (online)" or "CACTUS" df.attrs["online_fallbacks"] # queries that went online df.attrs["online_resolved"] # ...and those the services answered
The two services are two more columns in
provesid.sources.LOOKUPS,
pubchem_onlineandcactus(ONLINE_SOURCE_KEYS), with rows for CAS,
name, SMILES, InChI, InChIKey and, on PubChem only, DTXSID. Formula queries are never
sent online. A service's "not found" is a miss. Any other failure is logged
at WARNING and costs only that service's vote. Each fallback is logged at
DEBUG. New helpers:PubChemAPI.get_cids_by_inchi, which POSTs because an
InChI cannot travel in a URL path, andtools.candidate_from_pubchem_online
andtools.candidate_from_cactus. See
examples/search/online_fallback_demo.py. -
ChEMBL from its MySQL dump:
CheMBL(source="mysql"). Builds the same
2.4 GiB extract as the default route from ChEMBL's 2.1 GB MySQL dump, read
as a stream, so the 27.7 GiB release never exists on disk:CheMBL(source="mysql") # 2.1 GB down, ~4.5 GiB free CheMBL.build_from_mysql_dump("chembl_37_mysql.tar.gz") # from a dump you already have CheMBL.extract_digest(path) # {table: (rows, hash)}
The new
provesid.mysqldumpmodule reads exactly whatmysqldumpwrites
and raisesDumpFormatErroron anything else rather than guess a value.
Column types come from the dump'sCREATE TABLEand map to the affinity
SQLite gives them, so values are stored the way thesqliteroute stores
them:decimal4.00 becomes the integer 4 in both.extract_digest
fingerprints each table with a row count and an order-independent content
hash that includes each value's type, and is how the routes were compared.
On the real ChEMBL 36, written out as a mysqldump, all eight tables
(14.6 M rows) digest identically to thesqliteroute's extract. The build
took 6 min and 185 MB of memory. Provenance now recordssource_format.
The default stayssource="sqlite"until the same comparison has been run on
EBI's own dump.examples/chembl/download_source_demo.pyshows all three
routes. -
Computed descriptors on demand:
PubChemID.descriptors(). The database
no longer stores XLogP, TPSA or the atom and bond counts; this computes them
when asked, and names the model that answered:db = PubChemID() db.descriptors(2244) # RDKit, from the stored SMILES # {'CID': 2244, 'Source': 'rdkit', 'MolLogP': 1.3101, 'TPSA': 63.6, ...} db.descriptors(2244, ["XLogP"], source="pubchem") # PubChem's, over PUG-REST db.descriptors_table(cids) # a row per CID rdkit_descriptors("CCO") # any structure, no database
source="rdkit"(default) needs no network for a compound the database holds
and costs about 0.5 ms per compound; for one it does not hold, only the SMILES
is fetched. Names are PubChem's where the quantity is the same, except the
logP: RDKit's is Crippen's, not XLogP3, so it isMolLogP, and asking RDKit
forXLogPorComplexityraises an error saying where to get it. The
values are RDKit's, not PubChem's Cactvs values. Against 20 000 compounds,
heavy atoms and charge agree for all, donors for 94%, rotatable bonds 74%,
TPSA 70% (counted with S and P, as PubChem does), acceptors 63%.
examples/pubchem/descriptors_demo.py. -
pubchem_id.dbis built from PubChem's FTP site, and says where it came
from. The newprovesid.pubchem_ftpmodule builds the database
PubChemIDreads from a dated monthly snapshot ofCompound/Extras/, and
PubChemID(source=...)chooses between that and the Zenodo download:from provesid import PubChemID from provesid.pubchem_ftp import build_pubchem_id_db, list_releases db = PubChemID() # source="ftp" (default): build if missing db = PubChemID(source="zenodo") # download the 2.2 GiB prebuilt copy list_releases() # ['2026-09-01', ..., 'current'] build_pubchem_id_db(release="2026-09-01", keep_downloads=True) db.provenance()["release"] # '2026-09-01', and every file's MD5 db.xrefs(2244) # {'chebi': [...], 'dtxsid': [...], ...}
Which compounds. Those PubChem's own identifier file maps to a CAS number,
each checked against the check digit: 1 430 379 in the 2026-09-01 snapshot,
with 123 CAS rows rejected. The Zenodo database found its CAS numbers by
running\d{2,7}-\d{2}-\dover free-text synonyms, and 17 379 of them
(1.25%) are not valid CAS numbers.How. Eight files, each downloaded resumably, checked against the MD5
PubChem publishes beside it, streamed once to keep the compounds in scope and
deleted, so the free disk needed is the database plus the largest file
(7.4 GB), not the 15.4 GB total. The database is built beside its destination
and moved into place only when complete.include_inchi=Falseskips the
InChI file and computes InChI with RDKit;include_synonyms=Falseleaves out
the synonyms;keep_downloads=Truekeeps the files, and a later build reuses
any whose MD5 still matches.What it records. A
provenancetable (release, snapshot timestamp, build
time, row counts, builder version) and aprovenance_filestable (URL, MD5,
bytes, lines read and kept, per file).PubChemID.provenance()returns both.What it adds.
monoisotopicmass, and anxrefstable of DSSTox, ChEBI,
ChEMBL, EC and UNII identifiers from the same file, whichPubChemID.xrefs()
reads.Molecular weight is in none of the FTP files, so it is computed from the
formula (provesid.pubchem_ftp.molecular_weight) with
provesid.pubchem_ftp.ATOMIC_WEIGHTS--- IUPAC 2005 with twelve elements
fitted to PubChem's own weights --- and from the SMILES for isotopically
labelled compounds, whose formula PubChem writes unlabelled. PubChem's
weights cannot be reproduced exactly: it rounds them to between zero and
three decimals by a rule no function of the formula fits. This agrees with
PubChem to the second decimal for most compounds and is the more precise
number where PubChem rounds harder.scripts/build_pubchem_id_db.pyis now a command-line wrapper over the
builder, for refreshing the Zenodo copy; the CSV-and-regex version it
replaces, and its duplicate insrc/provesid/data/, are removed.
examples/pubchem/ftp_build_demo.pybuilds a miniature release served on
localhost. -
The cache is in a cache directory now, one per service, with a version in
every key. Three changes toprovesid.cache:import provesid provesid.get_cache_info(service="pubchem")["cache_directory"] # '/home/you/.cache/provesid/pubchem' (was /tmp/provesid_cache/pubchem) provesid.clear_cache(service="pubchem") # one service provesid.clear_cache(all_services=True) # all of them provesid.get_all_cache_info() # {'global': ..., 'pubchem': ...}
It persists. The default was
tempfile.gettempdir()/provesid_cachewhile
docs/advanced_caching.mdpromised the cache survived a restart. Most Linux
systems clear/tmpon boot, so it did not. It now uses
platformdirs.user_cache_dirthrough the newprovesid.utils.user_cache_path
—~/.cache/provesid/on Linux,~/Library/Caches/provesid/on macOS,
%LOCALAPPDATA%\USEtox\provesid\Cache\on Windows — overridable with
PROVESID_CACHE_DIR, and separate fromPROVESID_DATA_DIRbecause cached
responses are disposable and 2.4 GiB datasets are not. Entries written under
the old default are not migrated; they were never reliably there to migrate.The service list is data. Fourteen near-identical module-level functions
(clear_pubchem_cache,clear_cas_cache, … and sevenget_<svc>_cache_info
twins) plusexport_service_cacheandimport_service_cacheare removed.
Every cache function —clear_cache,get_cache_info,get_cache_size,
export_cache,import_cache,set_cache_warning_threshold,
enable_cache_warnings— now takesservice=, with the names in
provesid.CACHE_SERVICES;get_all_cache_info()and
clear_cache(all_services=True)cover the lot. An unknown service raises
ValueErrorinstead of silently writing to the global cache, where the
matchingclear_cachewould never look.get_service_cache()returns the
underlyingCacheManager, and builds it on first use, so importing
provesidno longer creates eight cache directories.Keys carry a version. Pickle stores an instance's
__dict__, so an entry
written before a cached dataclass gained a field comes back as an object
missing that attribute, and every caller that reads it raises on a machine
where only the package version changed. There are now three versions folded
into the key, and you bump the narrowest one that covers the change:
@cached(version=N)for one function, a client'sCACHE_SCHEMA_VERSIONfor
one client,provesid.cache.CACHE_KEY_VERSIONfor everything. Retired
entries are unreachable, not deleted;clear_cachereclaims the space.Also fixed:
export_cachedropped any entry whose cached value wasNone
(is not Nonewhere it meantis not _MISS), so a legitimately cached
Nonedid not survive an export/import round trip. -
The four SQLite clients can be closed, used in a
withblock, and queried
from several threads.PubChemID,CompToxID,ZeroPMandCheMBLeach
opened a connection in__init__and closed it in__del__, and nowhere
else. All four now inheritprovesid.SQLiteClient:with PubChemID() as db: # released at the end of the block inchi = db.cas_to_inchi("50-78-2") db = CompToxID() db.close() # or by hand; idempotent db.closed # True with PubChemID() as db: # a connection per thread with ThreadPoolExecutor(8) as pool: rows = list(pool.map(db.get_by_cas, cas_numbers))
The threading case is the one users met first: a single connection is bound
to the thread that created it, so the obvious way to resolve ten thousand
CAS numbers against a local 2.2 GB database died on the first worker with
sqlite3.ProgrammingError: SQLite objects created in a thread can only be used in that same thread. Each thread now gets its own connection and
cursor, opened on its first query;close()closes all of them, from
whichever thread calls it.Per-thread connections rather than
check_same_thread=False, which removes
the check and not the problem: every thread would still share one
self.cursor, and these clientsexecutein one statement andfetchone
in the next, so two threads interleaving those would read each other's rows.
A wrong answer is a worse failure than the exception it replaces.Threads are now possible, which is not the same as faster: a tight loop of
nothing but local lookups is slower on a pool than in a plain loop (5 000
get_by_cascalls: 0.29 s serially, 14.65 s on eight threads), because each
query takes tens of microseconds and the GIL handoff around it costs more.
That issqlite3under CPython, not this change — plainsqlite3.connect
per thread measures the same. A pool pays when each item also waits on
something: 400 lookups each followed by 20 ms of waiting took 8.52 s
serially and 1.17 s on eight threads.Searchis a context manager too, and closes the source clients it
constructed — but not one passed to its constructor, which belongs to the
caller. Querying a closed client or a closedSearchraises
DatabaseClosedError, aRuntimeErrorsubclass that names the client and
the file, rather than sqlite3'sCannot operate on a closed database.
__del__remains as a backstop, so code that never closes anything behaves
as it did. -
Installing ChEMBL now costs 2.4 GiB, not 27.7 GiB:
CheMBL(source=...).
CheMBL.compact()could already shrink a release already on disk, which
left the worst case untouched — a machine that had never had ChEMBL still
installed 27.7 GiB and had to be told to shrink it afterwards.The download now finishes by building the extract and deleting the release
it came from:CheMBL() # 5.8 GB transferred, 2.4 GiB installed (default) CheMBL(source="full") # ...or keep all 74 tables, 27.7 GiB
Both routes transfer the same archive: the choice is what stays on disk. The
extract is verified against the release —quick_check, a row count per
table, and 500 compounds compared column by column — before the release is
deleted, so the file that survives is the one that was checked.sourceonly describes a download. A release already on disk is opened as
it is: shrinking 27 GB iscompact()'s job, not something a constructor
should do unasked. Opening a full release now logs the one line that says
the option exists.A compaction that fails after a successful download is logged, not raised.
The release is in place by then and answers every query, so the failure
costs disk rather than function, and throwing away a 5.8 GB download over a
step that can be repeated with one call would be the worse trade. -
datasets.plan()now tells the truth about ChEMBL. Its registry entry
said 27.7 GiB installed; it is 2.4 GiB, from a 5.8 GB download, with a peak
of 33.4 GiB while the archive and the full release both exist. That peak is
the number a laptop actually fails on, and it is now the one that is
reported: for the four default sources, 8.9 GiB downloaded, 6.3 GiB
installed (down from 31.6 GiB), 37.3 GiB needed at the worst moment. -
A dataset manager, and
Searchno longer downloads 32 GB behind your
back. On a clean machineSearch("cas").search("50-00-0")constructed
four source clients that each default toauto_download=True, so one CAS
lookup fetched ChEBI, CompTox, PubChem and ChEMBL — about 32 GB, of which
ChEMBL is 87% and only enriches a structure the other sources already
found. Nothing announced the total, nothing asked, and nothing offered to
proceed with the sources already present. This package is for researchers on
laptops, where 32 GB is often the whole free disk.provesid.datasetsnow describes the five datasets without opening them, so
the decision can be made before the first byte moves:from provesid import datasets datasets.status() # present? size on disk? release? path? datasets.plan(["pubchem", "chebi"]) # what a download would transfer datasets.fetch("pubchem") # install by name; skips what is present datasets.remove("chembl") # reclaim the space, by name
status()andplan()return DataFrames and read only filenames and
stat, so they are instant with 30 GB of ChEMBL in the directory.plan()
also reports the peak requirement, which for ChEMBL exceeds the installed
size by the 5.8 GB archive sitting beside the database it extracts into — a
laptop with 28 GB free and 27.7 GB of ChEMBL to install still fails. -
Search(datasets=...), defaulting to"present".value behaviour "present"use whatever is on disk; name each missing dataset, its size and the fetchcall that installs it. The new default."auto"the behaviour before this release — download whatever is missing "required"raise MissingDatasetErrorin the constructor, before any query, naming the missing datasets and the exactfetchcallSearchalready degraded gracefully when a source was unavailable and
already warned that confidence is not comparable across runs, so"present"
needed no new machinery in the resolver — only that the clients stop
downloading behind the caller's back. -
One resumable, checksummed downloader for every bulk dataset:
provesid.datasets.download_file. Five modules had each grown their own
copy of "stream the response into a temporary file with a progress bar", and
all five shared the same three defects — no retry, no resumption, no
checksum. An interrupted 5.8 GB ChEMBL download started again from zero, and
so did an interrupted 2.2 GB PubChem one.Corruption was caught unevenly, too. The two gzipped downloads got truncation
detection for free from gzip's CRC and length trailer; the three plain SQLite
downloads had none. A file truncated in its interior opens cleanly and fails
much later, on the first query that touches a missing page — which a user
reads as a data problem, not a download problem.download_filestreams into a.partfile beside the destination, resumes
it with an HTTPRangerequest, and applies four checks in order, each of
which leaves the destination untouched if it fails:- the byte count against the size the server declared;
- an MD5, from
expected_md5or fetched fromchecksum_url— PubChem's FTP
mirror publishes an.md5beside every file; - the caller's own
verifycallback, handed the finished file; - an atomic rename.
from provesid.datasets import download_file download_file(url, "/data/CID-SMILES.gz", checksum_url=url + ".md5")
A file rejected by its checksum or by
verifyis deleted rather than kept:
it is already complete, so resuming it would fail the same check again. A
transfer that merely stopped is kept, and the next call continues from it —
but only when it came from the same URL, recorded in a.part.sourcemarker
beside it. A partial left by a different download is discarded instead of
being spliced onto this one, which a checksum would catch but only PubChem's
FTP mirror publishes one.PubChemID.download_database,CompToxID.download_database,
ZeroPM.download_database,CheMBL.download_databaseand
ChebiSDF.download_sdfare now one call each into this. -
CheMBL.compact()— a ChEMBL release without the 27 GB you never read.
A full ChEMBL release is ~30 GB across 74 tables. PROVESID opens eight of
them (molecule_dictionary,compound_structures,compound_properties,
molecule_synonyms,molecule_hierarchy,chembl_id_lookupand the two
pesticide tables) and reads no bioactivity data at all:activitiesalone is
24.3 M rows that nothing in this package has ever queried.Worse, a quarter of the whole database is a single column.
molfile— the
MOL block for every compound — totals 7.72 GB, andget_compoundwas the
only thing that selected it, for callers that never used it.CheMBL.compact()copies the eight tables, dropsmolfile, keeps only the
COMPOUNDrows ofchembl_id_lookup, and rebuilds just the indexes this
package's queries need. Measured on ChEMBL 36: 29.74 GB → 2.60 GB, 91.3%
smaller, in about a minute, with every public method returning the same
compounds — verified over 3 000 randomly chosen compounds.from provesid import CheMBL CheMBL.compact() # build chembl_37_provesid.db beside the original CheMBL.compact(remove_source=True) # …and delete the 30 GB once it verifies CheMBL.compact(keep_inchi=False) # ~1.55 GB; see the caveat below
The source is opened read-only, the extract is written to a temporary file
and checked against its source — SQLite'squick_check, a row count per
table, and 500 compounds re-read from both databases — before it replaces
anything.remove_sourcedeletes the original only after that passes.A later
CheMBL()opens the extract in preference to a full release of the
same number, so nothing else has to change.CheMBL.is_compactand
CheMBL.provenancereport what was opened; an extract records the release it
came from, the source file and its size, the build time, the PROVESID version
and its row counts. It also records a schema version, so a future PROVESID
that needs a ninth table warns and names the rebuild command instead of
failing withno such table.keep_inchi=Falsesaves a further ~1.0 GB by droppingstandard_inchiand
its indexes, butsearch_by_inchithen has no column to match andSearch
loses the InChI it reads straight from ChEMBL. Leave it alone unless disk is
genuinely short. -
One shared HTTP transport, and
Retry-Afterhonoured for the first time.
Every web-API client used to carry its own copy of "pause, request, decide
what the status code meant, maybe give up". The six copies had drifted
apart: onlypubchemviewretried at all, none honouredRetry-After, and
chebi,cascommonchemandopsinhad no rate limiting whatsoever.provesid.http.HTTPClientis now the single place that decides when to ask
again. It paces requests, retries HTTP 429, 5xx, timeouts and connection
errors with exponential back-off capped atmax_backoff, and prefers a
Retry-Afterthe service sent — in seconds or as an HTTP date — over its own
curve. The pacing applies to retries too: a service already shedding load is
not asked again faster than a healthy one. No rawrequestsexception
reaches a caller.Every web-API client is migrated:
resolver,pubchemview,pubchem,
chebi,cascommonchemandopsin. Onlyclassyfirestill holds
requestscalls of its own, and only because its service has been down
since February 2023. The bulk database downloads
(ChEBISDF.download_sdf,chembl,comptox,zeropm,pubchem's Zenodo
dataset) are deliberately left alone: a hundred-megabyte streamed file wants
resumption and checksums, not a 5-per-second pacer.Nothing changes for a caller. Each client passes its own exception classes to
the transport, soexcept NCIResolverErrorandexcept PubChemViewError
work exactly as before; what is new is that those classes also descend from
provesid.http.ServiceError/NotFoundError/ServiceTimeoutError, so a
caller can now catch every service at once. -
Pacing is now shared per host, not per client object. PubChem publishes
five requests per second per IP, and aPubChemAPIplus aPubChemView
in one process — which is the ordinary way to use this package, and what
Searchdoes — each kept their own clock. Each believed it was pacing
correctly while together they asked twice as fast as PubChem allows.min_intervalstays the client's own promise about how fast it will ask; the
clock it is measured against now belongs to the host.
provesid.http.RateLimiterholds one host's clock behind a lock, and
provesid.http.host_limiter(url_or_host)hands every client aimed at that
host the same one. ChEBI and OPSIN are both served fromwww.ebi.ac.uk, so
they share a clock too — which is correct, the limit being the host's.last_request_timestill reports when that client last asked. The
effective rate for a host is set by its most impatient client, so this stops
two clients from doubling a limit but not one client configured at
min_interval=0.01from exceeding it alone. -
max_elapsed, a ceiling on the total time spent waiting between retries —
"do not make the caller wait longer than this". PubChem answers a throttled or
blacklisted IP withRetry-After: 30, which the transport honours, so three
retries would be a ninety-second call;
tests/test_pubchemview.py::TestPubChemView::test_error_handlingmeasured
92.5 s. Dropping the retry would be worse, because a caller resolving ten
thousand compounds loses one to every transient 503.Both PubChem clients set
provesid.pubchem.RETRY_WAIT_BUDGET = 10.0, which
leaves the cheap curve intact — 1 + 2 + 4 for a transient 500 or a timeout —
while declining the 30-second throttle. That block does not lift in thirty
seconds: measured on 2026-09-19, waiting the full thirty and asking again
returned the same 503, so the wait buys nothing while every call pays it. A
throttledget_compound_by_cidnow fails in 1.1 s. Raise the budget with
api._http.max_elapsed = 180for an unattended bulk job that would rather
wait.max_elapseddefaults toNone, so every other client is unaffected. -
HTTPClient(session=...), so a client that keeps arequests.Session
for connection pooling and persistent headers makes its calls through it.
chebiuses this: itsUser-AgentandAcceptlive on the session, and an
ontology walk reuses one connection. Without a session the transport calls
requests.get/requests.postthrough the module, which is what lets a test
stub them. -
Exceptions carry the response detail.
ServiceErrornow has
status_code,urlandresponse, keyword-only and defaulting to None, so
raise PubChemError("...")by hand is unchanged. CAS Common Chemistry needs
this: it reports a rejected key as 401 and an unknown CAS number as 404, and
both have to become different strings in the dict it returns. -
retry_exhausted_cls, raised when a transient condition outlives the
retry budget, as distinct from a permanent error. PubChem passes
PubChemServerError, because its callers catch that one to skip and
PubChemErrorto fail. Defaults toerror_cls, so no other client notices. -
New exception classes, all exported from
provesidand all descending
from the shared bases:ChEBINotFoundError,ChEBITimeoutError,
CASCommonChemError,CASCommonChemNotFoundError,
CASCommonChemTimeoutError,OPSINError,OPSINNotFoundError,
OPSINTimeoutError.PubChemServerErrorandPubChemTimeoutErrorare now
exported too.ChEBIErrorwas a bareExceptionand is now a
ServiceError, soexcept ServiceErrorcatches ChEBI as well. -
tests/test_cascommonchem_offline.py, 19 offline tests.
tests/test_cascommonchem.pyskips itself entirely without a CAS API key, so
the module that gained the most in this change had no coverage on a developer
machine at all. The key is only a header, so a stub key and a stubbed
requests.getcover the lot. -
provesid.cache.is_empty_result, theskip_ifpredicate that keeps an
empty answer out of the cache. It was written twice, identically, in
pubchemandpubchemview; it now lives besideis_failure_result, which
is whereskip_ifbelongs. -
One parser for PUG-View's free-text values, with typed output. PubChem
reports every experimental property as prose written by whoever deposited it —
"138-140 °C","8.5X10-5 mm Hg at 25 °C","greater than or equal to 100 mg/mL","Vapor pressure, kPa at 20°C: 24"— and PROVESID had two parsers
for it that disagreed with each other: a 25-line one behindPropertyDataand
a 390-line property-specific regex cascade behindget_property_table, both
returning strings.The new
provesid.pubchemview_parsemodule holds the only one.
parse_value(text, heading)returns aParsedValuecarrying:value, orvalue_min/value_maxfor a range (a single value fills both
bounds too, so a numeric filter needs no special case);unit, spelling normalised —torrandmm Hgboth report asmmHg;value_si/value_min_si/value_max_si/unit_si, the same quantity in
K,Pa,kg/m³,mol/m³,Pa·s,m²/sorN/m;temperature_c, the temperature the measurement was made at, which is a
condition rather than the value and was the largest source of wrong answers;operator(>,<,>=,<=,~) when the entry bounds the value;qualitative(insoluble,miscible,negligible) when a word replaced
the number;text, always — nothing the parser cannot read is lost.
The
headingargument settles what a string alone cannot: a bare138under
"Melting Point" is 138 °C, while a bare1.19under "LogP" is dimensionless
and complete.Measured on 366 real value strings across ten compounds and ten properties:
90% yield a number, 8% a qualitative term, and the remaining 2% are entries
where PubChem states no value at all. Aspirin's melting point entries, which
are deposited variously as135 °Cand275 °F, now all read 408.15 K.Nothing is invented: an unrecognised unit is reported as written with no SI
conversion, and%,ppmandppbare never converted, since a composition
needs a density to become a concentration.Mfor molar is not recognised
either — it is indistinguishable from metres and from a stray capital. -
get_property_tablenow returns numbers. New columnsValueMin,
ValueMax,ValueSI,ValueMinSI,ValueMaxSI,UnitSI,Operator,
QualitativeandHeadingjoin the existing ones, and the full list is
published asPubChemView.PROPERTY_TABLE_COLUMNS. -
Every PUG-View heading works, not only the experimental ones. Both
response parsers hard-coded the path Chemical and Physical Properties →
Experimental Properties → *, but PUG-View nests a requested heading
wherever it sits in the compound's table of contents: "GHS Classification"
under Safety and Hazards, "Drug Indication" under Drug and Medication
Information. Those returned a full response and an empty result — data
reported as absent while it was right there._iter_informationwalks the
record instead, andget_property_table(2244, "GHS Classification")now
returns its 18 rows. -
get_property_summarygainednumeric_values,numeric_values_siand
units_si;export_properties_to_dictgained the parsed numbers as flat,
JSON-serialisable keys.PropertyDatagainedheadingandparsed. -
PubChemView.CACHE_SCHEMA_VERSIONis part of the client's cache key, so an
entry written before the shape of a result changed becomes unreachable rather
than being deserialised into the wrong structure. This mattered immediately: a
PropertyDatapickled by the previous version restores without aparsed
attribute, and everything reading it would raise. Bump the constant whenever a
cached return shape changes. -
Bulk property retrieval.
PubChemAPI.get_properties_for_cids(cids, properties)asks PubChem about a whole list of compounds in one request
rather than one request per compound: 450 compounds now cost 3 requests and
about 2 seconds, where the per-CID loop cost 450 requests and a minute and a
half. Lists are split intoPROPERTY_CHUNK_SIZE(200) CIDs per request, so a
retry redoes a chunk rather than everything. -
POST for long identifier lists. PUG-REST caps a URL at roughly 2000
characters, which a few hundred CIDs exceed._build_post_urlbuilds the
same endpoint without the identifier segment, and the bulk property path
switches to POST — identifiers in the body — once the list outgrows
URL_IDENTIFIER_LIMIT. Below that it keeps using GET, which PubChem prefers. -
Offline-first property lookup.
PubChemID.properties(cid, properties)
reads the local SQLite database first and consults PUG-REST only for what it
cannot answer, per the two-stage lookup the development principles ask for.
For most property work this means no network traffic at all: 473 of 1000 low
CIDs were served from disk in the same call, and the remaining 527 in three
batched requests.properties_for_cids()does the same for a list, andproperties_table()
returns a DataFrame with a row for every CID asked about. Each record carries
aSourcekey readingoffline,onlineor (in the table)missing.
use_online_fallback=Falsekeeps a lookup strictly local.PubChemID.OFFLINE_PROPERTIESmaps the 16 property names the database can
answer — formula, weight, exact mass, isomeric SMILES, InChI, InChIKey, IUPAC
name, title, XLogP, TPSA, complexity, charge and the H-bond, rotatable-bond
and heavy-atom counts — to their columns. Anything else
(MonoisotopicMass,ConnectivitySMILES, the 3D descriptors, the patent and
literature counts) is online-only, and asking for one sends the whole request
online rather than assembling a row from two different PubChem snapshots.A property with no value is absent from the result rather than
None, which
is how PubChem reports it —'XLogP' not in resultmeans PubChem computes no
logP for that compound, not that the lookup fell short. Values are normalised
to one type across both sources, since PUG-REST returnsMolecularWeightas
a string where the database holds a float. -
PubChemID(api=...)accepts thePubChemAPIused for fallback; one is
created on first use otherwise, so a strictly offline session never builds
one.
Fixed
-
A failed ClassyFire request is no longer cached.
submit_query,
query_statusandget_queryreturn None or the error response on a
failure, and each was cached, so one HTTP 429 stayed the answer for good:
query_status(1)returned None after the server had recovered. Only a
response below 400 is cached now. -
Examples re-recorded against the live services (2026-09-24).
ChEBI.get_compoundsreturns an entry per ID withexists, and the
record underdata, not the bare record.ClassyFireAPI.submit_queryis
answered HTTP 500. Query 1 holds 510 entities, not 655: the server's
pages hold fewer thanper_pageasks for, soentitiescan fall short
ofnumber_of_elements.PubChemAPI.format_search_compound_result's
"several results" example returned one, because PubChem maps "aspirin" to
one compound. -
A failed CACTUS request is no longer cached as "no data".
NCIChemicalIdentifierResolver.get_molecular_data,resolve_multiple,
batch_resolve,is_valid_identifier,search_by_partial_nameand the
nci_*functions turn a failed request intoNone,[]orFalse, and
were each cached, so one timeout or 503 stayed those answers for good.
Onlyresolveis cached now. It raises on a failure, and the others build
on it, so they still read its cache. The per-calluse_cache=keyword is
gone from those methods. Passuse_cache=Falseto the constructor
instead. -
Searchreports the current CAS number, not the smallest string.
Aspirin came out as11126-35-5and atrazine as11121-31-6, both
retired, because every source's CAS numbers were sorted as text and the
first was taken.extract_cas_valuesandmake_candidatenow keep the
source's order, which puts CompTox'sCASRNcolumn first and follows
PubChem's synonym order. ChEBI, ChEMBL and CACTUS do not rank their
numbers, so their candidates list them lowest registry number first (the
newtools.sort_cas_by_number). For 82.6% of the 41,313 CompTox
substances with more than one CAS, the current number is the lowest, and
for 42.5% it is the smallest as a string. The newtools.pick_casrn
chooses a hit's number from all its candidates and asks an unranked
source that lists several numbers last.apply_candidate_to_resultno
longer fillsCASRN. ChEBI's CAS numbers are read from itsCAS Registry Numbersfield only. Reading the whole row picked up fragments of the
InChI such as14-10-6. On the USEtox 3 substance list (10,748
InChIKeys),CASRNagrees with the list's own number for 9,051 rows,
up from 7,229, and 10 rows that agreed no longer do. -
A search by CAS reports the compound's number, not the query.
CASRN
was set to the number searched, so atrazine found by the retired
39400-72-1said39400-72-1. It now says1912-24-9, chosen as for any
other search, andquerykeeps the number searched. A miss leavesCASRN
empty. On the USEtox 3 list, 34 of 10,752 CAS searches report another
number: 14 were retired numbers, and 16 are current numbers whose hit is
a more specific structure, such as93685-81-5found as dodecane,
112-40-3. -
ZeroPM.get_cas_from_inchiandget_cas_from_inchikeyreturn their
numbers in a fixed order, the order ZeroPM stored its results in. The
query had noORDER BY. On 20,000 structures the order is the same as
before. -
NCIChemicalIdentifierResolver.get_molecular_dataparsescasand
stdinchikey. It already splitnamesinto a list and readmwas a
float.cascame back as CACTUS's text, several numbers in one string,
andstdinchikeyasInChIKey=LFQSCW…, which matches no bare key from any
other source.casis now a list in CACTUS's order, which is not a ranking
(ethanol's starts with 121182-78-3, not 64-17-5), andstdinchikeyis the
bare key.nci_cas_to_molandnci_id_to_molreturn the same.resolve
still returns CACTUS's text unchanged. Their cache entries were retired
(version=2). -
ClassyFire: a query too large for the server can be fetched in pages.
ClassyFireAPI.get_query(1)returnedNone. The server stops sending
after about 108 KB while answering HTTP 200, so the query never arrived
whole, andcurlfails on it too (exit 18). The client turned the error
into a bareNone, and it had no timeout. The raw calls now have a
60-second timeout and log what failed at WARNING, andget_querytakes
ClassyFire'spageandper_page. The new
ClassyFireAPI.get_classification(query_id)fetches pages of 10, which
arrive complete, through the shared transport. It joins them into one
dict and raisesClassyFireError(exported fromprovesid) when a page
fails. -
The PUG-View parser reads the ICSC's solubility words. The
International Chemical Safety Cards grade water solubility asnone,
very poor,poor,moderate,goodorvery good.
"Solubility in water: none"parsed to nothing at all, and
"…, g/100ml at 15 °C: 0.25 (poor)"kept 0.25 but lost the word. The word
is nowqualitative. Becausenone,poorandgoodare ordinary
English, they count only when they are the whole ICSC value, or follow its
number in parentheses. The list ispubchemview_parse.ICSC_SOLUBILITY_TERMS. -
PYOPSIN.get_id_from_listgives each name its own CML. py2opsin
returns a list's CML as the lines of one document, and each record'scml
held one of those lines:"<?xml …"for the first name,"<cml …"for the
second, and so on. The document holds one molecule per name, in order, so
it is now cut at the molecules. Each record'scmlis the document
get_CML(name)gives for that name alone, including for a name OPSIN
cannot parse. An empty list returns[]without starting Java. -
The PUG-View parser reads a unit with digits in a value's label. The
International Chemical Safety Cards write water solubility as
"Solubility in water, g/100ml at 25 °C: 0.18"(benzene).parse_value
found 0.18 and the 25 °C, but looked for the unit in the label with a
pattern that allowed no digits, sog/100mlsplit intog/andmland
the unit wasNone, with no SI value. It now readsg/100mL, or 1.8
kg/m³, andmg/100mlthe same way. A formula in the label, such as
C6H6, is not taken for a unit. -
Passing one source client to
Searchno longer turns off the others.
One flag recorded whether the clients had been set up, and passing any
client set it.Search("cas", chembl=CheMBL(db_path=...))therefore
never built ChEBI, CompTox or PubChemID. It ran on ChEMBL alone and gave
empty rows with confidence 0.0, which only a WARNING in the log
explained. Each source client leftNoneis now built on the first
search, under thedatasetspolicy, whether or not others were passed.
A client that is passed is still used as given and left open by
close(). A caller who passedchebi=Noneand so on to leave sources
out now gets them. To leave a source out, leave it out ofsources
(below). -
Searchpassed on CompTox's and ZeroPM's non-standard InChIKeys.
CompTox stores a key computed from a non-standard InChI (flagN, as in
PGRHXDWITVMQBC-UHFFFAOYNA-N) for 131,885 of its 1.15 million keys, and
ZeroPM a non-standard InChI and key for 18,710 of its 359,221 substances.
Such a key never equals the standard key another source publishes. Three
things went wrong.InChIKeycame back non-standard whenever CompTox
filled it. A DTXSID query then looked up the other sources by that key and
found nothing:DTXSID6020014(dehydroacetic acid) andDTXSID8020040
(aldrin) had one source in support, and now have four and three. And a
standard-key query missed the CompTox row, soDTXSIDstayed empty. Every
candidate now carries the standard InChI and key, recomputed from the
structure when the source's are not standard
(provesid.tools.standardize_inchi_and_key, called bymake_candidate).
CompToxID.get_by_inchikeyandZeroPM.get_id_table_from_inchikeyalso
try the key with its other flag, which finds the 98% of CompTox's and 96%
of ZeroPM's non-standard rows where only the flag differs. New helpers:
provesid.utils.is_standard_inchikeyandinchikey_flag_variants. -
ZeroPM's InChI lookups missed substances stored under a non-standard
InChI. They compared the InChI as a string, so a standard InChI
(InChI=1S/...) could not match one stored only asInChI=1/.... When
the string misses,get_id_table_from_inchiandget_cas_from_inchi
(and soget_cas_from_smiles) now compute the InChIKey and look it up
with either flag. This finds 536 of the 816 substances no standard InChI
matched, such as trans-1,4-cyclohexanediol (6995-79-5). The other 280
have relative stereo (/s2), which a standard InChI cannot express.
get_cas_from_inchikey,get_smiles_from_inchikeyand
batch_get_cas_from_inchikeynow try both flags, like
get_id_table_from_inchikey. Theinchiandinchikeycolumns of
get_id_table_from_inchihold ZeroPM's stored values. -
CASCommonChem.smiles_to_detailrarely returned the substance asked
for. It passed the SMILES to CAS's search, which matches a SMILES only as
the exact string CAS stores, and took the first hit.CCOfound nothing,
and CAS's ownOCCreturned ethanol-d6. It now searches by the standard
InChI that RDKit writes and keeps only records with that InChI. It prefers
those whose formula matches, which drops the dimers and polymers CAS files
under their repeat unit's InChI, and then the one with the most synonyms
(sodium chloride, not rock salt).CCO,OCCandC(O)Call give
64-17-5. A SMILES RDKit cannot read isstatus"Invalid SMILES", and no
request is made. Each hit costs one cached detail request, up to
CAS_SMILES_MAX_HITS(50). The result is cached likename_to_detail's. -
ZeroPM's mobility probabilities came back under each other's names. In
zeropm-v0-0-4.sqlite, the tablepm_probabilitiesstores
probability_of_min the column namedm_or_vm,vmin the one namedm,
andm_or_vmin the one namedvm. Upstream'srecreate_tables.sql
copies the table with a positionalINSERT ... SELECT *from a CSV that
orders those columnsm, vm, m_or_vm.get_pm_probabilities,
batch_get_pm_probabilitiesandget_all_zeropm_chemicalspassed the
columns through, so TFA read "mobile, not very mobile" (m0.995,vm
0.005) instead of very mobile (vm0.995), and atrazine's
probability_of_vmwas its probability of M or vM (0.986). All three now
read each value under its true name. The fix is checked three ways. The
identitiesnot_m + m_or_vm = 1andm + vm = m_or_vmhold in 99.9% of
rows, against 4% as stored. The 95,334 rows that match upstream's CSV by
value match it column for column. And TFA, acesulfame and 1,4-dioxane
read very mobile. The persistence columns were stored correctly. A test
fails when a ZeroPM release corrects the file.PM_PROBABILITY_COLUMNS
names the fields. -
OPSIN.get_id("")sent the request. The URL then ended inws/.json,
and OPSIN answered about the name "ws". A blank or whitespace-only name is
now aFAILUREwithmessage"empty name", and no request is made. -
datasets.status()andremove("chembl")could not see an interrupted
ChEMBL download. The registry looked forchembl_NN_sqlite.tar.gz, but the
download is saved aschembl_NN.db.tar.gz. A half-finished 5.8 GB.part
was therefore neither counted nor removed. Both routes' real archive names
are now listed. -
CheMBL.search_by_namescanned all 2.9 M compounds on every call, and its
truncated results were not reproducible. The method asked one question of
two tables:SELECT DISTINCT md.molregno FROM molecule_dictionary md LEFT JOIN molecule_synonyms ms ON md.molregno = ms.molregno WHERE LOWER(md.pref_name) = LOWER(?) OR LOWER(ms.synonyms) = LOWER(?) LIMIT ?
Neither arm can use an index in that shape —
LOWER(col) = ?defeats an index
oncol, and anORwhose arms live in different tables across aLEFT JOIN
defeats indexing altogether — so SQLite scannedmolecule_dictionaryin full
for every lookup. Adding expression indexes changes nothing on its own; the
query shape is the problem.It is now a
UNIONof two independently indexable lookups, one per table.
Measured on ChEMBL 36, exact lookups:per call full release, old query 764 ms full release, new query 221 ms compact()extract, old query560 ms compact()extract, new query10 µs The 10 µs comes from the
lower(pref_name)andlower(synonyms)expression
indexes thatCheMBL.compact()already builds into every extract; a full
release has no such indexes and still scans, but it scans two small queries
instead of a join.exact=Falsecannot be indexed either way — a
leading-wildcardLIKEis a scan by construction — and improves from 665 ms
to 275 ms on the extract for the same reason the join is gone.The rewrite returns the same compounds: 0 differences over 200 real names and
25 substring fragments. -
CheMBL.search_by_namereturned a different set of compounds run to run.
LIMITwith noORDER BYreturns whatever the query plan happens to produce,
so a name matching more compounds thanlimitsilently changed which ones
came back — after aVACUUM, an index change or a SQLite upgrade. Comparing
acompact()extract against its source found 25 of 60 names differing in
order, and none in content, which is what put the defect in view.Results are now ordered by
molregno, so a truncated result is thelimit
lowestmolregnovalues and is the same on every call and every copy of the
database. -
ChEMBL's archive extracted straight onto the database it was replacing.
download_database(force=True)renamed the extracted file ontodb_path
and validated afterwards, so an extraction or validation failure destroyed
the release already on disk — and then deleted what was left. The archive now
extracts to<db_path>.incoming, answers a query there, and is moved into
position only once it has. -
Three downloads renamed the file into place before checking it.
CompToxID.download_databasemoved the downloaded file onto the destination
and only then looked for thechemicalstable; a failed check therefore left
the broken file where the working database had been.ZeroPMhad the same
shape and deleted the database on failure, leaving nothing at all.Every download now validates the
.partfile before the rename, which is
whatPubChemIDalone already did. A failed download cannot replace or
remove a working database. -
OPSIN threw away the reason for every failure it reported. OPSIN answers a
name it cannot parse with HTTP 404 and a complete JSON body —
{"status": "FAILURE", "message": "notachemical12345 was uninterpretable due to the following section of the name: ..."}— and the body is the only place
that explanation exists.OPSIN.get_idmapped the status code through a
lookup table and returned without reading it, somessagewas empty for
every failure the module ever reported, including in the WARNING
get_id_from_listlogs, which therefore always read
Failed to get ID for x:with nothing after the colon.provesid.opsin.opsin_classifytreats a 404 as a success for exactly this
reason, andget_idreadsstatusandmessageout of the body. The
status-code table,OPSIN.responses, is gone — there is nothing left for it
to do. -
CAS Common Chemistry cached its own failures as answers.
cas_to_detail
andname_to_detailreport a failure in-band, by returning
{"status": "Timeout", "found": False, ...}, and both were@cachedwith no
skip_if. One timed-out request became a permanent "no such CAS number" on
disk. This is the same defect fixed elsewhere in this release:
is_failure_resultlooks forsuccess: False, and CAS saysfound: False.
Both methods now skip the cache unlessfoundis True — absence included, for
the reasonis_empty_resultgives. -
Old OPSIN and CAS cache entries are retired. Both clients cached failures
before this release, so a name or CAS number looked up during one momentary
outage sits on disk as a permanent failure for a record that exists — and
every cached OPSIN failure carries an emptymessage, because the old code
never read the body that explains it. Neither is fixable in place, so both
clients gained aCACHE_SCHEMA_VERSION = 2inside a new__cache_key__,
which makes version 1 entries unreachable. The CAS key deliberately excludes
the API key: two keys reach the same registry and get the same answer. -
A momentary OPSIN outage raised a bare
KeyError.get_idlooked the
status code up in a three-entry table, so a 503 — not one of the three —
failed inside the lookup rather than being reported. It is retried now, and
reported as a"FAILURE"whosemessagesays what happened. -
docs/api/pubchemview.mddocumentedPubChemView(pause_time=...), a
parameter that has never existed — both snippets raisedTypeError.
docs/api/nci_resolver.mddescribed the default pacing as "3 requests per
second"; it is 0.1 s between requests, so at most 10. -
The persistent cache never hit across processes.
CacheManager._get_cache_key
built its key withjson.dumps(..., default=str)over the call's arguments.
For a bound method the first argument isself, and the defaultstr()of a
client object embeds its memory address
(<PubChemView object at 0x7f...>), so every instance — and every
interpreter — produced a different key. No entry written by one run was ever
reachable from the next: every "cached" method inpubchem,pubchemview,
resolver,cascommonchem,classyfireandopsinre-fetched from the
network on every call while the cache directory filled with unreachable
duplicates.Arguments are now normalised by
cache.stable_key_partinto a form that is
stable across processes: objects reduce to the value they declare through
__cache_key__(), or to their fully qualified class name.PubChemAPI,
PubChemViewandNCIChemicalIdentifierResolverdeclare
__cache_key__as(class path, base_url), so two clients pointing at the
same endpoint share cache entries while two different endpoints stay apart.Existing cache entries are orphaned by the new key scheme. They are inert,
not incorrect; runprovesid.cache.clear_cache()(or
clear_all_service_caches()) to reclaim the disk space. -
A cached
Nonewas indistinguishable from a cache miss._load_from_disk
returnedNoneboth for "no entry" and for "the stored value isNone", so
any function whose result isNonere-ran on every call. It now returns a
_MISSsentinel. -
Failed lookups were cached as answers.
@cachedstored whatever a
function returned, including the{'success': False, 'error': ...}dicts and
the empty lists that the clients hand back after an HTTP 429, a PUG-View
ServerBusy(503) or a timeout. A single transient error therefore became a
permanent "this compound has no data".@cachednow skips storage for any
result thatcache.is_failure_resultrecognises, and takes an optional
skip_ifpredicate for clients that signal failure some other way. -
PubChemViewreported a failed fetch as an absent property.
extract_property_datacaughtPubChemViewError— the base class, which
covers an exhausted retry budget — logged "not found", and returned[].
Combined with the caching bug above, a 503 during
get_melting_point(2244)was stored as "aspirin has no melting point".
Transport failures now propagate; only a genuine 404 yields[].
get_property_tablemakes the same distinction: an empty frame always means
"no such data", never "the fetch failed". -
PubChemAPI.get_compound_synonymsswallowed every error into[], with
the same consequence. A 404 still returns[]; anything else raises.
get_compound_propertieskeeps returning the properties it retrieved when the
follow-up synonym request fails, but now records the failure under
synonyms_errorand, viaskip_if, is not cached while incomplete. -
PubChem sheds load behind 4xx statuses, so absence is now decided by the
fault code, not the HTTP status. Both services describe every error in the
body —{"Fault": {"Code": "PUGVIEW.NotFound", ...}}for a compound that
genuinely has no such data,PUGVIEW.BadRequestfor an unknown heading,
...ServerBusywhen the service is merely busy.pubchem.fault_codeand
pubchemview.fault_coderead it: a transient code is retried whatever status
carries it,NotFound/BadRequestis absence and is not retried, and any
other 4xx is a non-retryable error. Previously an unknown heading (HTTP 400)
cost four requests to learn a permanent answer. -
Absence is no longer persisted at all. Even with the classification above,
a wrongly-reported empty result would be permanent once cached, while
re-fetching an empty one costs a single cheap request. Every cached extraction
method inpubchemviewandPubChemAPI.get_compound_synonymsnow carries a
skip_ifpredicate, so only positive results are stored. The raw response
fetchers (get_property,get_experimental_properties) are unaffected: they
raise on absence and so never had an empty to store. -
Removed an unreachable
return cache_infoleft behind in
PubChemAPI.get_cache_info.
Changed
-
extract_cas_valueschecks the check digit. A CAS-shaped string
whose check digit is wrong is no longer a CAS number, so PubChem's
malformed001-02-2(atrazine) and001-02-7(caffeine) and most
number-shaped InChI fragments are dropped fromSearchcandidates. On
the USEtox 3 list by InChIKey (10,748 queries), 3 rows change, all from
PubChemID: two now report the list's number (914434-22-1→
13601-19-9,13330-20-7→8027-00-7), and one whose only "number"
was001-01-1reports none. By name (1,500) and SMILES (1,000), nothing
changes. -
The sdist leaves out
.claude/and.github/. -
provesid.configlogs instead of printing.set_cas_api_keyreturns
the config file's path,remove_cas_api_keyreturns whether a key was
stored, andshow_configreturns theget_config_info()dict. Each
logs at INFO what it used to print. -
Search(sources=...)chooses the offline databases, and replaces
use_zeropm. It takes a list of keys fromprovesid.sources.SOURCE_KEYS,
one key, or"all". The sources are queried inSOURCE_KEYSorder
whatever order they are given in, so the answer does not depend on it. A
source left out is never opened,datasets="required"does not demand its
dataset, and a client passed for it is ignored with a warning. Unknown
names, or none, raiseValueError. It is a preset setting like the one it
replaces:"balanced"and"strict"query ChEBI, CompTox, PubChemID and
ChEMBL, and"recall"adds ZeroPM.use_zeropm=Trueis now
sources="all", anddf.attrs["settings"]recordssourcesinstead.
Before, the only way to query fewer sources was to pass clients and rely
on the bug fixed above.Search._ALL_SOURCE_KEYSand
Search._DEFAULT_SOURCE_KEYSare gone. UseSOURCE_KEYSand
Search.PRESETS["balanced"]["sources"]instead. -
CompTox SMILES, DTXCID and formula lookups use indexes too, each built
on first use.get_by_smiles,get_by_dtxcidandsearch_by_formula
scanned 1.2 million rows, 0.14 to 0.16 s for each lookup that missed, and
Searchpaid this for each SMILES query. The first lookup by each column
now adds that column's index (0.6 to 1.1 s and 21 to 57 MiB each, logged
once), anddownload_databasebuilds all four with the name index.
A lookup then takes under 0.1 ms. The answers are unchanged: where
several rows share a SMILES, the first is still returned. A read-only
database is scanned as before.comptox.LOOKUP_INDEXESreplaces
comptox.INCHIKEY_INDEX. -
CompTox InChIKey lookups use an index, built on first use. The
downloaded database indexes DTXSID, CASRN and the preferred name only, so
eachCompToxID.get_by_inchikeyscanned 1.2 million rows (0.2 s), and a
skeleton search that missed took 0.9 s.Searchpaid this for each
InChIKey and InChI query, and for each name query withuse_opsin=True.
The first InChIKey lookup now addsidx_inchikey(about 1 s and 41 MiB,
logged once), anddownload_databasebuilds it together with the name
index. A lookup then takes about 0.04 ms. A read-only database is scanned
as before.sources.comptox_skeleton_searchmatches withGLOBinstead of
LIKE:LIKEignores case and so cannot use the index. InChIKeys are
upper case, so it finds the same rows. ACompToxIDbuilt with
object.__new__and_adopt_connectionnow has its index state, so it no
longer raises on its first InChIKey lookup or exact name lookup. -
The README describes the package as it now is: offline first, with
Searchin front. It opens with aSearchrun and its real output, then
coversdatasetsand the five databases with their download and on-disk
sizes, the online clients, and links to the executed tutorials. The Python
badge says 3.12+, matchingrequires-python. ClassyFire is no longer
listed as an available service, and the README points to Chebifier instead. -
The tutorials are executed notebooks, rewritten against the current API.
Each tutorial inexamples/is now an.ipynbwith the outputs of a run,
so the site shows what every call returns. The old.mdtutorials had
broken silently: every ChEBI field printedNone, PubChem View raised on
PropertyData.sourceand plotted °F melting points as °C, the CAS
tutorial ran on a stub that invented records, and the ZeroPM tutorial
wrote indexes and a view into the installed database. New:
examples/search/search_tutorial.ipynbresolves the 1,144 names of the ESOL
solubility set offline and checks every answer against the dataset's own
structures. The ClassyFire tutorial is now a short page saying the service
no longer classifies.scripts/validate_docs_local.shchecks that no
notebook holds an error output and that every link in one names a built
page. -
PubChem name searches match whole names by default.
PubChemAPI.get_cids_by_name,get_compounds_by_nameand
find_cids_comprehensivedefault toname_type="complete", which is
PUG-REST's own default. With"word", any name containing the word matched,
and PubChem's first CID was often another compound.
get_cids_by_name("caffeine")returned[9871508, 56841593, 3081207, 2519, ...]and now returns[2519]; ibuprofen goes from 303 CIDs, the first of
them 24848049, to[3672]. CAS numbers and substance-domain queries give
the same answers either way. Passname_type="word"for the old behaviour.
Searchalready asked for"complete"and is unaffected. The two cached
methods moved to cache version 2, so an answer cached under the old default
is not served for the new one. -
The documentation is rebuilt around the docstrings. Every API page is
now a short lead and a:::directive:docs/api/goes from 4 045
hand-written lines, which had drifted from the code, to 207, and every
public module has a page, including the
offline half that had none (PubChemID,CompToxID,ZeroPM,
REACHDossierID,datasets,taxonomy,cache,config,tools).
What the docstrings do not say moved into eight guides under
docs/guide/: installing the offline databases (new),Search, using the
local databases directly, experimental properties from PubChem, network
behaviour, caching, API keys and Chebifier. The home page and the quick
start lead withSearchanddatasetsrather than the online clients.docs/examples/(symlinks toexamples/) is deleted; a MkDocs hook,
scripts/mkdocs_hooks.py, serves the tutorials straight from
examples/at the same URLs. The.pydemos are no longer rendered as
notebook pages.docs/plans/moved toplans/, and the "Modernization" section left
the nav.- Docstrings are parsed with
docstring_style: auto:chembl,comptox,
reachandzeropmare NumPy-style and had been rendered as Google,
with their example outputs read as Markdown links. A GoogleReturns:
section is one value, where each wrapped line used to render as a
separate returned value. mkdocs build --strictnow also fails on a link to a missing page or
heading.- Docstrings cross-reference with Markdown,
[`Search`][provesid.search.Search],
in place of the Sphinx roles (:class:,:meth:…) that the site showed
literally; 591 of them are links now, and--strictfails on one whose
target is gone. Constants documented with#:comments, which
mkdocstrings does not read, carry attribute docstrings instead, so
DATASETS,Search.PRESETS,LOOKUPS,OUTPUT_COLUMNSand 74 others
now appear in the API reference. A typed return reads(dict): …, which
renders the type in the Type column, not as a name; bulleted lists inside
a section render as lists.
-
Searchasks its sources through one table,provesid.sources.LOOKUPS.
The 47 hand-writtenif client: try: ... except: logblocks in nine
methods ofsearch.pybecame one row per (lookup kind, source) and a single
driver,Search._collect. Adding a source is now one column in
sources.pyrather than an edit to every resolver. Results are unchanged:
437 result rows over seven identifier types and seven configurations
(default, ZeroPM, fuzzy, fuzzy with ZeroPM, InChIKey skeleton, Tanimoto,
strict with salt stripping) were compared with the previous code on the real
databases, and all of them are identical. The dead_candidates_from_name,
_fuzzy_name_candidatesand_most_complete_rowwere deleted, and
search.pyshrank from 3 392 to 2 721 lines. Only the wording of the
warning logged when one source fails changed, to
"<Source> <kind> lookup failed for <query>: <error>". -
Breaking:
PubChemIDandChebiSDFhave modules of their own. The
offline database clients were split off from the online clients they shared
a file with:PubChemID,rdkit_descriptors,RDKIT_DESCRIPTORSand
PUBCHEM_DESCRIPTORSmoved fromprovesid.pubchemtoprovesid.pubchem_id,
andChebiSDFfromprovesid.chebitoprovesid.chebi_sdf. The code
moved unchanged.from provesid import PubChemID, ChebiSDFworks as before,
whilefrom provesid.pubchem import PubChemIDmust become
from provesid.pubchem_id import PubChemID. A side effect is that
import provesid.chebino longer imports RDKit. -
Breaking:
PubChemID()builds a missing database from PubChem's FTP site
instead of downloading it from Zenodo. That is 15.4 GB transferred and
about 12 minutes of processing, against 2.2 GiB; passsource="zenodo"for the
download. A database already on disk is opened as before, whichever way it
was made, anddatasets.plan("pubchem")now reports the FTP route's cost. -
Breaking:
PubChemID.properties()no longer serves the eight computed
descriptors from disk.XLogP,TPSA,Complexity,Charge,
HBondDonorCount,HBondAcceptorCount,RotatableBondCountand
HeavyAtomCountare PubChem's model outputs, not data about a compound; they
now always come from PUG-REST, labelledSource='online', even from a Zenodo
copy that still stores them. Before, the same property came from a months-old
snapshot for local compounds and from live PubChem for the rest, with no way
to tell which.MonoisotopicMassmoves the other way, and is served from
disk by a database built from FTP. Which properties an open database can
answer isdb.offline_properties, and is also the default property list. -
PubChemID.get_by_cas_batch()andget_by_smiles_batch()return the
database's own columns rather than a fixed list that named the descriptor
columns, so a database built from FTP yieldsmonoisotopicmassand no
columns ofNone. -
Breaking:
Searchno longer downloads missing datasets by default. Pass
datasets="auto"for the old behaviour.redownload=Truenow requires
datasets="auto"and raises otherwise, rather than being silently ignored
and handing back a stale copy. -
CheMBL.get_compound()no longer returnsmolfile. It is a quarter of a
full ChEMBL database, nothing in PROVESID consumed it, and it is absent from
the extract above. Build a MOL block fromcanonical_smileswith RDKit when
you need one — RDKit is already a hard dependency. -
CheMBLprefers a compacted extract. Withchembl_37.dband
chembl_37_provesid.dbside by side,CheMBL()opens the extract. Release
number still wins first: an older extract does not beat a newer full release. -
OPSIN.base_urlishttps://www.ebi.ac.uk/opsin/ws/. The Cambridge
address it used to name,opsin.ch.cam.ac.uk, answers every request with a
301 to it (verified live, 2026-09-19), so the old URL worked and cost a
redirect per name.OPSIN.responsesis removed with it. -
PUG-REST and PUG-View read a bare 400 differently, so they have separate
classifiers.pubchem_classifyis nowpugrest_classifyand
pugview_classify. PUG-View takes the heading as a query parameter, so a 400
means "no such heading" — absence. PUG-REST takes its whole query in the URL
path, so a 400 means the path was wrong, and the caller needs PubChem's own
explanation of which property name it misspelled, which absence would discard.
The fault code still decides first for both, so the split only governs a 400
carrying no fault at all. -
PubChemAPI.pause_timeandlast_request_timeare properties over the
shared transport rather than plain attributes.pause_timeis still settable
mid-batch and still takes effect on the next request, retries included;
last_request_timeis read-only. -
ChEBI paces itself at 10 requests per second and retries twice with a
half-second base back-off, where it previously did neither. Two retries rather
than the transport's three: an ontology walk makes many small requests to a
fast service, where a 1-2-4 second curve costs more than the request it is
protecting. EBI publishes no per-IP figure for the ChEBI 2.0 API, so the
pacing is politeness rather than a quoted limit.A ChEBI timeout message changed with the move: it now names the URL and the
number of attempts. -
CAS Common Chemistry paces itself at 5 requests per second and retries,
where it previously did neither. A 401 is never retried, because asking again
with the same key cannot help. -
An unresolvable identifier now costs the NCI resolver one request instead
of four. CACTUS answers an identifier it cannot resolve with HTTP 500
carrying the body<h1>Page not found (404)</h1>— verified live on
2026-09-19 againstthis_is_definitely_not_a_chemical_12345, while
α-glucoseanswers 200. Its status code is not a reliable guide, so
provesid.resolver.nci_classifyreads the body: a 5xx carrying that page is
absence and is raised at once, every other 5xx keeps its usual retryable
reading. Two live tests that took 10.1 s and 9.6 s now take 0.57 s each.The exception type changes with it: an unresolvable identifier now raises
NCIResolverNotFoundError, which is what it always meant, rather than
NCIResolverError("Internal server error").NCIResolverNotFoundErroris a
subclass, soexcept NCIResolverErroris unaffected.Conversely, a genuine transient failure from CACTUS — a 429, a real 5xx, a
timeout — is now retried, where before it was raised on the first attempt. -
PubChem's fault-code classification has one home.
fault_codeand
TRANSIENT_FAULT_CODESwere duplicated verbatim inpubchemand
pubchemview, withpubchemviewcarrying the larger set. Both now live in
provesid.pubchem, joined byABSENCE_FAULT_CODESand by
pubchem_classify, the classifier both services share.
provesid.pubchemview.fault_codeis gone; import it fromprovesid.pubchem. -
NCIChemicalIdentifierResolver.pause_timeand
PubChemView.min_request_intervalare now properties over the transport's
interval. Reading them is unchanged; setting one still takes effect on the
next request, retries included. -
NCIChemicalIdentifierResolverlogs to a module logger instead of the root
logger, anddownload_imagegains the retry and pacing the rest of the
client already had. -
get_property_table'sExperimentalValuecolumn is now a float, not a
string. An entry that reports a range leaves it NaN and fillsValueMinand
ValueMaxinstead, rather than putting"138-140"in a column callers were
expected to plot.Unitis normalised, andTemperatureis a float in °C
instead of a string like"25°C". -
PubChemView._extract_experimental_value_and_unitand_parse_value_string
are gone, replaced byparse_value. Between them they were 415 lines of
property-specific regex cascade, and the two disagreed: the same string parsed
one way throughget_melting_pointand another throughget_property_table. -
Entries that state no value — PubChem's pointers to its own external tables,
such as{"Value": {"ExternalTableName": "iupacpka"}}— are left out of
property results instead of appearing as blank rows. -
The four module-level convenience functions in
pubchemview
(get_experimental_property,get_all_experimental_properties,
get_property_values_only,get_property_table) are no longer cached
themselves. Each delegates to a cachedPubChemViewmethod, so the second
cache only kept a duplicate copy of the same payload on disk — under a key
that did not carryCACHE_SCHEMA_VERSION, which is exactly how an upgrade
would have served a stale shape. -
get_compound_properties_batchnow issues one request per 200 CIDs
instead of one per CID, having been rebuilt onget_properties_for_cids. The
return shape is unchanged — one dict per CID with the properties plus
success,cidanderror— except that a CID PubChem has no record of is
now reported withsuccess=Falseand"No such compound"rather than
whatever the single-CID path happened to return. Theoutput_format
parameter is gone: only JSON can be reshaped into per-CID dicts, so the
parameter promised something it never delivered. Synonyms are not included,
as they need a request per compound; the method never returned them. -
Searchno longer targets the ZeroPM database. ZeroPM harvests regulatory
inventories rather than curating compounds, so its name→structure rows are
noisier than ChEBI/CompTox/PubChem/ChEMBL — while counting as a full
independent vote in the corroboration ranking that 0.6.0 made drive
confidenceandmin_source_support. The resolver now queries four sources
by default, and ZeroPM's database is not even opened.The
ZeroPMclass is untouched and stays fully available for direct use; only
Searchstopped consulting it. Passuse_zeropm=Trueto restore the previous
five-source behaviour. Azeropm=client handed to the constructor is ignored
(with a warning) unlessuse_zeropm=Trueis set too, so "disabled" does not
depend on how the caller happened to build the instance.Consequences:
source_detailsno longer carries a"ZeroPM"entry,
sources_availablelists four keys, and confidence values shift slightly
wherever ZeroPM used to vote. Fuzzy name queries lose recall — ZeroPM was the
only source doing true fuzzy retrieval, so a typo sharing no substring with
the real name (e.g."caffiene") now returns no match instead of a guess.
Precision is unaffected: a misspelling still never resolves to a different
compound, whichtests/test_search_precision_regression.pynow asserts
explicitly for the default source set. -
Search._SOURCE_KEYSis now a per-instance attribute reflecting the sources
that instance targets; the full catalogue lives inSearch._ALL_SOURCE_KEYS
and the default set inSearch._DEFAULT_SOURCE_KEYS.
Removed
ChEBI.get_complete_entityandChEBI.batch_get_entities. They were
compatibility aliases. Useget_compound, andbatch_get_compounds, which
returns the same{CHEBI:<id>: record}dict (its pause defaults to 0.1 s,
not 0).get_compoundsasks for several IDs in one request, but returns
ChEBI's response as it is: each ID maps to an entry withexists, and the
record is underdata.MANIFEST.in. The build backend is hatchling, which never read it.schema_documentation.txtis no longer shipped inprovesid/data/. It
was ChEMBL's schema for release 36, and the releaseCheMBLinstalls is
whichever is latest.chembl.pyandexamples/chembl/README.mdnow point
at ChEMBL's page for each release,
https://ftp.ebi.ac.uk/pub/databases/chembl/ChEMBLdb/releases/chembl_<N>/schema_documentation.html.
provesid/data/examining_pubchem.py, a scratch script that read a CSV
that isn't there, is gone too.examples/notebooks/is gone. Its two CSVs (4 MB) had been unused since
their notebook was deleted.