Repository navigation
v1.0.0
Changelog
Upgrading
-
From 0.4.
Bookshelf 1.0 reads from the Bookshelf platform API rather than the S3 bucket,
and Migrating from 0.4 walks through the upgrade.- Python 3.12 or newer is required.
- pandas and PyArrow are core dependencies, and scmdata is no longer one.
Installbookshelf[scmrun]to readScmRunobjects, andbookshelf[publish]to record notebooks. $BOOKSHELF_REMOTEis replaced by$BOOKSHELF_URL, which names the platform API.$BOOKSHELF_CACHE_LOCATIONis replaced by$BOOKSHELF_CACHE_DIR.
The old name still works but warns, and bookshelf 2.0 removes it.BookShelfandLocalBookstill work and warn at every call.
Bookshelf 1.1 removes them, once the feedstocks have migrated.
An unpinnedload()can now resolve a newer edition than 0.4 did, so pinedition=where it matters.
-
From a 1.0 beta or release candidate.
These are the changes since the betas that can break existing code, grouped by area.Reading data
- The asynchronous surface is removed, so
AsyncBookshelfand the otherAsynchandles are gone.
CallBookshelfthroughasyncio.to_threadfrom async code. - The promised API returns SDK-owned types instead of generated models, and
bookshelf.modelsis private. Book.metadataandVolume.metadataare plain dicts, with properties such asbook.versionandvolume.titlealongside them.BookEntry.entryis removed. Usename_in_book,visibilityandresource_type()instead.ResourceInfo.typeandResourceInfo.hashare renamedresource_typeandcontent_hash.
ResourceInfo.recordandResourceInfo.from_record()are removed.ResourceTypemembers are upper case, such asResourceType.TIMESERIES.
Comparing one with a string still works.- The converters read the whole resource through the content cache, rather than a server-trimmed preview.
query()is removed. Use a converter for complete data, orpreview()for a bounded sample.- Filters are passed as
filters={"region": ["World", "OECD"]}rather than as keywords. preview()returns aDataPreview, andschema()is renamedseries_metadata().Resource.metadataandResource.typeare replaced bydescribe()andresource_type().as_scmrun()raisesNonUniqueMetadataErrorfor rows with duplicate metadata, rather than averaging them.Bookshelf.list_books()is removed. Usevolume.versions,volume.latestandvolume.editions(version).correct_book()takes keyword arguments in place of amodels.BookCorrection.- pandas and PyArrow are core dependencies and the
dataframesextra is gone.
Polars is no longer installed, so install it yourself foras_polars().
Errors
- A missing entry raises
EntryNotFoundError, and a filter on an unknown column raisesSelectionError.
Both are stillKeyErrors. volume["missing"]raisesVersionNotFoundError, which is both aKeyErrorand aNotFoundError.- The 400 and 422 error is renamed
RequestValidationError. - A response of the wrong shape raises the new
ContractError.
GatewayErrornow means only a non-JSON response from something in front of the API. item_errorsmoved fromAPIErrortoConflictError.- Data reads, 500 responses and connect timeouts are no longer retried.
Configuration and authentication
- The default API URL is the production deployment.
Logins are stored per deployment, so runbookshelf auth loginagain. bookshelf.STAGING_API_URLis removed. Pass the staging URL asbase_urlor set$BOOKSHELF_URL.- A base URL with a query string, a fragment, a
user@part or a trailing/v1raisesConfigurationError. - The OS keychain store is removed, so
$BOOKSHELF_USE_KEYCHAINdoes nothing. - Agent identities are removed, along with
bookshelf auth switchand the--agent,--claimand--emaillogin flags.
A stored agent login is ignored.
Publishing
- A recipe has
volume:,defaults:,build:andbooks:sections, and every book states its ownlicense. - A recipe
uriinput must be a publichttpsURL. - Bundles are recorded at schema 3.10, and a bundle recorded by b18 or later still loads.
logical_key=is renamedname=onUsed,RegisterItemand the registration calls.- The
dedupeargument is removed. The server aliases matching bytes unless a bundle pins its tracking id. data_dictionary=moved from book drafting toDraftBook.attach().draft_book(citation_doi=...)is removed. Passdiscovery={"doi": ...}instead.uuid7,replay_bundleandrun_recordare imported frombookshelf.publisherrather thanbookshelf.
replay_bundleis synchronous and takes aBookshelf, andreplay_bundle_syncis removed.bookshelf recordrefuses a-pparameter the build does not assign at the top level,
and fails a build that callssys.exit.- A
bookshelf://reference spells its edition_eNNN, padded to three digits from_e001.
Command line
--api-urlgoes before the subcommand, as inbookshelf --api-url URL publish bundle.bookshelf record --booknames the version to build. The old--versionstill works but is hidden.cache prune --max-bytesis now--max-size-bytes, and the old flag still works but is hidden.- A refused
recordparameter and a malformed base URL exit 2. - The text output is rendered from the
--jsondocument, so parse--jsonrather than the text. - Several
--jsonkeys changed, as listed in the command line output rules.
showlistsentrieswithsize_bytes, counts end in_countand sizes end in_size_bytes.
searchrows andshow VOLUMEreportlatest_licenseandlatest_publisher. auth listexpirednow means only that the access token expired, andneeds_logincarries the old meaning.
The command line interface, its exit codes and its JSON output are outside the
semantic versioning promise, so they can change in a minor release. - The asynchronous surface is removed, so
Deprecations
-
Kept the 0.4 consumer API working on top of the platform, with a
DeprecationWarningat every call.bookshelf.BookShelfandbookshelf.LocalBookstill resolve,
and live inbookshelf.legacyalongside the oldUnknownBook,UnknownVersionandUnknownEditionerrors.BOOKSHELF_CACHE_LOCATIONis honoured as a fallback forBOOKSHELF_CACHE_DIR,
andBOOKSHELF_REMOTEor aremote_bookshelfURL now warns instead of being silently ignored.as_long_df(legacy_columns=True)reproduces the 0.4 long format.
(#203)
-
Deprecated three spellings, which still work with a warning and will be removed in bookshelf 2.0:
as_long_df(legacy_columns=True)warns with aDeprecationWarning. Use the tidyyearandvaluecolumns.$BOOKSHELF_API_URLwarns with aFutureWarning. Set$BOOKSHELF_URL.$BOOKSHELF_CACHE_LOCATIONwarns with aFutureWarning. Set$BOOKSHELF_CACHE_DIR.
(#302)
-
Moved
bookshelf.legacyand the rootBookShelfandLocalBookoutside the semver promise.
They are removed in bookshelf 1.1, once the feedstocks have migrated. (#310)
Features
-
Replaces the legacy V1 consumer library with the public Bookshelf SDK.
Retires the separatebookshelf-producerdistribution,
because its publishing capabilities now live in the SDK. (#136) -
Allows a recorded build to declare
visibilityin its recipe, alongside the collection, licence and authors.
An omittedvisibilityresolves to the recipe's value, then tohidden.
An invalid one is still rejected. (#137) -
Adds
bookshelf record,bookshelf validateandbookshelf publish,
so a feedstock drives the publishing surface through one entry point.recordrefuses to replace an existing bundle unless--forceis passed.validatechecks the book framing, the entries, and every managed resource against its recorded hash.publishreportsno-opfor an already published edition, and--dry-runreports without publishing.- A structurally invalid bundle exits 7.
recordneeds thepublishextra.validateandpublishrun on a core install.
(#138)
-
Adds the volume lifecycle and draft cleanup to the SDK and the CLI.
Bookshelf.create_volume,update_volumeanddelete_volume,
with the matchingbookshelf volumecommands.- Creation needs
WRITEand deletion needsADMIN, so a caller can create a volume it cannot delete. Bookshelf.discard_draftandupdate_draft, withbookshelf discard volume@version_eNNN.
(#142)
-
Adds
data_dictionary=onDraftBook.attach(), so a producer can declare what a book's columns mean.
The dictionary is recorded in the bundle manifest and sent on replay,
so it survives a record and replay round trip. (#145) -
Publishes the bundle format as a specification at
docs/explanation/bundle-format.md.
It describes the bytes on disk rather than the Python interface,
so an implementation in another language can be written from it. (#164) -
Adds catalogue discovery to
Bookshelf.search_volumes()finds volumes by free text plus the discovery filters the CLI already accepted.list_books()returns every book in one volume, oldest first, walking the pages itself.
(#166)
-
Adds
build.use("raw"), which resolves a resource the recipe declares for the selected version.- A
uriresource is fetched through the content cache and verified against its declaredsha256. - A
pathresource is read from beside the recipe, and its digest is computed as it is read. bookshelf.setup()now returns aBuildrather than abs, bookpair.
build.bsandbuild.bookreach the SDK underneath.
(#183)
- A
-
Bakes the editorial metadata a recipe resolves onto each book at publish,
so publishing a new version no longer rewrites what every earlier version says about itself.
Removescitationfrom the volume surface, which the platform moved onto the book. (#188) -
Adds the
bookshelf://scheme to a recipe'sresources:, so a feedstock can build on a published book.uri: bookshelf://primap-hist/v2.7_e002/by_countryresolves to the resource the platform already holds,
so nothing is fetched or catalogued andused=cites the original.- The entry may be left off where the book holds exactly one,
and the edition may be left off to take the newest. - A
bookshelf://resource states nosha256, and may leavetypeout.
(#194)
-
Added v5.1.0.0 of the HadCRUT global mean surface temperature anomaly dataset with the book name 'hadcrut'. (#195)
-
Adds
book.write(...), which registers an output and attaches it under one name in a single call.
book.add(*resources)attaches handles that were registered inside an explicitbs.activity(...)
block, each under the name it registered as.
The sugar records the same bundle as the layered form, so a build may mix the two.
A resource written without atype=is catalogued astabular, and a producer statestimeseries
where the platform's timeseries treatment is wanted.
The implicit activity records under a fixedprocesskind, and its config is seeded from the
parametersbookshelf record -pwas invoked with, so the same build recorded twice is byte identical. (#199) -
Adds seven examples covering the producer behaviour the first set left unpinned.
complex-processingwrites several outputs whoseused=edges follow the processing.defaults-and-overridesshows what a book inherits fromdefaults:and what it replaces,
for the discovery fields and fordefaults.resources:.figurespublishes a png beside the frame it plots, as a document entry carrying no data dictionary.mixed-visibilitypins the precedence rule: the caller, then the book'svisibility, thenhidden,
with one resource narrowed inside a public book.fetch-from-webfetches one upstream url, verified against its declared digest and served from the
cache on later runs, so an upstream change surfaces as a digest failure rather than as a golden diff.low-level-apiis a plain script with its own command line that records a bundle through
RecordingBookshelfdirectly, for a pipeline that publishing is only a small part of.
The runner discovers it by itsrecord.pyrather than by a recipe.reissuerecords one version twice with the processing changed and the data unchanged,
and the test suite asserts that everything the seal covers is identical across the two runs.
(#202)
-
Adds
source_urlmetadata to a checked-in resource, linking to the commit the file was read at.
The bytes are re-hosted, so this is what ties them back to the repository they came from.
The link is omitted when the file itself is uncommitted or has moved away from the commit,
and when the repository states no origin, holds no commit, or sits on an unrecognised forge.
A change elsewhere in the clone does not cost the link. (#205) -
Lets a resource carry its own catalogue metadata, so a book assembled from other people's data
credits them on the thing they made rather than on the book as a whole.
tags,description,authors,doi,citation,licenseandlicense_urlare now declared
underresources:in a recipe for a declared input, and passed tobook.writefor a produced output.The bundle manifest records the fields per resource, and replay sends them.
Also pins down what
authorsandvolume.maintainerseach claim.
authorscredit whoever made the data, so a feedstock that reproduces somebody else's dataset
names them rather than whoever runs the build.
maintainersare whoever runs the feedstock, which is a contact rather than a credit. (#208) -
Added
bookshelf upload FILE --type TYPE, which puts a file on the bookshelf as an input that belongs to no book
and prints thebookshelf://sha256/<hex>URI a recipe declares it by.
A dataset that is embargoed or too large for a repository is uploaded once and named by its digest,
so a book built from it cites the original rather than a re-hosted copy.- A recipe resource may now name
bookshelf://sha256/<hex>beside the book coordinate. Bookshelf.register_fileuploads and registers a file from Python, attributing it to no activity.Bookshelf.resource_by_hashresolves a digest into the resource your organisation holds for it.
(#213)
- A recipe resource may now name
-
Adds
bs.volume(name), which resolves a volume into a handle that knows its versions and editions.
Finding the versions of a volume meant listing its books and reading the labels off each row.
The handle asks the platform once and exposesversions,latestandeditions(version),
and indexing it by version resolves the book:volume["v2.6"], orvolume.book()for the newest. (#215) -
used=now records an input the recipe names bybookshelf://sha256/<hex>.
An uploaded input belongs to no book and the bundle records nothing for it,
so the manifest cites it under a newused_digestsfield on the resource record. (#220) -
Adds
bookshelf preview upload, which stores a feedstock pull request's candidate books as one preview on the platform.
It authenticates with the GitHub Actions OIDC token, so the workflow needs theid-token: writepermission and holds no Bookshelf credential. (#224) -
Added
type="figure"anddata=tobook.write, which saves a matplotlib figure as a png and records the values it plots beside it.
Added thefiguresextra, which installs matplotlib. (#229) -
Added
caption=andalt_text=to figure writes, which land in the figure's discovery metadata.
A public figure without analt_textis now refused when it is written, before any bytes upload. (#230) -
A figure written from a matplotlib figure now also records an svg beside its png master.
Replay uploads the svg with the figure, and the platform serves it as the figure's vector format. (#235) -
Added
Bookshelf.ensure_authenticated()which confirm the API accepts the client's credential.
A missing or spent stored login now starts a browser login from a local terminal,
or a device code login from a notebook, an SSH session or a terminal with no browser.
Without a person to log in, it raises the newAuthenticationRequiredError. (#238) -
Added a GitHub Actions credential step to the ambient chain.
SettingBOOKSHELF_AUTH=github-actionsin a job with theid-token: writepermission
lets the client read its organisation's books with the token GitHub mints for the job,
so a workflow needs no stored Bookshelf secret.
The token is read-only, and a fresh one is minted whenever the API refuses one. (#241) -
Exposed
bookshelf.uuid7()so producers can mint tracking ids before writing data.
Addedwhole_book: trueto recipe resources sobuild.use(name)can resolve an edition's entries by name. (#243) -
Added
register(..., role="plan")for registering a method card as the plan an activity followed.
A plan takes noused=and needs aname, and the platform links it to every output of the activity.
Bundles record the plan'srole. (#244) -
Added
Bookshelf.correct_book(),
which correct a published book's discovery or metadata without minting an edition. (#246) -
Exposed what a publisher tool otherwise copied from the SDK's private modules.
- Exported
bookshelf.PRODUCTION_API_URL. - Added
Bookshelf.get_or_create_volume, which returns the volume and whether this call created it,
treating a 409 from a concurrent creator as already present. - Exported
APIErrorand its typed subclasses, such asNotFoundErrorandConflictError.
(#248)
- Exported
-
Added a
cacheargument toBookshelf,
so a caller can choose theContentCachewithout reaching into a private attribute. (#250) -
Allows a recorded build to place a resource its organisation already holds in its book, under any unused name:
book.attachtakes a handle frombuild.use(), abookshelf://reference or a tracking id.- Replay and previews send the placed resource's
tracking_id, and a placement adds no lineage. bookshelf validatelists every placement.
(#251)
-
bookshelf preview uploadaccepts a bundle whose book entries are pointers, and sends each one by itsexternal_uriwith no bytes. (#253) -
Adds
bookshelf.auth,
so a library calling another Climate Resource service can reuse the Bookshelf credential.default_auth()returns anhttpx.Auth,
andstrict=Trueraises on a spent login instead of continuing anonymously.access_token()returns a current bearer string for clients not built onhttpx.
(#257)
-
Carries a recorded resource's
tracking_idthrough the bundle and replay,
so the platform registers the resource under the id the producer pinned.
Replaying a pin needs a platform that acceptsReplayResource.tracking_id. (#259) -
Restored
bookshelf.__version__and exportedDataFrameSupportErrorfrombookshelf. (#261) -
bookshelf searchandbookshelf show VOLUMEprint the volume'skeywords.
bookshelf show VOLUME@VERSIONprints the book's ownlicense,doi,citationandrelease_url. (#275) -
Every SDK error can now be imported from
bookshelf.
The newly exported errors are:RequestValidationErrorAuthConfigurationErrorActionsTokenErrorOAuthErrorOAuthProtocolErrorEntryNotFoundErrorSelectionErrorInvalidBundleErrorInvalidRecipeErrorInvalidReferenceError
(#285)
-
Adds
server_side=Truetoas_df(),as_long_df(),as_polars(),as_arrow()andas_scmrun().
The platform then selects the rows and years, so only the selection is transferred.
It selects the same rows as the default read of the cached file, and is never capped.
Without anorderthe rows come in the platform's order, which can differ from the file's.Adds
ordertoas_df(),as_polars()andas_arrow(), naming columns to sort by with-for descending.
A year column works as a key, and missing values sort first on either route.Adds
preview()to every resource handle.
It takes the same selection pluslimit,order,top_nanddrop_constant.
Itsordertakes dimension columns only, and cannot be combined withtop_n.
It fetches one row beyond the limit, socompletenessreportspartialonly when rows were left out.Adds
download(destination), which copies the verified file to a path the caller owns.
as_path()still returns the cached file, which the cache may evict later. (#286) -
Added
int_yearstoas_df(), which labels a timeseries' year columns as integers rather than strings. (#288) -
Adds
refresh=TruetoBookshelf.bookandVolume.book.
It skips the remembered edition and resolves the book and its entries afresh. (#289) -
Adds
dropna=Truetoas_long_df(), which leaves out the years with no value without ever building their rows.
This keeps a sparse resource far smaller in memory.
The default is unchanged and still keeps a row for every year. (#294) -
A 429 is now retried for any method, honouring
Retry-After, and raises the newRateLimitErroronce the retries run out.
A malformedBOOKSHELF_URLraises the newConfigurationError, and the CLI exits 2 for it. (#295) -
CLI additions for scripts:
bookshelf showaccepts abookshelf://reference.auth token,auth logoutandcache pathtake--json.search --jsonends with a{"page": ...}line carrying the total and the next offset.volume createaccepts--licenseas well as--licence.record --booknames the version to build.--versionstill works but is hidden, because it clashed withbookshelf --version.
(#301)
-
Added
ResourceTypeandVisibilityenums, plusIdentity,VolumeSummary,VolumeSearchResults,Facets,Facet,
FacetValue,SeriesMetadata,Problem,ItemErrorandBookCorrectionvalue types, all importable frombookshelf.Exported
CacheSummary,ContractErrorandVersionNotFoundErrorfrombookshelf.
correct_book(),update_draft()anddiscard_draft()accept a book id as aUUIDas well as a string,
sobook.book_idcan be passed straight in. (#302)
Improvements
-
Makes
OAuthErroraBookshelfError,
so oneexcept BookshelfErroraround a login catches every flow failure.
The browser and device-code flows raised a bareExceptionsubclass,
so a caller had to catch two unrelated trees to cover a single login.Restores the coverage signal to CI, which the SDK adoption had dropped.
Coverage is measured overpackages/bookshelf/src,
and the root and package configurations now state the same source and the same gate. (#136) -
Reads git provenance through
gitpythonrather than parsinggitoutput. (#137) -
Records a resource with the book's visibility instead of always recording as
hidden.
visibilityinbookshelf.yamlnow sets the tier of the book and of everything the build records.
Passingvisibility=on an individual registration still sets that one resource,
and a build that declares nothing still recordshidden. (#144) -
Generates the client's mechanical operation methods from the
build_*andparse_*pairs in_core/ops.py.
CI regenerates and diffs the result, so a new API operation cannot be silently left unexposed. (#154) -
Retires the seams that only ever had one adapter.
retryandcacheare gone fromBookshelfandBookshelfClient,
andrandis gone fromRetryPolicy.delay. (#157) -
Moves the rules that decide whether a bundle is a replayable published book onto
Bundleitself.
Bundle.validateasserts them and raises the newInvalidBundleError,
andBundle.read_validatedreads and asserts in one call.
bookshelf validatekeeps its output and its exit codes. (#159) -
Publishes a recorded bundle behind
bookshelf.publisher.publish_bundle,
which drafts once and returns aPublishOutcome
carrying the kind, the edition, the resource count and the bundle hash.
bookshelf publishproduces the same output, and the same exit codes, as before. (#161) -
Slims the
publishextra tonbformatandnbconvert, the two libraries notebook capture imports.
Dropspapermill, which brought the wholeaiohttpstack,
along with thejupyter-clientandipykernelpins it needed.
bookshelf recordno longer refuses to run whenpapermillis absent. (#162) -
Splits
bookshelf.publisher.recordinto the driver, the recording adapter and the recipe,
which now live inbookshelf.publisher.record,bookshelf.publisher.recording
andbookshelf.publisher.recipe.
The public interface is unchanged, and every name exported frombookshelf.publisherbehaves the same. (#164) -
Records the pyarrow writer version in the bundle manifest header, under a new optional
writerblock.
Parquet output is not stable across pyarrow versions,
so a change in the recorded content hashes now explains itself. (#178) -
Collapses the four hand-rolled copies of the token exchange onto the credential providers.
bookshelf auth tokennow refreshes a stored login that carries a refresh token but no recorded expiry,
where it used to print it as it stood. (#193) -
Sends the processing fingerprint when drafting a book.
draft_booknow takes the[code_ref, config_hash]pairs of the runs that generated a book's
members, and a recorded book states the fingerprint of the activity that produced it, so
bookshelf validatereports it.
Processing is provenance and never enters the seal, so a rebuild whose code changed but whose data
did not converges on the existing edition. (#199) -
The local cache now remembers each resource's hash and every pinned book edition,
so a warmas_path()orfetch()onbook(volume, version, edition=n)makes no request at all.
A remembered edition is rechecked with one request once a day, tuned byBOOKSHELF_CACHE_BOOK_TTL. (#214) -
Printing a volume, a book, a book entry, a resource or a draft book now describes what is inside it.
The repr is a header line over named sections, the shape xarray and scmdata already use:
the entries and their types, then the reader and explorer calls the type actually supports.
A section with nothing in it is left out, so a document entry does not advertiseas_scmrun(). (#215) -
Moved staging login to the dedicated Bookshelf WorkOS application. (#231)
-
Bundled the production WorkOS client id and pointed
PRODUCTION_API_URLathttps://bookshelf.climateresource.com.au. (#232) -
Moved
gitpythoninto thepublishextra, because only the produce path reads a git repository.
Consumers no longer install it.
Turned thetestextra into a dependency group, so it no longer ships in the published wheel metadata. (#239) -
Downloads each resource once when several processes or threads share a content cache.
Concurrent fetches of one file wait on a lock under<cache>/.locks/instead of downloading it again.
ContentCache.fetchexposes this for callers that fill the cache themselves. (#255) -
Locks the credentials file for every change,
so processes refreshing tokens at the same time no longer lose each other's writes.bookshelf auth whoamireports client credentials as kindmachineinstead ofuser. (#256) -
Selects an external pointer on the platform rather than transferring all of it and selecting locally. (#286)
-
Cached parquet reads such as
as_df()now skip the year columns outsideyear_minandyear_max
and filter rows on text and missing-value filters while the file is read, before the rows reach pandas.
Filtered reads of large resources use a fraction of the memory they did.
A cachedtabularread now numbers its rows from 0 after filtering or ordering, as aserver_sideread already did. (#288) -
Lowers the default
book_ttlfrom one day to one hour, so a remembered pinned edition is checked again sooner. (#289) -
Rendered help, usage errors and tracebacks as plain text, so CLI output no longer changes under a TTY or
FORCE_COLOR.Improved the CLI output in several places:
- Added
bookshelf --version. searchsays when more results exist and how to page, and points at--facetswhen filters match nothing.search --typelists the valid resource types, and--licenseis accepted alongside--licence.search --facetsrefuses a query or filters, which the facets endpoint would ignore.auth listmarks a credential that cannot be renewed without logging in again.auth whoami --offlinereports permissions as null, because they are unknown without the API.showseparates its per-edition and per-resource blocks with a blank line.- Byte counts use binary units, so the 5 GiB cache cap reads
5.0 GiB, and timestamps are given to the second.
(#291)
- Added
-
Tidied the messages
bookshelf validateprints.
A schema error names each field without pydantic's documentation links.
A bundle from a newer schema major names the upgrade as its remedy rather than re-recording.
The summary reports an absolutebundle_path, asrecorddoes. (#292) -
Lowers the peak memory of a filtered cached read, most of all when the filter keeps most of the rows. (#294)
-
Cached reads of wide timeseries such as
as_df()now keep the text label columns dictionary encoded and build the index from the dictionary codes.
The frame is unchanged, but large resources load in about half the memory and time under pandas 2. (#296) -
A read timeout is no longer retried, so a server that stops answering fails after one 30 second timeout instead of about 93 seconds, and the error names the timeout. (#297)
-
as_long_df(dropna=True)now shares the label values between years with the same gaps, so it never costs more memory thandropna=False.
On a dense resource under pandas 3 this cuts the peak memory by about a factor of six.
Cached wide timeseries reads are also faster under pandas 3, and now beat the reads before #296 on both pandas 2 and 3. (#298) -
CLI exit codes are documented on the new command line page and listed under
bookshelf --help:- A conflict exits 8 and a response outside the API contract exits 9, where both exited 1.
- Incomplete credential settings, a bad selection and a build outside a git repository exit 2, where they exited 1.
AuthenticationRequiredErrorexits 3, where it exited 1.- A bundle directory that does not exist exits 2, where it exited 7.
(#301)
-
A resource type, visibility or other enumerated value that a newer platform adds no longer fails to parse.
It arrives as a member of its own, so a resource of a new type stays readable withfetch(),download()andas_path(). (#302)
Bug Fixes
-
Fixes a batch of defects found in review of the SDK adoption:
as_long_dfno longer raisesKeyErroron a wide frame that carries year columns and no dimensions.- Selects the staging WorkOS client from a host label rather than any occurrence of the word in the API URL.
- The browser login no longer fails when an incidental request reaches the loopback callback
before the redirect. - An unparseable server timestamp now surfaces as a validation error rather than a bare
ValueError. - Pairs batch registration outcomes with their request items by the server-reported index.
- Raises a typed error when a registration response commits nothing, instead of
IndexError. - Repairs naive timestamps when listing resources, so both resource endpoints agree.
- A browser login no longer fails because a recent one still holds a port in
TIME_WAIT. - Rejects a malformed cache digest instead of storing a file the cache can never see again.
(#136)
-
Reports why record mode cannot start, rather than reporting one ambiguous cause.
A failure to derive the code reference now names the unmet requirement,
andsetupoutside a recording says no recording is active rather than blaming a missing argument.
An invalidvisibilityin the recipe is reported as a Bookshelf error rather than aTypeError. (#137) -
Fixes publishing a second version of a feedstock whose source data has not changed.
Recording wrote each generated resource's inputs back over the resources already recorded,
so the first output listed itself as its own input and replay then failed to resolve it.
Bundles recorded before this still replay. (#143) -
Normalises the git remote URL before recording it as a book's
code_ref, so only the addressing part is stored.
Producers running from CI should check whether theiroriginis configured in a form that embeds a token, and rotate it if so. (#149) -
Writes the credential store atomically. (#150)
-
Confines the SDK retry policy to requests that are safe to replay,
so a transient 5xx can no longer duplicate a write.- Only idempotent methods are retried on a 5xx. A
POSTor aPATCHnow surfaces the error to the caller. - A network failure on a write is only retried when the connection never came up.
501and505are no longer retried on any method.
(#156)
- Only idempotent methods are retried on a 5xx. A
-
Raises a typed
DataFrameSupportErrorwith an install hint fromas_polars()when Polars is missing,
instead of a bareImportError. (#157) -
Bumps pint to 0.25.3 and flexparser to 0.4 in the lockfile.
The old pair failed to import under Python 3.13 with a frozen dataclassTypeError,
which broke the scmrun extra outright. (#182) -
Adopts the
release_urlandcitationdiscovery fields from the live contract.
The server started returning them andDiscoveryProfileforbids unknown fields,
so listing volumes failed validation and broke the docs build. (#184) -
Falls back to anonymous access when a stored login cannot be refreshed, instead of failing the call.
A spent credential used to deny the caller the public books that need no credential at all.
The warning that replaces the failure names the fix:bookshelf auth logoutto discard the stored credential, orbookshelf auth loginto claim a fresh one. (#187) -
Fixed reading a stored wide timeseries file, whose date-stamped year columns were taken for dimensions. (#203)
-
Fixed published-book iteration to yield entry names in platform order,
solist(book)lists them in order. (#204) -
Records a checked-in
path:resource as managed bytes rather than as a pointer at its repository
path. The platform only accepts anhttpspointer it can fetch again, so a bundle carrying a
checked-in input could be recorded and validated but never published. (#205) -
Stopped a pre-release bump from consuming the changelog fragments.
They describe the release the pre-releases are leading up to, so building the changelog under a
borrcversion files them against a version nobody reads the notes of. (#206) -
A live registration that stated no authors was refused by the platform with a validation error.
An unstated discovery fact is now left off the wire rather than sent as null. (#213) -
Replaying a bundle sent every discovery fact the manifest omitted as an explicit null,
which the platform refused with a validation error naming each resource.
An unstated fact is now left off the replay request, matching what the live path already did. (#217) -
Serialises string, binary, list, struct and categorical columns from a pandas frame
to the same parquet bytes as the equivalent polars frame,
so a record hashes the same whichever library built it.- Casts those columns to large offsets and categorical indices to
uint32,
the widths polars already writes. - Drops library metadata from every column, such as the category labels polars attaches.
- Changes the hash of a resource built from pandas on pandas 2.x,
from a pandas frame holding binary, list, struct or categorical columns,
or from a polars frame holding categorical or enum columns.
Re-recording such a book publishes new bytes rather than converging. - Reads a polars
Enumcolumn back from the parquet file asCategorical.
(#222)
- Casts those columns to large offsets and categorical indices to
-
Preview uploads now carry the build's activity and each resource's lineage, metadata and discovery,
so a book published from a pull request keeps its provenance.
A checked-in input recorded withregister_filetravels under the preview too,
registered with its lineage but attached to no book. (#234) -
Fixed
as_scmrun()dropping timeseries whose values were all missing. (#236) -
Caps
httpxbelow 1.0.
The1.0.devprereleases removehttpx.Auth, sobookshelffailed on import when prereleases were allowed. (#237) -
Re-verified a cached
uriinput before a recorded build uses it,
and raisedHashMismatchErrorwhen a download does not match its declared digest.Kept a resource larger than the cache cap readable after it is downloaded,
where the eviction used to remove it before the caller could read it. (#250) -
Skips writing a refreshed token back to a stored login when that login changed since it was loaded,
so a refresh cannot overwrite a newer login or bring back one that was logged out. (#256) -
Fixed
bookshelf show --help, which crashed because its argument help was read as markup. (#261) -
Applied a recipe's
volume:section to the volume on publish.
Recording used to dropmaintainers,keywords,update_cadenceand the deprecation fields silently.
Sent only the fields the recipe states, so a volume fact it leaves out keeps its value. (#277) -
Refreshes a pinned edition's book metadata and visibility when the edition is checked again after
book_ttl.
A correction made withcorrect_bookpreviously never reached a cached edition. (#289) -
Fixed
bookshelf showreporting the resource id astracking_idand leavingcontent_hashnull.Refused malformed CLI addresses with exit 2 before they reach the API.
Dot segments, a second@, and malformed or non-ASCII editions used to parse and then fail with exit 5,
andshow .printed a traceback.Reported an unusable cache directory as a usage error naming
BOOKSHELF_CACHE_DIRinstead of a traceback.Named the
$BOOKSHELF_TOKENremedy when the server rejects the env token, rather thanbookshelf auth login. (#291) -
Fixed
bookshelf validatepassing bundles edited by hand after recording.
It now refuses:- a resource name recorded twice, or a book entry that appears twice
- a managed resource whose recorded
sizediffers from its bytes - a book
visibilityother thanhidden,orgorpublic - a volume or version the platform cannot address, such as one over 100 or 50 characters, or
../../etc processingthat does not match the activity- an activity whose
config_hashis not the digest of its parameters and whoseactivity_idis not derived from them - a pointer the platform would refuse, such as
file:///etc/passwd, or one whose bytes still sit in the bundle - a byte file that is a symlink resolving outside the bundle, which replay also refuses now
Validate also hashes resources in chunks, so its memory no longer grows with the largest file.
(#292)
-
Made
bookshelf recordrefuse unsafe input before running the build, and record a clean code ref.--forceno longer replaces a file, a non-empty directory withoutmanifest.lock, or a directory holding the recipe, build file or working directory.- A
-pvalue that is malformed YAML, a date, binary, NaN or infinity now exits 2 before the build runs, rather than crashing after it or recording NaN asnull. code_refis read before the bundle is staged and before the build runs, so a clean clone no longer records+dirty, and a build that changes directory still records its clone.- A build that calls
sys.exitnow fails the record with a message, rather than exiting with no bundle. - An unreadable recipe, a key stated twice in a recipe and an invalid resource name now raise a
BookshelfErrornaming the problem. - A lone surrogate printed by a build no longer crashes the notebook render.
- Clearer messages for a numeric version in the recipe, a book coordinate input passed to
used=, a resource named like the executed notebook, and a build that writes no outputs.
(#293)
-
Accepts numpy arrays, pandas indexes and series, sets and numpy scalars as filter values and year bounds.
Previouslyfilters={"region": df.index.unique("region")}raisedTypeError.Refuses a malformed read before any request is made:
book()with a version that is not a non-empty string, or an edition below 1.preview()with alimitortop_nthat is not a positive whole number.resource_by_hash()with anything but asha256:digest, which now raisesValueErrorrather thanNotFoundError.
Names
int_yearswhenas_df(int_years=True)is refused on a tabular resource.An unknown filter column on a
server_side=Trueread now lists the columns, as a cached read does.(#294)
-
Fixed several transport failures.
- An HTML error page no longer appears in an error message, which now names the status and the URL instead.
bs.volume(".")andbookshelf show .no longer reach the volume list endpoint.- Network failures and timeouts now name the URL they failed on.
RequestValidationErrormessages now list the field errors.- Removing the cache directory during a session no longer breaks later reads, and a cache path blocked by a file raises the new
CacheDirectoryError. - Waiting for the stored credentials lock now times out after 30 seconds instead of hanging.
(#295)
-
A success response that is not the JSON the operation returns, such as the web app's page answering a base URL with the wrong path, now raises
GatewayError(CLI exit 6) naming the status, content type and URL, instead of a rawJSONDecodeError,ValidationErrororAttributeError.
A rate limited download from the content CDN is now retried, honouringRetry-After, and a 429 message now says it was rate limited and gives the status.
A failed token refresh no longer quotes an HTML body, and the CLI prints a warning as oneWarning:line rather than a Python warning with source lines.
bookshelf auth whoaminow says whether a rejected token expired, is malformed, or is revoked or issued for another deployment. (#297) -
Fixed several edge cases in consuming resources.
- A NaN,
pd.NAorpd.NaTfilter value now selects the missing rows, asNonedoes, on both the cached and the server side route. Listed with other values it raises the sameValueErrorasNoneinstead of silently dropping the missing rows. - A read no longer fails when another process clears the cached file under it. The file is downloaded again once.
- A read-only cache directory now raises
CacheDirectoryErrorinstead of a rawPermissionErrorwhen it misses, and still serves the files it holds. resource_by_hash()now accepts upper case hex instead of reporting the resource as missing.as_arrow()no longer adds an__index_level_0__column to tabular resources.
(#298)
- A NaN,
-
Fixed the packaging for 1.0.
- The wheel and the sdist now carry the MIT licence, and the sdist no longer ships the tests.
- Raised the floors that did not work:
typer>=0.26,nbconvert>=7.3,nbformat>=5.1andpyyaml>=6.0.2. - A notebook render that fails on import now names the underlying
ImportError. - Importing a removed 0.4 submodule such as
bookshelf.shelfnow raises anImportErrorthat links the migration guide.
(#300)
-
Fixes for the credentials file, versions and recorded activities:
- A credentials file written by a newer
bookshelfis read and never overwritten, where it was renamed tocredentials.json.unreadableand the newer install was logged out. Logging in or out against it fails with exit code 2. - A book version ending in
_eand digits, which cannot be addressed, is refused in recipes and bundles. - An activity's
runnerislocaloutside CI rather than the machine's hostname, andCI=falseno longer counts as CI.
(#301)
- A credentials file written by a newer
-
Fixes found by the 1.0.0rc2 smoke test:
series_metadata()on an entry that is not a timeseries raisesUnsupportedConversionError,
without asking the platform.- An error status the API contract does not declare, such as a proxy's JSON 403,
exits by its status class rather than 9. bookshelf auth loginexits 2 before contacting anything
when the credentials file belongs to a newer bookshelf,
or when no WorkOS client ID is known for the deployment.- Against a newer credentials file, errors suggest upgrading or setting
BOOKSHELF_TOKEN
rather than logging in or out, andauth listmarks an expired token asneeds_login. - A credentials file whose
versionis a string such as"3"is treated as newer rather than set aside. - A recipe refused for its content no longer suggests pointing
--recipeat a different file. - The
figuresextra installs seaborn, whichScmRun.lineplot()needs.
(#304)
-
Fixed
preview()failing validation on a resource with ageometrycolumn. (#310)
Improved Documentation
- Adds executed how-to guides covering both sides of the SDK:
reading a book, converting and plotting, reading asynchronously,
publishing a book, and cataloguing external data.
Each guide runs when the docs are built, so its output is real.
Also corrects the entry name in the existing examples, which wasmagicc-rcmipand ismagicc. (#166) - Adds
examples/, one directory per example, each a miniature feedstock with its own recipe, build
file and golden manifest.
examples/run_all.pyrecords every example, validates it and compares it against its golden, and it
exits non-zero when any of them fails.
The examples are both the reference thatcopier-bookshelf-datasetscaffolds from and the
regression fixtures that catch an accidental change to the bundle format. (#199) - Limits the API reference to the stable reading surface of the SDK.
Only what it documents is covered by Semantic Versioning.
The producer methods,bookshelf.publisher,bookshelf.modelsand the CLI can still change in any release. (#223) - Adds a standalone Authentication page covering every way to authenticate.
It also documents the credential precedence chain and where credentials are stored.
Configurationnow covers configuration alone and links across. (#238) - Added a migration guide for code that reads books with bookshelf 0.4,
and fixed stale claims across the authentication, development, recipe and bundle pages. (#261) - Clarified that a null
volume:field never clears a volume fact.
An explicit empty list such askeywords: []does clear that list. (#278) - Adds a command line page (output rules, exit codes, a generated command reference) and an addressing page for
volume@version_eNNN/entryandbookshelf://references. The configuration page now lists the CI and SSH variables the SDK reads, and the authentication page says the credentials file format is private. (#301) - The API reference no longer shows handle constructors, and says handles come from a client.
PartialRegistrationErrorand the publisher errors are promised as classes,
while an attribute returning a generated model is provisional.
The credentials page describes where the file lives whenXDG_CONFIG_HOMEis set. (#304) - Added an API reference page for
bookshelf.auth.
The promise now coversdefault_auth,access_token,StaticTokenandClientCredentials, and the other provider classes may change in any release. (#310)
Trivial/Internal Changes
Changes
- 58142d0 bump: version 1.0.0rc3 -> 1.0.0
- 94dfbda docs: fold the pre-release breaking changes into one upgrading note
- bea9eaa docs: add changelog for #311
- 00bdf04 chore: upgrade the locked packages with security advisories
- b3baa1c docs: add changelog for #310
- f0078f8 docs: settle the 1.0 promise for auth and the 0.4 API
- f97c94f chore: refresh the vendored openapi.json from platform 658f91f3