Skip to content

v0.10.0 - Hand-written API clients and bucket file transfer

Choose a tag to compare

View all tags
@github-actions github-actions released this 15 Sep 13:43
· 19 commits to main since this release
v0.10.0
ea72f7f

MATLAB Versions Tested

This release replaces the OpenAPI-generated API clients with hand-written ones on a shared HTTP base, adds file upload, download, and delete for Data Proxy buckets, and makes the Knowledge Graph stage argument consistent with the API. It also repairs a series of bugs in the bucket and KG functions, most of which were found by the new unit and live-integration test suites, and brings every public function and class up to the MathWorks help conventions.

Breaking changes

  • The generated namespaces ebrains.kgcore, ebrains.dataproxy, and ebrains.collaboratory no longer exist (#12). Use ebrains.kg.api.InstancesClient, ebrains.kg.api.QueriesClient, ebrains.bucket.api.BucketsClient, and ebrains.collab.api.CollabsClient instead.
  • KGStage no longer has an ANY member (#6). getInstance and getInstancesBulk take the stage as an ordered vector instead: pass ["RELEASED", "IN_PROGRESS"] where "ANY" was used before.
  • getInstance, getInstancesBulk, listTypes, and runDynamicQuery now default to the RELEASED stage (#6). Callers that omitted the stage and relied on drafts now get released data, or a 404 for an unreleased instance. Pass "IN_PROGRESS" or the two-stage vector explicitly.
  • EBRAINS_TOKEN is read from the environment only (#24). A token kept in the MATLAB secret store is no longer picked up and must be exported as an environment variable.

Added

  • ebrains.bucket.uploadFile, ebrains.bucket.downloadFile, and ebrains.bucket.getFileSize for transferring files to and from Data Proxy buckets, with progress display and Client= injection (#10). Downloads arrive in a .part file and are moved into place on completion, so a failed transfer leaves the target untouched.
  • ebrains.bucket.deleteObject for single objects and, with a trailing slash, whole folders (#16).
  • resetAll on the base IAM token client, which clears every OIDC token client singleton at once (723dd5d).
  • getInstancesBulk gains an optional missingIds output; a caller that asks for it gets no warning and handles the misses itself (d07a56e).
  • ebrains.bucket.createVirtualBucket gains the Client= name-value argument the rest of the namespace already had (#18).
  • codemeta.json describing the toolbox, kept current by the release workflow from this release onwards (#15).
  • The filedownload library is vendored as ebrains.external.filedownload, so getBucketObject no longer depends on an unrelated checkout being on the path (#13).
  • A scheduled live-integration workflow runs the tests tagged LiveIntegration weekly against the real Data Proxy, Knowledge Graph, and Collaboratory APIs, and the release workflow runs them again before packaging (#17).
  • Unit tests for the pure helpers, the mocked API clients, the three IAM token clients, and the transfer step of uploadFile and downloadFile. Statement coverage rises from about 45% to about 89% (#18, #19, #21).

Changed

  • The KG, Data Proxy, and Collaboratory clients are hand-written on a shared ebrains.common.internal.HttpClient, replacing the generated packages whose JSON mapper was slow on large responses and could not carry free-form JSON-LD bodies. The IAM classes move into the ebrains.iam namespace folder (#11).
  • getBucketSize reads the total from the bucket stat endpoint instead of listing and summing every object, which is about 15 times faster on large buckets and works on an empty one (#7).
  • getInstance with a stage vector retries the next stage only on 404; authorization and server errors are thrown immediately (#5).
  • The help text of InstancesClient, QueriesClient, authenticate, getTokenManager, the ebrains.iam package, the KG enums, ReturnOptions, BaseClient, and the ebrains.common.constant functions follows the MathWorks help conventions, with syntax paragraphs derived from each arguments block (#22, #23, #25, #26, #27, #28, #29).
  • The README documents the current API: the instances client, the bucket transfer functions, collab search, the client credentials flow, R2022b as the minimum release, and installation from the toolbox file (85f75e7).

Removed

  • The generated ebrains.kgcore, ebrains.dataproxy, and ebrains.collaboratory packages, their build scripts, build logs, and the committed Data Proxy spec (#12).
  • KGStage.ANY (#6).
  • The MATLAB secret-store lookup for EBRAINS_TOKEN, which cost a five-second probe per token client on headless runners and tied the constructor to functions that only exist from R2024a (#24).

Fixed

  • listBucketObjects returned only the first page. The Data Proxy caps a page at about 1000 objects regardless of the requested limit, so any larger bucket came back truncated (a0e4af8).
  • renameObject raised on every call, including successful ones, because the status check compared a number against "OK" (#7).
  • getBucketObject failed for every object since the vendored downloader started treating URLs as already encoded, and for any object inside a folder because / was sent percent-encoded as one path segment (#10).
  • Temporary URLs from the Data Proxy contain raw spaces, which the URI parser rejected; getDownloadUrl and getUploadUrl now percent-encode them (#10).
  • getBucketObject creates the folder that will hold the file, so object names with / download instead of failing at file open (#7).
  • createVirtualBucket produced file names with a literal \ for every object name containing a space, because of shell quoting in the touch call. Files are created with fopen instead, which also works on Windows (fd12bc3).
  • getInstancesBulk with a stage fallback reported misses against the wrong stage, advised retrying a stage already tried, and errored when exactly one id was missing. Misses are now reported once, after every stage was searched (d07a56e).
  • Server error text never reached the user from the KG clients, because the response body was read with isfield on an object (#5).
  • The bulk instances endpoint received a JSON string instead of an array when exactly one identifier was requested (#5).
  • updateInstance, replaceInstance, moveInstance, and deleteInstance did not strip the KG IRI prefix, so an @id taken from listInstances produced a URL nested inside a URL (#5).
  • QueriesClient.getQuery called a normalisation function that did not exist (#5).
  • Passing "ANY" to listInstances, listTypes, or runDynamicQuery was forwarded to the server and failed there; it now fails at argument validation (#6).
  • Three bugs in the vendored filedownload release: double percent-encoding of already encoded URLs, an invalid default for the upload request message that failed every call at validation, and a progress-monitor crash on the second update in command-window mode (#13).
  • The vendored downloadFile now raises on a non-2xx status instead of writing the error body to the target file as if it were the content (#10).
  • getDataSizeLabel(0) raised an index error, and so did any size at or above 1e18 bytes (#18).
  • namespacedir raised a generic output-count error for a namespace not on the path; it now names the namespace and the folder to add (#18).
  • Tests tagged Graphical no longer run in CI on R2022b, where a headless msgbox returns a placeholder figure (9a56a79).

Full Changelog: v0.9.2...v0.10.0

Assets 3
Loading