Skip to content

v1.15.0

Latest
Latest

Choose a tag to compare

View all tags
@PasteurBot PasteurBot released this 15 Oct 18:42
v1.15.0
7b22082
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Release v1.15.0

Action required

These changes only affect you if you use the experimental GPU transports (cuda_ipc), or read Tesseract.supported_gpu_transports.

Rebuild Tesseracts that use cuda_ipc

cuda_ipc now identifies GPUs by UUID instead of by ordinal, since two processes can number the same GPU differently (e.g. when a container only sees one of the host's GPUs). This changes the wire format. A 1.15 SDK raises if you request a GPU transport from a 1.13 or 1.14 runtime, and older SDKs cannot read GPU arrays from a 1.15 runtime. Upgrade client and Tesseract together. Calls without a GPU transport keep working across versions.

GPU transports are opt-in per request

Setting TESSERACT_GPU_TRANSPORT on the server now only enables a GPU transport. The server uses it only for requests that ask for it, and sends host copies otherwise.

  • SDK clients created with gpu_transport= (e.g. Tesseract.from_image(..., gpu_transport="cuda_ipc")) request it on every call, so they behave as before. All other SDK calls request gpu_transport=none.
  • Raw HTTP clients must ask for it explicitly, e.g. Accept: application/json+base64; gpu_transport=cuda_ipc.
  • The server rejects cuda_ipc inputs with a 422 error unless the transport is enabled.

GPU transport is only supported with a single worker

Serving a GPU transport other than none with num_workers > 1 now raises an error. The server keeps a response's GPU memory until the client confirms it has copied the arrays out, and that confirmation may reach a different worker.

Tesseract.supported_gpu_transports is removed

Use Tesseract.server_capabilities.gpu_transports instead (see below).

Highlights

Find a working GPU transport automatically, and share cuda_ipc servers safely

Whether cuda_ipc works depends on how client and server are set up (same host, same GPU visible to both, container isolation), which neither side can tell on its own. Tesseract.resolve_gpu_transport() finds out by exchanging a small GPU array with the server once per connection:

tess = Tesseract.from_url(url)
transport = tess.resolve_gpu_transport()  # "cuda_ipc" if it works from here, else "none" with a warning
tess.with_encoding(gpu_transport=transport).apply({"x": gpu_array})

If the Tesseract was created with a specific gpu_transport, that transport is returned once it is confirmed to work, and a RuntimeError explains why if it does not. Tesseract-JAX and Tesseract-Torch use this mechanism to keep GPU arrays on the device whenever possible.

cuda_ipc is also safe for concurrent clients and threads now. Previously, a server freed the GPU memory of every response at the start of the next request, so with several clients, a client could silently read another client's results. Each response's memory is now kept until the SDK reports that it has copied the arrays out (raw HTTP clients can do the same via the Tesseract-Exports-Done header, see the array encodings reference).

cuda_ipc overhead is also much lower. In our benchmark of a near no-op JAX Tesseract on an A100, a forward call with a 1 GiB array went from 110 ms to 12 ms, and an apply plus VJP from 275 ms to 28 ms. Several cuda_ipc edge cases are fixed too: PyTorch tensors that require grad, empty arrays, strided arrays, and exports from threads that have made no CUDA call before.

Choose the encoding per call with with_encoding

A served Tesseract's --output-format and TESSERACT_COMPRESSION now only set defaults, and TESSERACT_GPU_TRANSPORT only enables a transport. Each request picks its output format, compression and GPU transport through the Accept header. If the server cannot serve any of the accepted encodings, it returns 406 Not Acceptable before running the endpoint.

In the SDK, server_capabilities lists what the server accepts, with_encoding returns a view that requests a different encoding for every call made through it, and current_encoding tells you what a Tesseract or view requests:

with Tesseract.from_image("vectoradd", gpu_transport="cuda_ipc") as tess:
tess.server_capabilities.compressions  # ('none', 'lz4')
tess.with_encoding(gpu_transport="none", compression="lz4").apply(inputs)

Return on-disk arrays without loading them (experimental)

An Array output field now also accepts a BinrefArray, which references a buffer on disk. This is useful for Tesseracts that stream large results to disk to keep memory bounded. With json+binref output, the buffer is forwarded as is without reading it. Other formats load and encode it like a normal array, with a RuntimeWarning.

from tesseract_core.runtime.experimental import BinrefArray

def apply(inputs):
run_solver(out="result.bin")
result = BinrefArray.from_file("result.bin", shape=(N,), dtype="float64")
return OutputSchema(result=result)

BinrefArray.write(array) writes a NumPy array to a new buffer, and BinrefWriter packs many small arrays into a few shared buffers. See the new binref_passthrough example.

Smaller images, and conda without a conda base image

provider: conda no longer requires a base image that ships conda. If the base image lacks conda, Miniforge is installed in the build stage only and left out of the final image. The conda example, which now uses the default base image, shrinks from 984 MB to 563 MB. Large package_data is no longer stored up to three times in the image, and runtime startup is about 0.3 s faster because MLflow is only imported when it is used.

As a result of the package_data change, tesseract_api.py is now 444 (was 555) in built images, and /tesseract/input_data keeps 755 (was 777).

Other changes

  • Gradient paths match whole names. Several fixes make jacobian, jvp and vjp resolve paths exactly: a path no longer matches a field whose name merely contains it, dict keys are looked up before attributes, and absent optional leaves are skipped.
  • Fields called shape work over HTTP. The SDK failed to decode any response with a dict containing a shape key, including gradients of a differentiable input called shape.
  • The JAX VJP cache is faster. It no longer copies GPU inputs to the host for its key, and lookups no longer slow down with cache size.
  • GPU and binref inputs mix. With json+binref and an input_path, a request now sends GPU arrays via cuda_ipc and host arrays as binref files, instead of using one encoding for all inputs.
  • from_source Tesseracts now report the name, version and description from their tesseract_config.yaml.
  • Tesseract files and the metrics file are read and written as UTF-8 on every platform, which fixes failures on Windows.
  • Arrays with non-native byte order are converted before base64 and binref encoding.
  • Invalid JVP inputs now return a validation error instead of crashing.

What's Changed

Features

  • (sdk) Drop conda base image requirement and shrink built images (#803)
  • (sdk) Unify input array encoding and decouple binref from output format (#795)
  • (runtime) Forward on-disk binrefs through Array fields via BinrefArray (#743)
  • [breaking] Negotiate compression and GPU transport per request via the Accept header (#805)
  • [breaking] Auto-resolve a GPU transport that works between client and server, and make cuda_ipc safe for concurrent calls (#829)

Bug Fixes

  • (runtime) Convert array defaults to ShapeDType in abstract_eval schema (#796)
  • (runtime) Look up dict keys before attributes in gradient paths (#810)
  • Convert non-native byte order arrays before base64 and binref encoding (#811)
  • (runtime) Match whole field name when resolving gradient paths (#814)
  • (runtime) Convert recipe inputs to framework arrays (#802)
  • (runtime) Skip absent optional leaves when expanding gradient paths (#812)
  • (runtime) Accept any abstract_eval as eval_fn in jacobian_from_vjp (#815)
  • (runtime) Correct expected Jacobian shape when input or output shape is Ellipsis (#822)
  • (runtime) Accept PyTorch CUDA tensors in Jacobian result validation (#825)
  • (sdk) Remove the serve log on Windows when a Tesseract is removed (#831)
  • (runtime) Report invalid inputs to a JVP request instead of crashing (#835)
  • (runtime) Match gradient paths in full, not anywhere in the string (#836)
  • (sdk) Decode only what the server marks as an array (#837)
  • (sdk) Read and write a Tesseract's files as UTF-8 on every platform (#832)
  • (runtime) Write the metrics file as UTF-8 on every platform (#834)
  • (sdk) Pass the config's name, version and description to from_source Tesseracts (#841)
  • Show videos in forum mirrors of blog posts and stay within the forum's topic rate limit (#848)

Refactor

  • Drop DLPack detection from GPU transports, since JAX exposes CAI (#804)
  • Tidy follow-ups to per-request encoding and cuda_ipc pools (#828)

Documentation

  • Hackathon winners blog announcement (#809)
  • Run tutorial snippets as end-to-end tests and fix stale examples (#820)
  • Mirror blog posts and release notes to the forum as comment threads, fine-tune forum positioning (#792)

Performance

  • (runtime) Cut overhead when using cuda_ipc (#806)
  • (runtime) Key the JAX VJP cache by an on-device fingerprint for all arrays (#830)

Full diff: v1.14.0...v1.15.0

Discuss this release on the Tesseract Forum, including questions and help with upgrading.

Assets 2
Loading