Skip to content

v0.15.0

Choose a tag to compare

View all tags
@justapithecus justapithecus released this 02 Nov 02:03
v0.15.0
7b1229b
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Artifact bytes in Lode Volumes, written and read by range

Summary

Each artifact is now one Lode Volume that the runtime fills range by range, so an artifact up to 1 GiB never sits whole in memory. Scripts write large artifacts with emit.artifactWriter, Go readers open them with quarry/reader, and quarry prune removes them with their runs.

Highlights

  • Artifacts in Lode Volumes: the runtime commits each range as it arrives under artifacts/<dataset>/<run key>/volumes/<artifact_id>/. The commit record names the Volume snapshot and the SHA-256 the runtime computed.
  • Range frames: the executor sends artifact_open, ranges of at most 8 MiB in any order, and the artifact event, and the runtime acknowledges each range once it is durable.
  • emit.artifactWriter: a script declares an artifact's size and optional digest, writes it with writeAt in any order, and commits it once the ranges cover it.
  • Artifact reads: ListArtifacts and OpenArtifact in quarry/reader read a run's artifacts as a stream or by range, from a Volume or from the chunk records of earlier runs.
  • Retention: quarry prune removes a run's Volumes before its records, the sweep removes orphaned ones, and an unterminated run's age also counts its Volume commits.
  • Lode v0.14.0: commits succeed on Hetzner Object Storage, and a retried artifact range completes its commit.

Breaking Changes

  • --flush-mode, the policy.flush_mode config key, and policy.FlushMode are removed. A run that passes them fails to start.
  • Policies and sinks handle events only: IngestArtifactChunk, WriteChunks, the chunk statistics, policy.NewCompositeSink, and the types.ArtifactChunk family are removed.
  • The artifact_chunk frame is retired, and the run report counts ranges where it counted chunks.

Upgrade Notes

  • Remove --flush-mode from quarry run commands and flush_mode from the policy section of config files.
  • Drop WriteChunks from a custom policy.Sink or lode.Client, and read artifacts through quarry/reader.
  • A custom executor sends artifact_open and artifact_range frames and waits for each artifact_range_ack. The bundled executor does.

Full Changelog: v0.14.2...v0.15.0

Assets 15
Loading