Skip to content

v0.2.0

Choose a tag to compare

@github-actions github-actions released this 26 Oct 15:04
· 4 commits to master since this release
v0.2.0
aec21bd
  • Breaking: Requires Swift 6.3 or later. Deployment minimums are iOS 18
    and macOS 15 for all library and command-line products. Swift 6.0 through
    6.2 remain on 0.1.x.
  • Breaking: Benchmarks measure several metrics. Timed samples record
    wall-clock and CPU time and values a workload records with
    Benchmark.record(_:for:); a separate counting pass records Swift runtime
    heap allocations, allocated bytes, retains and releases (swiftAllocations,
    swiftAllocatedBytes, swiftRetains, swiftReleases; C malloc and
    Objective-C allocations are not counted) on Apple platforms; and a step
    measured in a process of its own records its peak memory.
    Sample, measurement rows, statistics, baselines, budgets, verdict causes
    and Dimension curves are keyed by metric, and Report JSON no longer carries
    schemaVersion. Reports and baselines written by 0.1.x cannot be read:
    decoding a report fails with an error that says to generate it again, and
    the CLI says to record a baseline again.
  • Short workloads are timed in batches: automatic batching grows each
    sample's call count until it lasts the target duration and reports per-call
    values. BatchingPolicy configures it and --batch-size fixes it.
  • swift package benchmark run, check and trace measure each step in a
    host process of its own, which isolates steps from each other and lets each
    record its peak memory. With --xctrace-output, one traced process measures
    every selected step.
  • Breaking: Comparisons use the median by default instead of the mean, and
    compare each Dimension size or argument row with the same row of the
    baseline. The measured metric's direction decides which change regresses.
    CLI options are --statistic, --threshold <n>%|<metric>=<amount>,
    --budget <metric>:<statistic>:<amount> and
    --case-budget <suite>:<case>:<metric>:<statistic>:<amount>, replacing
    --baseline-metric, --threshold-percent, --threshold-ns and the
    nanosecond budget forms.
  • Breaking: check refuses to judge a debug build unless given
    --allow-debug. Reports warn when a baseline came from different run
    conditions, when the run is a debug build, and when a comparison has too few
    samples or too much noise for its threshold.
  • Breaking: The CLI rejects unknown options and no longer accepts the
    sample alias. --optional-diagnostic and --required-diagnostic are
    removed; exit status 30 reports a missing required trace or a failed
    recording. --swift-version records the compiler that built the host.
  • Breaking: The markdown report format, the Memory product, the MetricKit
    placeholder types and diagnostic metrics (DiagnosticMetric,
    MetricProvider, CaseReport.diagnostics) are removed. Trace attachments
    remain.
  • Breaking: Report's public API keeps the report model, comparison
    policies, baselines, ReportRecorder, ReportBuilder, timeline attribution
    and BenchmarkHost.run(discoveries:arguments:metadata:). BenchmarkHost.run
    no longer throws; it exits with status 2 for an invalid request or
    configuration and 1 for any other failure. The benchmark command, report
    renderers and formats, the host protocol, xctrace recording and trace
    exporters are no longer public.
  • Breaking: Step, row and report identifiers keep suite and case names as
    written, with punctuation percent-encoded, so names that differ only in case
    or punctuation stay distinct. An argument row is identified by its values'
    descriptions instead of its position, so reordering arguments keeps baseline
    rows matched.
  • Breaking: Benchmark.Event.Kind drops the discovery, suite,
    diagnostic, attachment and run-ended kinds, which no run emitted.
  • Breaking: The BenchmarkTesting .benchmark trait takes only a
    configuration. BenchmarkTestingAdapter and BenchmarkXCTestAdapter take
    the BenchmarkRunner to measure with.
  • The Macros package trait, enabled by default, declares the Instruments
    macros. Depending on swift-benchmark with traits: [] gives the Instruments
    runtime API without building swift-syntax for it, with Swift 6.4 or later.
  • The command plugin discovers declarations with SwiftSyntax alone, including
    @Test attributes on the function's line and members of extensions, and
    compiles the host with SwiftPM's toolchain. With Swift 6.3, run it with
    swift package --disable-experimental-prebuilts benchmark ..., because
    SwiftPM 6.3 cannot build its SwiftSyntax tool beside macros that use
    SwiftSyntax prebuilts. Hosts find the package's resource bundles and link
    XCFramework binary targets and the C++ standard library.
    SWIFT_BENCHMARK_DISCOVERY_BACKEND and
    SWIFT_BENCHMARK_SWIFTSYNTAX_TOOL_PATH are removed.
  • The plugin builds only the targets that declare benchmarks and their
    dependencies. A release run in a package that declares benchmarks in any
    test target still builds every test target, whatever the run selects.
  • A release host is compiled with optimization, so generated testing bridges
    run optimized inside measured batches, and the plugin no longer loads module
    maps from inside XCFrameworks.
  • Traced commands no longer hang when xctrace or the benchmarks print more
    than a pipe holds.
  • Timeline attributions of argument rows belong to their own row's samples.
    The iteration span records the row in TimelineCorrelationKeys.row, and
    BenchmarkExecutionContext.rowID names it for observers; previously every
    row's iteration matched every other row's samples.
  • A configuration the host rejects, and a baseline or report file that cannot
    be read, exit with the configuration status 2 instead of 3.
  • BenchmarkXCTest builds with swift-corelibs-xctest; attachments(for:)
    needs Apple's XCTest.