Skip to content

v0.2.0

Choose a tag to compare

@izzet izzet released this 29 Jul 05:18
· 18 commits to main since this release
aedede7

WisIO 0.2.0 is a modernization release. Every Python version 0.1.1 supported
had reached end of life, so the package could not be installed on a current
interpreter at all. This release moves the floor to 3.10, adds 3.13, makes the
analysis several times faster, and ships a web app.

Breaking changes

  • requires-python moves from >=3.8,<3.11 to >=3.10, and 3.13 joins the
    supported set. CI covers 3.10 through 3.13.
  • New required dependency: betterframe[dask]>=0.3.0, which supplies the
    engine adapter behind the pandas view path below.
  • DFTracer Parquet output changed schema. The hash, fhash and hhash
    columns are now string rather than Int64, because current DFTracer writes
    hex digests that overflow a signed 64-bit integer. Anything reading WisIO's
    Parquet output directly will see the difference.
  • Dependency floors raised: dask ~=2023.4 to >=2024.1, numpy unpinned
    from ==1.24.3. pandas stays below 3.0 deliberately -- copy-on-write and the
    default string dtype are a migration of their own.
  • The DFTracer reader is now dftracer-utils, replacing the zindex-based
    reader.
  • The [darshan] extra is 3.10-3.12 only. pydarshan publishes CPython
    wheels only to cp312; on 3.13 the extra resolves to nothing rather than
    installing a build with no libdarshan-util.so that fails at import. WisIO
    itself supports 3.13 -- DFTracer and Recorder analysis work there and are
    covered by CI.

Performance

Views are small -- on the DFTracer fixture the largest is 78 rows -- but each
one was paying for a Dask graph to be built and shipped. The cost tracked view
count, not trace size: 60x the data made no difference. Views that are provably
small are now computed in pandas, and the view loop, scoring and rule engine run
on whichever frame they are handed.

fixture wall clock perspectives attach reasons
dftracer-posix 68 → 19s 9.91 → 0.85s 10.15 → 3.30s
dftracer-dlio 59 → 23s 9.93 → 0.99s 10.05 → 3.63s
recorder-parquet 54 → 21s 8.76 → 0.96s 9.49 → 3.70s

Characterization drops from ~270s to ~97s. Both engines are held to the same
result by a test that runs the whole analyzer twice and compares the Parquet
output, dtypes exactly and values to a tolerance well inside floating-point
summation order.

WisIO Web

A Streamlit app for uploading traces and reading the findings, sized to fit
Streamlit Community Cloud's resource limits. Findings are grouped by view in an
accordion, led by severity, capped at the top 20 per view, and restricted to
root and logical views so a real trace does not produce hundreds of entries.
Uploads are capped at 20 MB; the worker mode is chosen from the upload size.

Analysis output

Each bottleneck now says why it is severe, in both the console and the web
app: how much of the I/O time it takes relative to its share of operations.

Fixes

  • The cluster and client are shut down when a run finishes. They were left
    running, which a one-shot CLI run never notices because process exit takes the
    workers with it. Any process analyzing more than one trace leaked a full
    cluster per analysis, and the next analysis started its own on top.
  • Dask end-to-end path repaired for modern dask-expr. Meta derivation runs
    predicates against synthetic frames, delayed() cannot unpack dask-expr
    collections, and map_partitions does not substitute collection kwargs --
    each broke a different part of the pipeline.
  • Expensive work moved out of the dask-expr query optimizer.
    repartition(partition_size=) measured memory while the graph was still being
    built, which deadlocked on 3.10 and 3.11 and merely cost several times too
    much on 3.12.
  • An optional reader that fails on import no longer takes down the package.
    The guard caught ModuleNotFoundError only, but pydarshan raises
    RuntimeError and a broken zindex_py wheel raised ImportError.
  • import wisio works off the main thread. dask_jobqueue calls
    signal.signal at import, which fails on a worker thread.
  • Builds against current Arrow headers, which need C++20 for std::span,
    std::popcount and std::bit_width.
  • DXT proc_name uses the real hostname.