Skip to content

v0.9.0

Choose a tag to compare

@leetucker leetucker released this 05 Nov 19:39
· 21 commits to main since this release
  • Building blocks for treatment-effect estimators in hdfe_stream.core, moved from
    match-stream so that match-stream and diff-stream share them. Their names are stable (listed
    in hdfe_stream.core).
    • core.influence: influence-function variances from per-cluster sums (one-way and
      multi-way), the weighted mean difference, and the joint variance of estimators fitted on
      different tables. ClusterScoreAccumulator builds the one-way clustered variance batch by
      batch, in memory that does not grow with the number of clusters: 0.8 GB, against 4.5 GB,
      for 2 million clusters and 20 estimators.
    • core.mestimation: a fitted logit or probit first stage, and the correction of an
      influence function for estimating it.
    • core.ipw: inverse-probability weights, and the IPW difference in means corrected for the
      estimated score.
    • core.balance: a weighted two-group balance table (means, standardized mean differences,
      variance ratios, eCDF gaps) with MatchIt's conventions. Covariates with zero variance are
      allowed: a standardized difference whose scale is 0 is 0, or ±inf when the means differ,
      and the variance ratio is inf when only the control group's variance is 0, null when both
      are.
  • Less memory reading Parquet in batches. core.store.iter_batches and iter_groups no
    longer use pyarrow's pre-buffering, which held the selected columns of the whole file as it
    read: 4.7 GB extra for a 4.4 GB file, now 0.3 GB. Packages built on them gain most: logit-stream
    and dpd-stream fits peak 0.25-0.5 GB lower. Results are unchanged.
  • Memory tests. pytest -m resource fits OLS and a logit on a 10-million-row panel in a
    child process under a 1.6 GB limit, which a fit holding the 0.5 GB panel in memory would
    exceed (2.6 GB on macOS and 2.0 GB on Windows, where the memory measured includes mapped
    files or untouched allocations). CI runs them on Linux, macOS and Windows; the default
    pytest run skips them. The test extra now includes stream-limits[testing].