v0.11.0

Adds a refit bootstrap and a warning class for experimental methods to hdfe_stream.core, for
the packages built on it. Fits are unchanged.

• A refit bootstrap for packages built on hdfe_stream.core (core.refit). It
  resamples whole units, or whole clusters of units (each copy's units with IDs of their
  own), with replacement, writes each draw to disk in bounded memory, and refits a
  consumer's estimator on it, in the same process or in worker processes with their share
  of the job's threads and memory. The workers start once and fit one draw after another
  (restarted every 200 draws), with one resample per worker on disk at a time. Failed draws
  are recorded, never redrawn; RefitBootstrapResult gives bootstrap standard errors and
  percentile, basic and normal intervals. Its parts are stable names too: core.partition (whole-group partitions on
  disk), core.resample (DiskGroupResampler) and core.process (ProcessBatch). They
  were diff-stream's, and diff-stream's refit_bootstrap now uses them.
• ExperimentalWarning (hdfe_stream.ExperimentalWarning, also
  hdfe_stream.core.report.ExperimentalWarning): a warning class for methods that go beyond
  what their authors developed. Packages built on hdfe_stream.core subclass it for their own
  experimental features, so filtering this class silences all of them.

v0.10.0

Works with pyfixest 0.70 as well as 0.60, and accepts pyfixest 0.70's formula syntax. Fits with many
variables no longer slow down sharply under a tight memory limit. The RMSE with analytic weights
is defined as in fixest.

• pyfixest 0.60 through 0.70. Formulas, etable, summary and iplot work with either.
  Install either version with hdfe-stream[formula].
• Fixed-effect interactions can be written firm_id:year, as in pyfixest 0.70.
  firm_id^year still works with both versions, without a deprecation warning.
• Instrumental variables can be written y ~ x + [endog ~ instrument] | fe
  (pyfixest 0.70 and later). y ~ x | fe | endog ~ instrument works with both versions.
• Several outcomes can be written sw(y, y2) ~ x. y + y2 ~ x still works.
• Varying slopes can be expressions (pyfixest 0.70 and later): worker_id[t, t ** 2].
  They can also be on an interacted dimension, worker_id:firm_id[t].
  Slopes without the intercept, worker_id[[t]], are not available yet and say so.
• Coefficient and model names are pyfixest's own for the version installed. With pyfixest
  0.70, a model's fml is pyfixest 0.70's spelling of it (y ~ 1 + x, a:b).
• For packages built on hdfe_stream.core: hdfe_stream.core.formula.parse_formula returns
  the models of a formula in one form for both pyfixest versions, in place of reading
  pyfixest's parsed formula directly. pyfixest_formula_api is unchanged.
• The RMSE of a fit with analytic weights no longer depends on the scale of the weights. It is
  now the weighted sum of squared residuals divided by the sum of the weights, as in fixest and
  pyfixest 0.70, where it was divided by the number of rows. Coefficients, standard
  errors and R-squared are unchanged. pyfixest 0.60 still divides by the number of rows.
• Benchmarks re-run with pyfixest 0.70.0, whose default demeaner (LSMR) is the one compared against;
  the alternating-projections (MAP) demeaner is no longer benchmarked.
• Fits with more than 8 variables are much faster under a tight memory limit. The solve needs
  an array with a row for every fixed-effect level and a column for every variable. When that
  array was more than a fifth of the memory left under max_memory_gb, the fit instead wrote a
  file of every cell's sums (30 GB or more on a large panel) and read all of it back for each
  group of 8 variables. Now the array goes into a file of its own, which is far smaller (0.8 GB
  for 2 million levels and 50 variables), and the solve reads it one group at a time. The estimates agree with
  the fit that keeps the array in memory to 1e-15. Fits with 8 or fewer variables, and fits where the array fits
  in memory, are not affected.
• The array's file stops counting against the memory limit once the first pass ends. The
  pages of a memory-mapped file count as memory in use, and with them counted a fit near its
  limit could find no room left and choose slower methods for the solve. The file is now released
  when the first pass finishes and read back a block at a time.

v0.9.2

Faster fits on large panels with many variables, and less scratch disk. Results are unchanged.

• Large fits with more than 8 variables are faster. An example is Sun-Abraham's
  estimator in diff-stream, which has a variable for each cohort and period. Before, the fit
  worked through the variables 8 at a time, and for each group of 8 it read the whole file of
  stored cell sums from disk. When that file is bigger than the computer's memory, those
  repeated reads were most of the run time. The fit now gets what it needs from the sums as it
  writes them, so the file is not read again. On 8 million units (64 million rows), the
  solving step dropped from 284 to 19 seconds and the whole fit from 542 to 294 seconds.
  The process also holds less memory: its peak fell from 20.6 to 7.1 GB, because the file's
  pages no longer pile up in memory. When the file is small enough to stay in memory,
  there is little change in speed.
• Less scratch disk. That file of cell sums is now written only when something reads it:
  the within solver, the cell-based way of building the normal equations (the default for 8
  or fewer variables), or the old one-group-at-a-time path described above. For other fits, a
  wide design for example, it is not written at all. In the same Sun-Abraham fit with a
  million units, scratch disk use fell from 3.6 to 0.4 GB, and the saving grows with the
  number of rows.
• Memory limit. Building what the fit needs in one go takes the number of fixed-effect
  levels times the number of variables times 8 bytes. If that is more than a fifth of the
  memory left under max_memory_gb, the fit works through the variables 8 at a time, as in
  0.9.1, which is slower but uses less memory. The rhs_block option still sets the size of
  those groups.

v0.9.1

• HDFEResult.solve_normal_equations. Solves a linear fit's normal equations for
  right-hand sides built from rows: for weight columns v_j of a frame, u_j = S^- Z'v_j, with S
  the fit's weighted Gram matrix of fixed effects and covariates. The result,
  NormalEquationsSolution, gives the solutions as fixed-effect tables and covariate
  coefficients, and predict gives z'u_j for any rows. For a weighted sum of predictions, w_i
  z_i'u_j is the weight that estimation row i's outcome gets in it, which is what imputation
  estimators (Borusyak, Jaravel and Spiess) need for their standard errors; diff-stream uses it
  for BJS with high-dimensional fixed effects. It needs a fit made with
  keep_intermediates=True. Nothing else changes.
• pyarrow 16 or later. The floor was pyarrow 14, but pyarrow 14 and 15 were built against
  NumPy 1 and fail to import with NumPy 2, which the other floors allow: the lowest versions
  resolved on Python 3.12 gave an installation that could not import. The NumPy floor is
  unchanged.

v0.9.0

• Building blocks for treatment-effect estimators in hdfe_stream.core, moved from
  match-stream so that match-stream and diff-stream share them. Their names are stable (listed
  in hdfe_stream.core).
  • core.influence: influence-function variances from per-cluster sums (one-way and
    multi-way), the weighted mean difference, and the joint variance of estimators fitted on
    different tables. ClusterScoreAccumulator builds the one-way clustered variance batch by
    batch, in memory that does not grow with the number of clusters: 0.8 GB, against 4.5 GB,
    for 2 million clusters and 20 estimators.
  • core.mestimation: a fitted logit or probit first stage, and the correction of an
    influence function for estimating it.
  • core.ipw: inverse-probability weights, and the IPW difference in means corrected for the
    estimated score.
  • core.balance: a weighted two-group balance table (means, standardized mean differences,
    variance ratios, eCDF gaps) with MatchIt's conventions. Covariates with zero variance are
    allowed: a standardized difference whose scale is 0 is 0, or ±inf when the means differ,
    and the variance ratio is inf when only the control group's variance is 0, null when both
    are.
• Less memory reading Parquet in batches. core.store.iter_batches and iter_groups no
  longer use pyarrow's pre-buffering, which held the selected columns of the whole file as it
  read: 4.7 GB extra for a 4.4 GB file, now 0.3 GB. Packages built on them gain most: logit-stream
  and dpd-stream fits peak 0.25-0.5 GB lower. Results are unchanged.
• Memory tests. pytest -m resource fits OLS and a logit on a 10-million-row panel in a
  child process under a 1.6 GB limit, which a fit holding the 0.5 GB panel in memory would
  exceed (2.6 GB on macOS and 2.0 GB on Windows, where the memory measured includes mapped
  files or untouched allocations). CI runs them on Linux, macOS and Windows; the default
  pytest run skips them. The test extra now includes stream-limits[testing].

v0.8.0

• Fit resources (CPUs and memory) are constrained to your job's limits automatically. On a cluster (PBS, Slurm, or a container with cgroup limits), importing hdfe_stream caps the threads of Polars, numba, BLAS and pyarrow at the job's CPU allocation, and with Polars 2.x gives Polars a memory budget so its streaming operations spill to disk instead of growing. Caveats: the Polars budget is a target for its buffers, not a hard cap on the process, and Polars 1.x has no such option, so there only the threads are capped. The caps must be in place before Polars is imported: import hdfe_stream first, or call stream_limits.set_limits() at the top of the script. With no limit found nothing changes. See "Resource limits and profiling" in the documentation.
• profile=True: estimate a fit's memory and disk before running it. feols_stream, fepois_stream, feglm_stream and leave_out_kss take profile=True and return a hdfe_stream.Profile instead of a result: one streaming scan of the data, the model's arrays sized from what it finds, an expected size and an upper bound for each, the peak by phase, and recommend() / recommend().pbs as a resource request. Nothing is fitted and no run directory is made. The shared code (Profile, Component, scan_source) is in hdfe_stream.core.profile for logit-stream and dpd-stream, which take the same option. See "Resource limits and profiling" in the documentation.
• Depends on the new stream-limits package (standard library only), which does the limiting for all three packages and works on its own; its documentation covers the thread, memory and scheduler details.
• max_memory_gb and memory_reserve: choices follow the memory left. max_s_gb's default (25% of physical memory) and the CRV3, bootcluster and IV-diagnostics guards (25%) used the node's memory and a fixed share of it, which a correctly sized PBS job always hit. They now ask whether their exact allocation fits in the headroom: the limit (the job's allocation, or max_memory_gb), less a reserve (memory_reserve, default 0.15), less what the process holds. S is built if it fits, otherwise stream_cg streams it; per-thread accumulators drop to fewer copies; pass 0's sort buckets shrink in a small allocation; what cannot shrink raises MemoryError up front with the numbers.
• Removed: max_s_gb. It capped the explicit S and the per-thread accumulators separately from the rest of the memory. The headroom rule above subsumes it: lower max_memory_gb to tighten the whole fit, or give solver="stream_cg" to skip the explicit S. Passing max_s_gb is now a TypeError.
• n_threads= is scoped to the fit. It used to call numba.set_num_threads in the constructor and never restore it; it now holds numba and BLAS to that many threads for the length of fit() (and of leave_out_kss) and restores them.
• diagnostics["peak_rss_gb"]: the peak resident memory of each fit, from the kernel's high-water mark (no polling).
• New hdfe_stream.core.resources and hdfe_stream.core.profile (stable names).

v0.7.0

• device="cuda" runs GLMs without fixed effects on the GPU. Poisson,
  logit and probit with no fixed effects (feglm_stream, fepois_stream) did
  nothing on the GPU before: its solver exists only with two or more fixed
  effects. Now the rows are uploaded once and every IRLS step (the family's
  terms, the weighted cross-products) runs on the device, so later steps do
  not read the Parquet rows again. If the rows do not fit in the device's free
  memory the fit warns and uploads them again at each step. Estimates agree
  with the CPU's to rounding.
• Benchmarks for OLS, IV and logit/probit with zero or one fixed effects
  against scikit-learn, polars-ols, polars-ds, statsmodels, linearmodels,
  glum, pyfixest and R. Also the same models on a GPU against cuML and
  CuPy. Results and discussion added to documentation. The benchmark
  extra now installs the comparison packages.

v0.6.2

• Fixed incompatibility with polars-runtime-64 (#5). With that runtime installed, polars' row indices are UInt64 rather than UInt32, whatever the size of the data, and the level codes and row_id built from them failed in NumPy (bincount) and numba. They are now Int64. No estimate changes.

v0.6.1

• The blocked copy that stacks a chunk's columns is used only where making
  the copy pays off, in wider designs. This fixes a minor performance regression
  introduced in 0.6.0 based on a fixed cost to the first fit in every process. No
  estimate changes.

v0.6.0

• IV diagnostics on 2SLS results (HDFEResult.iv_diag, and in
  summary()), under the model's vcov, with ivreg2's conventions:
  Kleibergen-Paap rk LM underidentification and rk Wald weak-identification
  F (Anderson's LM and Cragg-Donald's F under iid errors),
  Sanderson-Windmeijer conditional F statistics, first-stage partial and
  Shea partial R^2, Montiel Olea and Pflueger's effective F with its
  critical value (one endogenous regressor), the Anderson-Rubin test and
  (one endogenous regressor) confidence set, the endogeneity C statistic
  (ivreg2's endog()), and Hansen's J (Sargan's statistic under iid
  errors).
• Log-likelihoods and information criteria. Results have aic and
  bic (the coefficients plus the fixed effects' degrees of freedom, as
  fixest counts them), OLS results the normal loglik (with analytic
  weights, sum(log w) / 2 added, as statsmodels' WLS), GLM results
  loglik_null (constant only), and GLMs with fixed effects
  loglik_fe_null and pseudo_r2_within (fixest's within pseudo R2: what
  the covariates add beyond the fixed effects).
• Incidental-parameter bias correction for logit and probit with one or
  two fixed effects (bias_correction=True, or {"time": ..., "lags": L}
  for predetermined regressors): the analytic correction of Fernández-Val
  and Weidner (2016). Not for Poisson (no such bias with one or two fixed
  effects), nor three-way models.
• GLMs rescale analytic weights to average 1 over the estimation sample
  (Stata's convention for maximum likelihood): before, the iid vcov and the
  log-likelihood depended on the weights' scale (weights x 10: iid standard
  errors / sqrt(10)). Coefficients and robust/clustered vcovs are unchanged;
  fepois iid standard errors with analytic weights now differ from
  pyfixest's, which uses the weights as given.
• Stable names in hdfe_stream.core, listed in its docstring and pinned
  by a test: they change only in a minor release, with a changelog entry. This
  permits use by other (forthcoming) estimation packages.
• hdfe_stream.core.store (new): a block store of rows (BlockStore)
  for estimators that read every row in each iteration, with what pass 0
  needs to fill it. hdfe-stream's own estimators do not use it, but it is
  available for other estimators.
• iter_groups sorts within groups by sort_by on its in-order path
  too
• Results record each vcov's small-sample factor, for a caller estimator that
  converts to other conventions.
• Faster row reads. Stacking a chunk's columns into the row-major
  design (_stack, every pass of every model) is a blocked parallel copy:
  0.04 s instead of 0.47 s for 2 million rows of 39 columns.
• GLMs (IRLS): BLAS on one thread, smaller chunks. OpenBLAS's threads and
  numba's contended in the row passes, the more the smaller the chunks; up
  to 128 columns the passes now hold BLAS to one thread. Adds a dependency
  on threadpoolctl. Estimates are unchanged to rounding.
• GLMs without fixed effects: a fused, parallel row pass. Their IRLS
  row pass was NumPy on one thread, so it gained nothing from more
  cores; it is now one numba sweep like the fixed-effects path's. Estimates are
  unchanged to rounding, and do not depend on the number of threads.
• Buckets sized by the rows' width. Pass 0 sorts each bucket in memory;
  rows_per_bucket now defaults to about 384 MB of rows at their width
  (500,000 to 20,000,000 rows) instead of 20,000,000 rows. The sort itself is
  unchanged.
• Linear models without fixed effects also default to 64 MB chunks, reducing
  peak memory.