v3.0.0

v3.0.0 finishes the v3 work begun in v3.0.0-alpha. This is a substantially more powerful and robust, yet faster feature set, having cut much of the redundancy and fixed a range of implementation issues.

Every registered operation was reviewed, many were fixed or redefined so that they are robust and reproducible, and redundant features were removed. The default feature set now has 6392 features from 927 master operations, down from 7709 features from 1080 master operations in v2.0.0. Please note that future developments to this project are planned to happen in pyhctsa.

Before you upgrade

• Recompute; don't mix results across versions. Only 3207 of the 6392 feature names also exist in v2.0.0, and some features that kept their names now compute different values.
• Renames and deregistrations are logged in FeatureSets/renamed_operations.csv (old_name,new_name,reason, where an empty new_name means deregistered), with a reason for each. Every v2.0.0 feature name that is no longer registered can be traced through it to its current name or to its removal (6176 entries: 3505 renames and 2671 deregistrations, some chained).
• Values changed without a rename:
  • All features that use the 'mi' delay (it now has a more robust definition; see below).
  • The GARCH operations (MF_GARCHfit, MF_GARCHcompare) no longer use the Econometrics Toolbox's estimate. hctsa specifies the likelihood (Gaussian, GJR and Student-t), the bounds and a deterministic optimizer (a grid of starts, then Newton steps with the analytic Hessian), so its fits are reproducible. The variance model is reported only when a likelihood-ratio test finds conditional heteroskedasticity (otherwise the coefficients are 0), and outputs computed from the conditional variance are NaN when the fit stops at the lower bound of the variance constant. Standard errors and other outputs that did not respond to the variance structure were deregistered; 16 GARCH features remain.
  • The MF_StateSpaceCompOrder AIC, which is now per sample.
  • The MF_CompareAR loss, which is normalized differently.
  • DN_Moments is standardized by the population standard deviation (orders 3 and 4 are the skewness and kurtosis).
  • Sample entropy (EN_SampEn) is NaN when there are no matches of length m+1 (the estimate is undefined; it was 0), and EN_MSE uses refined composite multiscale entropy (Wu et al. 2014), with summaries over scales NaN if any scale is undefined.
  • WL_modwtvar uses the Percival & Walden MODWT wavelet-variance estimator; ST_MomentCorr uses only full windows.
  • The two Kraskov CO_AddNoise operations average 5 noise draws per noise level (was 10), halving their cost; the other settings still use 10 (CO_AddNoise has a new numDraws input).
  • NL_c1 uses at most 1000 reference points (was 2500), which changes values only for series longer than 2000 samples [for speed--tests showed only small differences but substantial time saving].
  • Bug fixes in PP_Compare/PP_Iterate (NaN only when the processed series is floating-point round-off), the HMM training log-likelihood (MF_hmm_*), NL_nlpe (delay from the analyzed segment), NL_PersistentHomology, CO_Embed2_AngleTau, PP_SchreiberDenoise and EN_DispEn.
  • Sinusoid fits (PP_Compare sin1, SP_SinusoidFit) fit the constant jointly, need at least one cycle in the window, and converge, so exact sums of sinusoids fit to round-off.
  • Repeated points and tied distances (within 1e-12 of the range) are handled consistently in NL_FNN, NL_LargestLyap, NL_EVTLocalDim, NL_c1 and NL_ReturnTime: repeated points are not neighbors, and equally near neighbors are averaged. This changes values only on series with exact repeats or ties. NL_c1 and NL_FractalDimensions are NaN where no estimate exists (exact periodic orbits, an optimizer stopped at its bound), and NL_DVV uses a two-pass variance.
  • TISEAN d2 now writes its correlation sums at full precision (previously 6 significant digits), which changes NL_d2 and NL_GPCorrSum values slightly. The NL_d2 Takens statistics use only the embedding dimensions with a valid estimate (at least 33 pairs, at most 2m).
  • Gaussian-process fits (MF_GP_Hyperparameters, MF_GP_FitAcross) run to convergence (previously they stopped after 50 optimizer evaluations), within bounds: the length scale lies between the sampling interval and the span of the data, and the signal and noise amplitudes are at least 1% of its standard deviation. Where the fitted model does not beat its white-noise special case in a likelihood-ratio test, the length scale is reported at its bound, one sampling interval, so a series without Gaussian-process structure gets that value rather than wherever the optimizer stopped. The predictive outputs are those of the smooth component, with all noise treated as observation noise. Outputs that were not reproducible or were redundant were deregistered (111 features), including the periodic-kernel fits and all MF_GP_LocalPrediction calls.
• Parallel computation on Linux: operations that call TISEAN or ripser now read each program's output through a file. On Linux under load (e.g. TS_Compute with many parallel workers on one node), MATLAB's system() could return part of a program's output to the next call, giving truncated output that either errored or was parsed into a wrong value (e.g. a NaN from PP_SchreiberDenoise). If you computed TISEAN-based features in parallel on Linux with an earlier version, recompute them. In 'series' mode, TS_Compute now also reports each operation's error message.
• Re-run install.m after updating, because some compiled C sources changed.
• Matlab R2022a or later is required.
• The mySQL database backend was removed in v3.0.0-alpha. hctsa now works only with local .mat files.

Highlights since v2.0.0

• Sampling-rate-robust delays. BF_GetTau gives every operation the same two delay rules:

  • 'ac1e': the 1/e crossing of the autocorrelation function.
  • 'mi': the first minimum of the Kraskov automutual information, capped at 'ac1e'. It is now computed on tie-aware ranks and smoothed, so it is stable under tiny perturbations of the input.

  Each operation's adaptive delay was chosen by test, and rate-adapted (_dec) variants sit alongside the tau = 1 originals.

• Robust, deterministic, portable features.

  • A portable random-number generator (BF_Random, MRG32k3a) makes stochastic features reproducible across MATLAB releases and platforms.
  • Fits that depended on where an optimizer stopped were replaced with specified algorithms: Theil–Sen lines, variable-projection exponential and sinusoid fits, Burg partial autocorrelations, and deterministic multi-start HMMs. Gaussian-process fits run to convergence.
  • Explicit bin edges and bandwidths; features that changed under rounding-level input perturbations were redefined or deregistered.

• Fewer toolbox dependencies. The System Identification Toolbox is no longer needed: all state-space, ARMA and AR model fits (including those in PP_ModelFit and in the whitening step of the GARCH operations) use hctsa's own implementations. The GARCH fits and their ARCH and Ljung–Box tests no longer need the Econometrics Toolbox, which only 7 master operations (19 features: SY_VarRatioTest, SY_PPtest, SY_KPSStest and the Ljung–Box test of HT_IndependenceTests) still use. Fewer operations need the Curve Fitting Toolbox.

• Correctness fixes and less redundancy. Many operations were fixed (the larger fixes are in the v3.0.0-alpha notes), length bias was removed, and near-duplicate, constant and by-construction features were deregistered. A final library-wide review deregistered 140 more features, each with a partner at Spearman |ρ| ≥ 0.99 on both the Empirical1000 and a 1000-series synthetic corpus, and features that were not reproducible across noise or surrogate seeds were removed (e.g. 40 CO_AddNoise fields).

• Theiler windows. Neighbor-based operations (recurrence, dimension, Lyapunov, prediction and nearest-neighbor estimates) exclude pairs of points closer in time than one autocorrelation time (BF_TheilerWindow, {'ac',1}), replacing a mix of fixed sample counts and fractions of the length. Feature names record the window (_ac1).

• Faster. The slowest operations were profiled and many made several times faster with outputs unchanged, e.g. subspace (state-space) identification about 18 times faster, sinusoid fits about 20 times, Lempel–Ziv complexity about 10 times. The GARCH fits are 3–4 times faster with an analytic Hessian, which changes their values only for fits stopped at a parameter bound. The surrogate test no longer evaluates the costly first minimum of the automutual information (fmmikraskov, still callable). On a 1000-series synthetic corpus the full feature set takes about 31 s per series on one core.

• New operations.

  • v3.0.0-alpha added NL_ZeroOneTest, NL_LyapSpec, NL_RQA, NL_PersistentHomology, SP_Specparam, SP_Bicoherence, SC_MFDFA, SY_SlowFeatureAnalysis and more.
  • New since the alpha:
    • SP_Rhythmicity: phase persistence, with an analytic spectral null that separates sustained oscillations from bursts.
    • PH_KramersMoyal, CO_MatrixProfile, EN_DispEn, EN_FuzzyEn, EN_BubbleEn, EN_RandomizeOrdinal, DN_TailIndex, CO_Quantilogram, SP_EnvelopeStats and ST_PeakIntervals.
    • SD_SurrogateTest gains amikraskovmi: the Kraskov automutual-information surrogate test at the 'mi' delay as well as at lag 1, which detects nonlinear structure that builds up over several steps (e.g. in oversampled flows).

• Redesigned NL_EmbedCluster. Its outputs are now sep_pp, sep_mahal, nongauss and valley. It is deterministic and takes about 0.03–0.05 s per call, down from about 2 s.

• Clearer names.

  • SP_Summaries wmax_* → wq_* (spectral quantiles), with centroid → wq_50.
  • NL_BoxCorrDim → NL_BoxCountEntropyRate.
  • HT_HypothesisTest is split into HT_MarginalTests and HT_IndependenceTests.
  • The time-domain fits of DN_SimpleFit move to SP_SinusoidFit, and DN_RemovePoints is split, with part moving to CO_RemovePoints.
  • SB_BinaryStretch → SB_BinaryGapHomogeneity.

• Better-conditioned embedding dimensions and dimension estimates.

  • 'fnn' (false nearest neighbors) chooses the embedding dimension at a 0.05 threshold everywhere, rules a dimension out only on a fraction resting on at least 60 points, and, when it sets the dimension for another algorithm, is capped at floor(2 log10 N) (Eckmann & Ruelle 1992). Registered calls write it as {'fnn',0.05} (fnn005 in feature names), and NL_FNN's firstunder* outputs are now dim*.
  • NL_TakensEstimator sets its length scale by a fraction of pairs (c001 in names) and uses the unbiased estimator; it is NaN with too few pairs or distinct distances, or a dimension above 2m.
  • NL_ReturnTime's ratios statrtys/statrtym are replaced by bounded shares in [0, 1], fracsdrt/fracmeanrt.

• Removed: SC_FastDFA. The DFA in SC_FluctAnal covers it.

• Docs, workflow and tests. Every registered operation has a standardized docstring. TS_Compute('series') parallelizes over time series (much faster for many short series), with optional autosave. TS_Combine merges many files in one pass. OutputToCSV writes values with 15 significant digits (previously 5 for the feature matrix and 6 for time-series data). New robustness test suites were added.

Downloads and installation

• Recommended: git clone --recurse-submodules https://github.com/benfulcher/hctsa.git, then run install.m, which also compiles TISEAN and the mex files. If you clone without --recurse-submodules, startup.m fetches the catch22 submodule the first time it runs in an interactive MATLAB session.
• GitHub's automatic "Source code" archives do not include the catch22 submodule. To download rather than clone, use hctsa-v3.0.0.tar.gz, attached below, which includes it.
• catch22 is optional. No operation in the hctsa feature set needs it, but it is required for the 'catch22' feature set (TS_Init(...,'catch22')).

Moving to Python: pyhctsa

pyhctsa 2.0.0 implements this feature set (6392 features, 927 master operations). On a 1000-series synthetic corpus, 6383 of the 6392 features agree with hctsa v3.0.0 at Pearson r ≥ 0.99.

Full changes: v2.0.0...v3.0.0 and v3.0.0-alpha...v3.0.0.

If you use hctsa, please cite Fulcher & Jones, Cell Systems (2017) and Fulcher, Little & Jones, J. R. Soc. Interface (2013).