NNS.boost: LPM.VaR probability threshold, two-panel diagnostics, repl… - #58
Merged
OVVO-Financial merged 6 commits intoJul 18, 2026
Conversation
…icated multivariate final stack Threshold semantics: the public [threshold] argument is now a probability supplied to LPM.VaR over the learner-trial objective distribution, never a literal objective-score cutoff. Distinct internals threshold.probability (0.80 max / 0.20 min defaults; 1 / 0 under extreme) and learner.threshold (the LPM.VaR objective cutoff) replace the old quantile()-derived value, and neither overwrites the other. Survivor selection and every epoch re-test use learner.threshold with the same directional rule. The status line now reads "Learner threshold probability = %.2f; objective cutoff = %.6f"; the misleading "Learner Accuracy Threshold" message is gone. Diagnostics: feature.importance = TRUE draws a two-panel plot on one device - the learner-trial objective histogram with its LPM.VaR cutoff line, then the horizontal feature-frequency bar plot - via small internal helpers (.nns_boost_plot_*) so invocation is unit-testable. Original par() settings are restored only after both panels; the panels complete before an early features.only return; the final NNS.stack call never plots. Learner trials fit genuine multivariate NNS.reg (dim.red.method = NULL) and, for cross-sectional data, draw a fresh validation holdout per trial; time-series trials keep the chronological terminal block. Exhaustive enumeration no longer implies anything about epochs: epochs re-test only the surviving combinations on fresh (chronological for ts) splits. Final estimate: keeper-feature counts convert to integer replication factors (freq / min(freq), rounded, floor 1) that replicate the original predictor columns for train and test; the replicated multivariate matrices feed one NNS.stack(method = 1, stack = FALSE, folds = 5, balance = FALSE, optimize.threshold = FALSE, ncores = 1) call and results come from $reg / $reg.pred.int. No synthetic scalar X* and no dim.red.method weighting. A new public folds argument (default 5) passes through to that call. Tests: threshold-helper unit tests (defaults, extreme, supplied probability, LPM.VaR equivalence, no fivenum/quantile), plot-helper tests (both panels invoked under mfrow = c(2, 1), cutoff line uses learner.threshold, neither drawn when feature.importance = FALSE, nonempty PDF), and the iris integration runs (probability message, distinct cutoff, epochs under exhaustive enumeration, 10 results, weights sum to 1, replicated multivariate final stack verified via trace). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AxBV5HmXHY5nCUvW9SwDvb
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 594f18f1c3
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| # Give every survivor approximately equal re-test exposure. Randomizing the | ||
| # balanced schedule avoids ordering effects without allowing exposure count | ||
| # alone to masquerade as feature importance. | ||
| epoch_survivor_id <- rep(seq_len(survivor_count), length.out = epochs) |
There was a problem hiding this comment.
When epochs is smaller than length(reduced.test.features) (for example when a caller lowers the threshold probability or requests only a few epochs), rep(seq_len(survivor_count), length.out = epochs) truncates the schedule to survivors 1:epochs before the later sample() only shuffles those IDs. This leaves every later threshold-passing subset untested, so feature frequencies and the final replicated stack can be driven by learner/exhaustive ordering rather than by all survivors; sample from all survivor IDs before limiting the epoch schedule.
Useful? React with 👍 / 👎.
… not clipped The horizontal bar plot clipped long predictor names (e.g. "Petal.Length" rendered as "l.Length") under the default left margin. The panel helper now sizes its own left margin from strwidth() of the labels, scoped with an on.exit restore so the surrounding two-panel mfrow layout is untouched. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AxBV5HmXHY5nCUvW9SwDvb
…ding A factor or character DV.train now yields the original class labels (a factor response keeps its level order and orderedness, a logical response returns logical) instead of the internal 1..K codes; numeric class responses continue to return their original numeric values. Prediction intervals are decoded identically, and the output gains $class.levels (mirroring NNS.stack) so the coding is always inspectable. The documented iris example accuracy check becomes a direct label comparison. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AxBV5HmXHY5nCUvW9SwDvb
…odes to labels Classification output returns to the historical numeric coding: integer class codes with a base category of 1 for factor/character responses (numeric class responses keep their original numeric values), for both $results and $pred.int. $class.levels remains in the output so the label is always recoverable as class.levels[results]; the documented example shows both the numeric accuracy check and the label recovery. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AxBV5HmXHY5nCUvW9SwDvb
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
…icated multivariate final stack
Threshold semantics: the public [threshold] argument is now a probability supplied to LPM.VaR over the learner-trial objective distribution, never a literal objective-score cutoff. Distinct internals threshold.probability (0.80 max / 0.20 min defaults; 1 / 0 under extreme) and learner.threshold (the LPM.VaR objective cutoff) replace the old quantile()-derived value, and neither overwrites the other. Survivor selection and every epoch re-test use learner.threshold with the same directional rule. The status line now reads "Learner threshold probability = %.2f; objective cutoff = %.6f"; the misleading "Learner Accuracy Threshold" message is gone.
Diagnostics: feature.importance = TRUE draws a two-panel plot on one device - the learner-trial objective histogram with its LPM.VaR cutoff line, then the horizontal feature-frequency bar plot - via small internal helpers (.nns_boost_plot_*) so invocation is unit-testable. Original par() settings are restored only after both panels; the panels complete before an early features.only return; the final NNS.stack call never plots.
Learner trials fit genuine multivariate NNS.reg (dim.red.method = NULL) and, for cross-sectional data, draw a fresh validation holdout per trial; time-series trials keep the chronological terminal block. Exhaustive enumeration no longer implies anything about epochs: epochs re-test only the surviving combinations on fresh (chronological for ts) splits.
Final estimate: keeper-feature counts convert to integer replication factors (freq / min(freq), rounded, floor 1) that replicate the original predictor columns for train and test; the replicated multivariate matrices feed one NNS.stack(method = 1, stack = FALSE, folds = 5, balance = FALSE, optimize.threshold = FALSE, ncores = 1) call and results come from $reg / $reg.pred.int. No synthetic scalar X* and no dim.red.method weighting. A new public folds argument (default 5) passes through to that call.
Tests: threshold-helper unit tests (defaults, extreme, supplied probability, LPM.VaR equivalence, no fivenum/quantile), plot-helper tests (both panels invoked under mfrow = c(2, 1), cutoff line uses learner.threshold, neither drawn when feature.importance = FALSE, nonempty PDF), and the iris integration runs (probability message, distinct cutoff, epochs under exhaustive enumeration, 10 results, weights sum to 1, replicated multivariate final stack verified via trace).
Claude-Session: https://claude.ai/code/session_01AxBV5HmXHY5nCUvW9SwDvb