Editing Cupy installation instructions - #47
Open
dwysocki wants to merge 1 commit into
Open
Conversation
Most users will not want to compile Cupy from source. I have expanded the Cupy installation guide to explain how to get the version appropriate for your system.
leofang
reviewed
Jan 6, 2021
| The code uses cupy to access GPUs. If you don't have one, the code will still work. | ||
| If you do need one, make sure to install cupy **on a machine that supports GPUs ** | ||
|
|
||
| You probably do not want to pip install `cupy` directly, but rather a pre-compiled version for your system's CUDA version. To find your system's version (assuming it's installed) run `nvidia-smi`, and look for "CUDA Version" in the top-right. For version X.Y, the appropriate cupy package will be called `cupy-cudaXY` (e.g., if you have CUDA 11.0, install `cupy-cuda110`) |
Collaborator
There was a problem hiding this comment.
This is strictly speaking not accurate. CUDA Version here refers to the maximal CUDA Toolkit version supported by the driver version reported on its left, but what you really need is the runtime library version installed locally, which can be decoupled/fall behind this version (say you have CUDA 10.2 installed, but you upgrade the driver to 11.1). Perhaps it's better to use nvcc --version if there is a local installation available to check.
FYI in some environments where there is no CUDA Toolkit installed locally, but has the driver installed, you can install CuPy from Conda-Forge:
conda install -c conda-forge cupy cudatoolkit=X.Y
X.Y is again up to the driver version. The cudatoolkit contains all CUDA runtime libraries (but no nvcc or the compiler toolchain).
oshaughn
pushed a commit
that referenced
this pull request
Aug 7, 2026
…ptation for replicas Both PR #47 findings. [P1] Full-support detection could discard the only real backstop. The check defaulted un-annotated samplers to full-support and counted a nominally broad member even after it had been RANGE-RESTRICTED. Reproduced: [unrestricted AV, restricted GMM] reports _full_support_members == [0], yet the restricted GMM set _has_broad, so member 0 was warm-started and contracted to V=0.095 -- nothing covering the prior box, which is the silent low bias this mechanism exists to prevent. Now: has_unbounded_support defaults FALSE and must be declared; only members that are BOTH declared broad AND still full-range (per _full_support_members) count. mcsamplerEnsemble declares it as a property keyed on gmm_defensive_frac > 0 -- the uniform defensive component is the actual guarantee, since Gaussian tails underflow to exactly zero far from the mode, so gmm_defensive_frac=0 correctly reports False. AV declares False. [P1] Replicas retained portfolio-level adaptation. clear_warm_state() rebuilds the members but not the portfolio's own learned state: portfolio_weights, portfolio_quality and its observation counts, portfolio_probe_ptr, portfolio_draw_iteration, breakpoint progression and per-member n_ess histories. Replicas therefore scheduled themselves from what earlier replicas learned, so the between-replica scatter -- the entire quantity being measured -- still understated the error. Adds reset_adaptation(), which does clear_warm_state() plus a restore of every field to its POST-SETUP value (snapshotted in setup, so an explicit initial weighting or breakpoint schedule is preserved rather than replaced by a hard-coded guess). The replica loop calls it. Four tests, each verified to fail when its defect is reintroduced, including the requested assertion that all portfolio-level state is identical across replicas. Noted while testing, NOT fixed here: mcsamplerEnsemble.bootstrap_from_samples rebuilds its integrator without the caller's gmm_defensive_frac, so a warm start silently restores the 0.05 default. Same class as the gmm_dict config loss fixed in #45; filed separately rather than widened into this PR. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
oshaughn
pushed a commit
that referenced
this pull request
Aug 7, 2026
…ore claiming coverage PR #47 follow-up. has_unbounded_support read gmm_defensive_frac > 0 as a guarantee, but add_defensive_component() was only called by fit_gmm_adaptive. The fixed-component fit paths (mcsamplerEnsemble and MonteCarloEnsemble) called GMM.gmm(...).fit(...) directly, and gmm_adaptive defaults to None -- so the DEFAULT configuration requested a defensive component and never installed one, and the portfolio then contracted its AV member believing it was covered. Worse than a missing component: gmm.score() FLOORS its return at 1e-300, so the member always LOOKS like it has density everywhere. That floor is a guard against log(0), not coverage -- a sample landing there carries weight L*p/q ~ 1e300 and would wreck the estimate rather than support it. Measured, a fixed-component fit to a tight cloud returns exactly the floor at the far corner of the prior box for every d >= 4: d=2 2.9e-273 -> 1.3e-04 (with the defensive component) d=6 1.0e-300 -> 7.6e-10 This also corrects a claim I made in PR #47: "the GMM member's unbounded support is what has protected production". It is not unbounded support, it is a numerical floor. The measured |lnZ bias| <= 0.05 with a displaced seed means the far region was never sampled in those runs, not that it was covered. Fixes: add_defensive_component sets model.defensive_frac as a verifiable marker; both fixed-component fit paths now install it; has_unbounded_support inspects the installed models and reports False if ANY trained group lacks it, or if gmm_defensive_frac <= 0. While untrained it trusts the config -- sound only because every fit path now installs the component, which test_every_fit_path_installs_the_defensive_component pins. Note this changes the DEFAULT GMM proposal: fixed-component fits now carry a 5% defensive component (the documented gmm_defensive_frac default, previously inert). That is the intended behaviour per its own docstring and it bounds the importance weights, but it is a shared-sampler change -- gate results reported separately. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
oshaughn
pushed a commit
that referenced
this pull request
Aug 7, 2026
…led reruns, and is wired in Three P1s from review of #49. [P1] The confirm step recognised only PASS -> non-PASS. The comparator also blocks REGRESSION(metrics) -- JS, pull, width, correlation, evidence bias or n_eff worsening beyond tolerance -- and those rows produced "no blocking regressions to confirm" and exit 0. Our own gate v11 had exactly such a row (GMM d8_n1_s303, n_eff 448->210), which would have been waved through. Fixed structurally rather than by adding a second branch: compare_shape_results now exposes classify() / is_blocking() / blocking_keys() as the SINGLE definition of a regression, and confirm_regressions imports them. Two copies of that logic will always drift apart; there is now one. Verified the refactor reproduces the v11 verdict exactly (2 blocking). The confirm step now sees both rows where it previously saw one. [P1] Failed reruns were silently skipped, so worse == same == 0 read as "not confirmed". Now: a candidate that produces no record where the base did counts AGAINST the candidate (crashing is worse than passing, not missing evidence); a verdict requires --min-valid usable pairs, defaulting to all seeds; and too few valid pairs is INCONCLUSIVE with a nonzero exit, never a silent clear. [P1] Confirmation was documented but never invoked. compare_shape_results gains --confirm-base-checkout / --confirm-cand-checkout / --confirm-repeats and returns the confirmed verdict as its exit code, so the comparison workflow enforces it. Without those flags it still exits 1 on a blocking row, and now says explicitly that the row was NOT confirmed rather than implying it was. Adds test_confirm_regressions.py (5 checks, all on the dangerous direction -- the ways a confirmation can wrongly CLEAR a real regression). CI on this PR is red at "Set up job" on 4 jobs; the same runner-provisioning failure hits #47, which shares no files with this change, and no failing job reaches a step that executes repository code. Infrastructural, not from here. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
oshaughn
pushed a commit
that referenced
this pull request
Aug 7, 2026
…ust at setup Two PR #47 findings: the defensive component was installed correctly and then lost again during normal operation. [P1] Warm bootstrap disabled the portfolio's opt-in. bootstrap_from_samples() calls setup() to rebuild the integrator as one full-dim group, and did so bare -- resetting gmm_defensive_all_paths to False and refitting the warm GMM with no defensive component, AFTER the portfolio had already decided on the strength of that flag that it was safe to contract its AV member. It now carries the defensive config forward from the existing integrator. [P1] Updates absorbed the defensive component. _merge() blends component i with the fitted component order[i] for every i in range(self.k) and never consults self.adapt, so the broad component -- marked adapt=False precisely so it would be left alone -- drifted toward the fitted cloud on every update while defensive_frac stayed set and has_unbounded_support kept reporting coverage. Measured, far-field density fell 3.09e-07 -> 2.04e-08 after a single update cycle. update() now detaches the defensive component, updates the real ones and reinstates it; density is constant at 3.09e-07 across 4 updates with the component's weight, mean and covariance unchanged. The tests previously only checked initial installation. The two added here exercise the warm-bootstrap path and repeated updates, and assert on the actual far-field density and on the defensive component's own parameters rather than on the marker attribute -- the marker is exactly what stayed true while the guarantee evaporated. Both verified to fail when their defect is reintroduced. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
oshaughn
pushed a commit
that referenced
this pull request
Aug 7, 2026
Conflict was structural, not semantic: both sides added a method immediately before bootstrap_from_samples, and the two bootstrap signatures differed. Resolution keeps BOTH bodies -- the merged diagnostic and reset_adaptation -- with the single signature that carries keep_backstop_cold. Verified after resolution: 22 portfolio tests and all 5 of the diagnostic's own tests pass.
oshaughn
pushed a commit
that referenced
this pull request
Aug 7, 2026
…plicas, and a lifecycle-durable coverage guarantee Warm-start coverage honesty + genuinely cold MC-error replicas (PR 161 blockers 1 and 2)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Most users will not want to compile Cupy from source. I have expanded the Cupy installation guide to explain how to get the version appropriate for your system.