sync(npx-panel): ior null on unknown denominator; stays byte-identical to mirror - #97
Merged
Conversation
…l to mirror Mirror PR #24. ior was 0.0 wherever the CRSP denominator is unknown, making 'institutions hold none' and 'no denominator' the same value on 42.85% of leg 2. Now null for unknown, 0.0 only for a measured zero. NOTE FOR THE GRID SIDE: merge_panel.sas needs the matching change. In mirror the coupled fix was to merge_panel.py's 'ior_crsp <= 1.2' filter, which drops nulls in polars and would have silently deleted that 42.85% at the merge. SAS WHERE clauses treat missing the same way — 'where ior_crsp <= 1.2' excludes missing values — so if the SAS merge carries that filter it needs 'or ior_crsp is missing' before this reaches a grid run. Kept byte-identical to mirror so leg 2 stays one implementation. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0193ue9M5okF31PoPetXXBmL
edwinhu
added a commit
that referenced
this pull request
Jul 27, 2026
…otal" (#98) The four pivotalness flags were `abs(forpct-50) <= coalesce(ior,0)`. The coalesce turns "we have no denominator for this firm" into "this firm's owners hold nothing" at the moment the flag is set, so a row with no measurable ownership was recorded as KNOWN NOT PIVOTAL — indistinguishable from one measured at 0.4% and genuinely not pivotal. It re-conflated downstream exactly what the upstream null exists to protect. #97 stopped leg 2 writing ior = 0.0 for an unknown denominator; this put the zero back one join later. 42.85% of leg-2 rows have no CRSP denominator and the ISS-side tso has its own gaps, so this is a large minority of the panel, not an edge. `inst_pivotal_net` four lines below already did it correctly, and its comment states the whole argument: "null where lending is unknown, so it cannot quietly fall back to the gross measure." Same reasoning, four columns earlier — the right remedy was already written in the same paragraph as the wrong one. Changes inst_pivotal / mf_pivotal / passive_pivotal / index_pivotal from {0,1} to {0,1,.}. Digest B moves on those four columns by design. Claude-Session: https://claude.ai/code/session_017AL2p8VC1ULxSibcCdEAsh Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Mirror PR #24.
iorwas0.0wherever the CRSP denominator is unknown, making "institutions hold none" and "no denominator" the same value on 42.85% of leg 2. Now null for unknown,0.0only for a measured zero.In mirror the coupled fix was to
merge_panel.py'sior_crsp <= 1.2filter — polars drops nulls on that comparison, so the change alone would have silently deleted 42.85% of the leg at the merge.SAS behaves the same way:
where ior_crsp <= 1.2excludes missing values. Ifmerge_panel.sascarries that filter it needsor ior_crsp is missingbefore this reaches a grid run, or the same silent deletion happens there.Kept byte-identical to mirror so leg 2 stays one implementation.