Skip to content

alphaXiv/knowledge-using-gap-codex

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

2 Commits
 
 
 
 

Repository files navigation

Knowing–Using Gap replication

A compact replication of the Knowing–Using Gap on Qwen2.5-1.5B: direct recall separates sharply from two-hop use; activation patching reveals latent signal but does not recover the answers.

Replication scope

Behavioral result

Atomic recall and two-hop accuracy over training

On 1,000 synthetic chains, atomic recall reached 100% at epoch 4 while two-hop accuracy stayed at 0% through epoch 31.

Training loss after memorization saturation

Claim-level result

Replication verdict by claim

Mechanistic patching summary

Patching moved answer-relevant signal—best source states clustered in early and late layers—but never made the answer rank first.

Four residual-post layer-swap heatmaps

Mean residual-post layer-swap heatmap

Red cells improve the correct token's log-probability; stars mark each case's best source→target pair. Exact recovery remained 0/4.

Experiments

Stage Scale Result
Toy end-to-end 8 chains 100% recall · 12.5% use
Full behavioral run 1,000 chains 100% recall · 0% use
Patching at saturation 4 scans + 1,000 fixed-pair tests fixed pairs: 0%
Oracle validation 16 scans superseded after semantic mismatch
Faithful residual-post oracle 12,544 patches +16.16 nats · 0/16 recovered
Layer-swap heatmaps 4 × 784 patches +16.72 nats · 0/4 recovered

Synthetic single-token labels replace the unreleased STaRK experiment data. Results therefore support the paper's central behavioral claim, not its cross-domain or exact-recovery numbers.

About

No description, website, or topics provided.

Resources

Stars

5 stars

Watchers

0 watching

Forks

Releases

No releases published

Packages

 
 
 

Contributors