Skip to content

data(cpu): fill process node, L3 cache and boost clock for 581 CPUs - #132

Merged
Seungpyo1007 merged 2 commits into
mainfrom
data/cpu-wikipedia-specs
Jul 29, 2026
Merged

data(cpu): fill process node, L3 cache and boost clock for 581 CPUs#132
Seungpyo1007 merged 2 commits into
mainfrom
data/cpu-wikipedia-specs

Conversation

@Seungpyo1007

@Seungpyo1007 Seungpyo1007 commented Jul 29, 2026

Copy link
Copy Markdown
Member

What

581 CPU records gain 982 previously-null fieldsprocess_node 400, l3_cache_mb 580, boost_clock_ghz 2 — from the Wikipedia CPU list pages (Intel Core desktop/mobile, AMD lists).

Coverage moves: process_node 24% → 34%, l3_cache_mb 19% → 34%.

Why this source

3,001 of 3,977 CPUs are bulk stubs: schema-complete, values null, and their architecture holds a brand family ("Xeon", "Core") rather than a microarchitecture — so the node is not derivable inside the dataset (Xeon alone spans 250 nm to 5 nm). An external, citable source was required. The Wikipedia list pages carry the codename + lithography in the section heading and L3/clocks per row, and are not bot-blocked.

How correctness was enforced

These tables do not share a column layout (some carry an extra iGPU / GPU-clock column), so a positional read silently shifts values. An early pass produced L3 = 95 MB (actually the TDP column), TDP = 2011 (the release year), and L3 = 4600 (the "HD 4600" iGPU). Guards:

  1. Columns are located by header name, with rowspan/colspan expansion — never by index.
  2. Confirm before write: a row is used only when the data we already hold agrees — base clock within 0.05 GHz and TDP within 1 W. Base clock agreed on 233/234 correct matches, making it a reliable SKU-identity check; TDP disagreement flags a shifted table. 456 unmatched / 7 disagreeing rows were dropped rather than guessed.
  3. Plausibility bounds (L3 0.5–64 MB) as a second net.
  4. MSRP deliberately not imported — Wikipedia lists launch price, which is not what msrp_usd holds (120 of 129 disagreed; e.g. our i7-860 = $44). Different semantics, not an error to "fix".

Spot-checks against known specs pass: i7-2600K → 32 nm / 8 MB, i3-2100 → 32 nm / 3 MB, i7-4790K → 22 nm / 8 MB / 88 W.

Only empty fields are written; nothing existing is overwritten, verified is untouched, and the sourcing Wikipedia page is appended to source_urls.

python -m app.validateData validation passed.

Closes #1

581 CPU records gain 982 previously-null fields (process_node 400,
l3_cache_mb 580, boost_clock_ghz 2), sourced from the Wikipedia
"List of Intel Core desktop/mobile processors" and AMD list pages.

Only empty fields are written; nothing existing is overwritten and
`verified` is untouched. Each parsed row was accepted only when the data
already held for that SKU agreed with it -- base clock within 0.05 GHz and
TDP within 1 W -- because these tables do not share a column layout and a
positional read silently shifts values (an early pass produced "L3 = 95 MB"
from a TDP column and "TDP = 2011" from a release-year column). Rows that
disagreed were dropped rather than guessed. MSRP was deliberately not
imported: Wikipedia lists launch price, which is not what msrp_usd holds.

Refs #1
@TechEngineBot

Copy link
Copy Markdown
Member

🔎 Data verification — Tier 0 (offline existence/trust)

Scored by app.verify; posted by TechEngineBot. Informational only — the structural gate (app.validate) is separate and authoritative for merge.

Changed records in this PR

581 record(s) scored.

%%{init: {"theme":"base","themeVariables":{"pie1":"#3fb950","pie2":"#d29922","pie3":"#f85149","pieStrokeWidth":"0px","pieOpacity":"1"}}}%%
pie showData
    title Verification bands — all records
    "Green" : 581
    "Yellow" : 0
    "Red" : 0
Loading
Category Distribution Total 🟢 🟡 🔴 🟢 %
cpu 🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩 581 581 0 0 100.0%
All 🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩 581 581 0 0 100.0%

Full-dataset baseline

105742 record(s) scored.

%%{init: {"theme":"base","themeVariables":{"pie1":"#3fb950","pie2":"#d29922","pie3":"#f85149","pieStrokeWidth":"0px","pieOpacity":"1"}}}%%
pie showData
    title Verification bands — all records
    "Green" : 13773
    "Yellow" : 90057
    "Red" : 1912
Loading
Category Distribution Total 🟢 🟡 🔴 🟢 %
brand 🟩🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨 207 24 183 0 11.6%
soc 🟩🟨🟨🟨🟨🟥🟥🟥🟥🟥🟥🟥 2104 123 680 1301 5.8%
smartphone 🟩🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟥 93396 11602 81206 588 12.4%
tablet 🟩🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟥 3455 174 3266 15 5.0%
watch 🟩🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟥 433 11 415 7 2.5%
pda 🟩🟩🟨🟨🟨🟨🟨🟨🟨🟨🟨🟥 140 27 112 1 19.3%
gpu 🟩🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨 2030 245 1785 0 12.1%
cpu 🟩🟩🟩🟩🟩🟨🟨🟨🟨🟨🟨🟨 3977 1567 2410 0 39.4%
All 🟩🟩🟨🟨🟨🟨🟨🟨🟨🟨🟨🟥 105742 13773 90057 1912 13.0%

green = authoritative source + complete + consistent · yellow = plausible, needs confirmation · red = sparse/weak source or a hard contradiction. Promotion to verified runs in the scheduled verify-network workflow.

Regenerates site/public/v1/cpus for the 581 records changed in the previous
commit, using the engine at the pinned submodule commit so scoring output is
unchanged. Only pages whose content actually differs are committed -- app.dump
rewrites created_at/updated_at on every record, and those 7,374 timestamp-only
pages are left alone.

Refs #1
@Seungpyo1007
Seungpyo1007 merged commit 0ef30d4 into main Jul 29, 2026
5 checks passed
@github-project-automation github-project-automation Bot moved this from In Progress to Done in TechAPI-Project Jul 29, 2026
@Seungpyo1007
Seungpyo1007 deleted the data/cpu-wikipedia-specs branch July 29, 2026 05:03
@TechEngineBot

Copy link
Copy Markdown
Member

TechEngine change review: PASS

Check Result
python -m app.validate PASS
python integrity_check.py TechAPI/data --strict PASS

Changed data

Category Added Modified Deleted Added verified Added unverified Added Kaggle-sourced
brand 0 0 0 0 0 0
soc 0 0 0 0 0 0
smartphone 0 0 0 0 0 0
tablet 0 0 0 0 0 0
watch 0 0 0 0 0 0
pda 0 0 0 0 0 0
gpu 0 0 0 0 0 0
cpu 0 0 0 0 0 0

Changed record examples

  • No data file changes detected.

Heuristic review

  • Heuristic warnings: none found.

@TechEngineBot

Copy link
Copy Markdown
Member

TechEngine validation stats: PASS

Data summary

Category Total Verified Unverified Missing verified Tracked Verified % of tracked
brand 207 10 197 0 207 4.8%
soc 2104 123 1981 0 2104 5.8%
smartphone 93396 8453 84943 0 93396 9.1%
tablet 3455 174 3281 0 3455 5.0%
watch 433 11 422 0 433 2.5%
pda 140 27 113 0 140 19.3%
gpu 2030 244 1786 0 2030 12.0%
cpu 3977 986 2991 0 3977 24.8%
all 105742 10028 95714 0 105742 9.5%

Warning

Tracked verified coverage is below 50% for watch 2.5% (11/433), brand 4.8% (10/207), tablet 5.0% (174/3455), soc 5.8% (123/2104), smartphone 9.1% (8453/93396), all 9.5% (10028/105742), gpu 12.0% (244/2030), pda 19.3% (27/140), and 1 more.
Tracked coverage excludes records missing the verified field; see the Missing verified column for those records.
This does not fail validation. Keep imported records verified: false until manual audit, but treat this as follow-up verification work before relying on the affected categories as curated data.

Validation notes

  • Full advisory outlier listings are suppressed on successful runs because they are dataset-wide and mostly stable between PRs.
  • Failure runs still include a detailed log excerpt for debugging.

Key output:

## app.validate
## integrity_check.py --strict
loaded CPU=3977 GPU=2030
✅ integrity gate: no hard anomalies.
Integrity section Flagged lines
structural 0
CPU name/tier consistency (desktop mainstream only) 0
CPU single>multi (cinebench/geekbench — should be multi>=single) 0
CPU era-vs-score outliers 8
CPU cross-source ratio outliers (possible wrong-variant) 152
GPU cross-source ratio outliers + sanity 18

Seungpyo1007 added a commit that referenced this pull request Jul 30, 2026
…kipped

Recovers records that the merged batches (#132-#135) left untouched because of
three separate mismatches in how the pages name things, not because the data
was unavailable:

- model codes are not always digit-first (Core 2 uses E6600, Atom uses N270),
  and the table parser required a leading digit;
- the Atom page is built entirely from {{cpulist}} templates and had never been
  run through that parser;
- the Opteron page lists a bare model-number column with no branding column, so
  the family name now comes from the page itself.

Same confirm-before-write rule as before (base clock within 0.05 GHz, TDP within
1 W, L3 must be a real capacity and must not equal the row's TDP): 1,610 rows
parsed, 1,053 matched a record, 885 passed, 210 still had a gap to fill.

Verified against known parts: Clarkdale i5-6xx/i3-5xx 32 nm, Haswell i7-4770R
22 nm, Broadwell i7-5775R 14 nm, Atom 330/D410 45 nm, Atom D2500 32 nm,
Opteron Shanghai 6 MB L3. Older families whose rows failed the cross-check
(Core 2 E-series, Atom N270, early Opteron) were left empty rather than guessed.

Refs #1
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

data Dataset changes enhancement New feature or request

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

Massive dataset rebuild: CPU + brand + GPU + smartphone + SoC (1989-2026)

2 participants