Skip to content

data: import GSMArena cigarplug batch — 2,498 SoC-linked phones + tablets/watches/pdas#98

Merged
Seungpyo1007 merged 8 commits into
mainfrom
data/import-staging
Jul 1, 2026
Merged

data: import GSMArena cigarplug batch — 2,498 SoC-linked phones + tablets/watches/pdas#98
Seungpyo1007 merged 8 commits into
mainfrom
data/import-staging

Conversation

@Seungpyo1007

Copy link
Copy Markdown
Member

What & why

Second GSMArena batch from the cigarplug/scrape-gsma dataset (10,679-row GSMArena scrape). PR #97 imported the tablet/watch/pda subset; this brings in 2,498 new records — mostly smartphones — with each phone linked to a real SoC in the catalog.

Category Added
smartphone 2,379
tablet 101
watch 9
pda 9
total 2,498

All records land verified: false. Smartphones satisfy the stricter phone schema: integer ram_gb, ranges (battery 500–12000 mAh, weight 50–500 g), and a soc FK resolved to an existing SoC via a precise die-number + marketing-name matcher (unmatched chipsets are skipped, not guessed — avoids the fuzzy-match trap).

Commits (each Refs #1): 5 smartphone brand-group commits (Samsung/Huawei/Honor · LG/Xiaomi/Oppo/Vivo · Motorola/HTC/ZTE/Sony · Asus/Lenovo/Nokia/Meizu/Realme · remaining), tablet, watch+pda, and a surgical dump refresh.

Closes #1

Source

GSMArena via cigarplug/scrape-gsma (https://github.com/cigarplug/scrape-gsma), *-scrapegsma-<row> slugs. raw_source preserves the original chipset/CPU/GPU/memory strings; variant.source_row cites the CSV row.

Dump

Regenerated via TechEngine app.dump at pinned submodule c0928f9 (scoring v2.0.0). Only the 2,498 new records' detail+score files, the four category indexes and the manifest counts are committed (smartphones 90118→92497, tablets 3354→3455, watches 424→433, pdas 131→140) — no churn on unchanged records.

Checklist

  • python -m app.validate — ✅ passed (full dataset)
  • python TechEngine/integrity_check.py data --strict — ✅ no hard anomalies
  • git diff --check — ✅ no whitespace errors
  • Correct data/<category>/<brand>/<year>/<base>/<slug>.json paths, kebab-case unique slugs
  • source_urls present on every record

GSMArena device dump (cigarplug/scrape-gsma) for Samsung Galaxy,
Huawei and Honor lines, with SoC linked to the known SoC catalog.
Bulk import, verified:false.

Refs #1
GSMArena device dump (cigarplug/scrape-gsma) for LG, Xiaomi/Redmi/Poco,
Oppo and Vivo lines. SoC-linked, verified:false.

Refs #1
GSMArena device dump (cigarplug/scrape-gsma) for Motorola, HTC, ZTE
and Sony Xperia lines. SoC-linked, verified:false.

Refs #1
…, Realme

GSMArena device dump (cigarplug/scrape-gsma) for Asus, Lenovo, Nokia,
Meizu and Realme lines. SoC-linked, verified:false.

Refs #1
GSMArena device dump (cigarplug/scrape-gsma) for the remaining brands
(Infinix, BLU, Alcatel, Gionee, Allview, Wiko, Tecno, Microsoft,
Micromax, Panasonic, BlackBerry, Archos, BQ and others). SoC-linked,
verified:false.

Refs #1
Additional GSMArena tablet records from the cigarplug/scrape-gsma
dump not covered by the earlier batch. verified:false.

Refs #1
…g dump)

Remaining GSMArena smartwatch and PDA records from the
cigarplug/scrape-gsma dump. verified:false.

Refs #1
Add the 2,498 new GSMArena (cigarplug/scrape-gsma) records to the
static v1 dump — smartphone/tablet/watch/pda detail + smartphone
score entries — and bump the category index counts and manifest.
Regenerated via TechEngine app.dump at pinned submodule c0928f9
(scoring v2.0.0); only the new records and affected indexes are
committed.

Refs #1
@Seungpyo1007

Copy link
Copy Markdown
Member Author

🤖 Review — PASS

Check Command Result
Schema/self-check python -m app.validate ✅ passed (full dataset)
Integrity gate python TechEngine/integrity_check.py data --strict ✅ no hard anomalies
Whitespace git diff --check origin/main..HEAD ✅ none
Engine dump app.dump @ c0928f9 ✅ 104,825 records, no serializer errors

Data-change summary

  • Added: 2,498 — smartphone 2,379, tablet 101, watch 9, pda 9
  • Modified/Deleted: 0
  • All verified: false, each with source_urls.
  • Smartphones carry an integer ram_gb and a soc FK resolved to the existing SoC catalog (die-number + marketing-name exact match; unmatched chipsets skipped rather than guessed).

Example records

  • data/smartphone/google/2019/pixel-4-xl/google-pixel-4-xl-scrapegsma-5.json (soc snapdragon-855)
  • data/smartphone/samsung/2015/galaxy-a3-2016/…-scrapegsma-7959.json
  • data/tablet/acer/2011/iconia-smart/acer-iconia-smart-scrapegsma-1771.json

Heuristic warnings

  • A handful of voice-calling phablets (e.g. HP Slate voicetab) land under smartphone — genuinely ambiguous GSMArena devices; acceptable for a verified:false import.

Site build

  • No site/src changes. site/public/v1 dump regenerated (scoring v2.0.0); only new records + affected indexes/manifest committed.

@Seungpyo1007

Copy link
Copy Markdown
Member Author

📊 Dataset stats (post-merge projection)

Metric Count
Total records 104,825
verified: true 10,028
verified: false 94,791
missing verified 6
Verified ratio 9.57%

Warning

Verified ratio is 9.57% (down slightly from 9.80% as 2,498 new verified: false records land). Expected for the bulk-import campaign — promotion to verified happens later via the TechEngine verify-network reality check. The 6 records missing a verified field pre-date this PR.

Campaign progress toward +50k: cigarplug/scrape-gsma is now largely exhausted (~2,871 total imported across PR #97 + #98). Remaining headroom needs additional datasets (Kaggle phone dumps, PhoneDB, AllPhones SKU variants) — tracked in #1.

@Seungpyo1007 Seungpyo1007 merged commit 887d12a into main Jul 1, 2026
5 checks passed
@github-project-automation github-project-automation Bot moved this from In Progress to Done in TechAPI-Project Jul 1, 2026
@TechEngineBot

Copy link
Copy Markdown
Member

🔎 Data verification — Tier 0 (offline existence/trust)

Scored by app.verify; posted by TechEngineBot. Informational only — the structural gate (app.validate) is separate and authoritative for merge.

Changed records in this PR

No records scored.

Full-dataset baseline

104825 record(s) scored.

%%{init: {"theme":"base","themeVariables":{"pie1":"#3fb950","pie2":"#d29922","pie3":"#f85149","pieStrokeWidth":"0px","pieOpacity":"1"}}}%%
pie showData
    title Verification bands — all records
    "Green" : 10029
    "Yellow" : 92332
    "Red" : 2464
Loading
Category Distribution Total 🟢 🟡 🔴 🟢 %
brand 🟩🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨 189 10 179 0 5.3%
soc 🟩🟨🟨🟨🟨🟥🟥🟥🟥🟥🟥🟥 2104 123 680 1301 5.8%
smartphone 🟩🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟥 92497 8453 82925 1119 9.1%
tablet 🟩🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟥 3455 174 3253 28 5.0%
watch 🟩🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟥 433 11 412 10 2.5%
pda 🟩🟩🟨🟨🟨🟨🟨🟨🟨🟨🟨🟥 140 27 107 6 19.3%
gpu 🟩🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨 2030 245 1785 0 12.1%
cpu 🟩🟩🟩🟨🟨🟨🟨🟨🟨🟨🟨🟨 3977 986 2991 0 24.8%
All 🟩🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟥 104825 10029 92332 2464 9.6%

green = authoritative source + complete + consistent · yellow = plausible, needs confirmation · red = sparse/weak source or a hard contradiction. Promotion to verified runs in the scheduled verify-network workflow.

@TechEngineBot

Copy link
Copy Markdown
Member

TechEngine change review: PASS

Check Result
python -m app.validate PASS
python integrity_check.py TechAPI/data --strict PASS

Changed data

Category Added Modified Deleted Added verified Added unverified Added Kaggle-sourced
brand 0 0 0 0 0 0
soc 0 0 0 0 0 0
smartphone 0 0 0 0 0 0
tablet 0 0 0 0 0 0
watch 0 0 0 0 0 0
pda 0 0 0 0 0 0
gpu 0 0 0 0 0 0
cpu 0 0 0 0 0 0

Changed record examples

  • No data file changes detected.

Heuristic review

  • Heuristic warnings: none found.

@TechEngineBot

Copy link
Copy Markdown
Member

TechEngine validation stats: PASS

Data summary

Category Total Verified Unverified Missing verified Tracked Verified % of tracked
brand 189 10 179 0 189 5.3%
soc 2104 123 1981 0 2104 5.8%
smartphone 92497 8453 84044 0 92497 9.1%
tablet 3455 174 3281 0 3455 5.0%
watch 433 11 422 0 433 2.5%
pda 140 27 113 0 140 19.3%
gpu 2030 244 1786 0 2030 12.0%
cpu 3977 986 2991 0 3977 24.8%
all 104825 10028 94797 0 104825 9.6%

Warning

Tracked verified coverage is below 50% for watch 2.5% (11/433), tablet 5.0% (174/3455), brand 5.3% (10/189), soc 5.8% (123/2104), smartphone 9.1% (8453/92497), all 9.6% (10028/104825), gpu 12.0% (244/2030), pda 19.3% (27/140), and 1 more.
Tracked coverage excludes records missing the verified field; see the Missing verified column for those records.
This does not fail validation. Keep imported records verified: false until manual audit, but treat this as follow-up verification work before relying on the affected categories as curated data.

Validation notes

  • Full advisory outlier listings are suppressed on successful runs because they are dataset-wide and mostly stable between PRs.
  • Failure runs still include a detailed log excerpt for debugging.

Key output:

## app.validate
## integrity_check.py --strict
loaded CPU=3977 GPU=2030
✅ integrity gate: no hard anomalies.
Integrity section Flagged lines
structural 0
CPU name/tier consistency (desktop mainstream only) 0
CPU single>multi (cinebench/geekbench — should be multi>=single) 0
CPU era-vs-score outliers 8
CPU cross-source ratio outliers (possible wrong-variant) 152
GPU cross-source ratio outliers + sanity 18

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

data Dataset changes enhancement New feature or request

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

Massive dataset rebuild: CPU + brand + GPU + smartphone + SoC (1989-2026)

2 participants