Eight models on real flooring takeoff: 40 floor trials plus base and wall tile

[QuisutDeus1]

I ran eight models through the real OpenTakeoff MCP, using OpenRouter, against takeoffs I had already reviewed and approved. The models had to draw their answers with MCP tools; prose and self-reported totals were not scored.

Kimi K3 led floor-area overlap at 91.0%, with 4.6% average net-area error and a 5.5-minute median runtime. Its boundary F1 within one physical inch was only 14.9%. That gap matters: a convincing floor envelope still needs work at the wall faces.

This is a small, bounded workflow experiment using a custom harness. It is not a general model ranking or a certification that any output is ready for a bid.

Floor results

40 trials: the same five private sheets for each of eight models, one corrected-protocol trial per model/sheet. The reference contains 66 approved floor regions. Sheets are equally weighted in the averages. The original plan was ten sheets; I reduced it to the first five in the original order before any trial started on the dropped sheets.

Model	Input	Drawings / 5	Mean floor IoU	Boundary F1, 1 inch	Mean area error	Median time	Cost, 5 trials
Kimi K3	vision	5/5	91.0%	14.9%	4.6%	5.5 min	$5.930
GLM 5.3	text	5/5	73.4%	15.4%	33.6%	3.3 min	$2.794
DeepSeek V4.1 Flash	vision	4/5	73.4%	9.8%	22.9%	6.8 min	$0.757
DeepSeek V4 Pro 0813	text	5/5	62.7%	6.7%	58.3%	6.5 min	$0.881
GLM 5.3 Flash	vision	3/5	51.8%	8.3%	44.4%	6.5 min	$0.478
Qwen 3.8 Flash	vision	3/5	40.3%	6.3%	58.4%	12.8 min	$0.396
MiniMax M3	vision	1/5	3.0%	0.1%	96.9%	10.2 min	$1.174
Qwen 3.8 2.4T-A95B	text	0/5	0.0%	0.0%	100.0%	16.1 min	$3.042

Empty drawings count as zero overlap. Partial drawings at a limit are scored as saved. A completed status only means the model ended its run; it does not certify complete or correct scope. There were no provider-error or harness-error terminations in the final 48 trials. Qwen 3.8 2.4T-A95B produced no committed floor geometry before its time cutoffs; that is a result of this workflow and allowance, not proof that the model cannot do takeoff.

Per-sheet floor overlap and termination reasons

Model	F01	F02	F03	F04	F05	Terminations
Kimi K3	93.4%	94.9%	87.1%	89.8%	89.9%	3 completed, 1 turn limit, 1 sheet cost limit
GLM 5.3	60.1%	89.1%	93.3%	48.1%	76.6%	5 completed
DeepSeek V4.1 Flash	95.4%	94.4%	87.7%	0.0%	89.3%	4 completed, 1 time limit
DeepSeek V4 Pro 0813	44.4%	97.8%	61.4%	49.8%	60.2%	5 completed
GLM 5.3 Flash	91.5%	0.0%	0.0%	86.1%	81.2%	2 completed, 2 turn limit, 1 time limit
Qwen 3.8 Flash	0.0%	88.0%	85.5%	0.0%	28.1%	4 time limit, 1 turn limit
MiniMax M3	14.9%	0.0%	0.0%	0.0%	0.0%	4 turn limit, 1 time limit
Qwen 3.8 2.4T-A95B	0.0%	0.0%	0.0%	0.0%	0.0%	5 time limit

Base and wall tile

Eight additional trials on one private six-sheet source packet. Each model received the same settled estimating scope and height directions, but no answer quantities or reference traces. Base is measured as installed length; wall tile is traced wall length times the applicable wall height. Both scopes are drawn on the same plan sheet.

Model	Base run F1	Base quantity error	Wall run F1	Wall quantity error	Correct-height predicted wall area	Time	Cost	End reason
Kimi K3	0.0%	100.0%	0.0%	100.0%	0.0%	10.2 min	$1.587	sheet cost limit
Qwen 3.8 2.4T-A95B	0.0%	100.0%	0.0%	100.0%	0.0%	12.9 min	$0.758	time limit
Qwen 3.8 Flash	0.0%	100.0%	0.0%	100.0%	0.0%	13.1 min	$0.092	time limit
DeepSeek V4 Pro 0813	53.2%	18.9%	50.4%	5.4%	30.0%	12.0 min	$0.273	completed
DeepSeek V4.1 Flash	0.0%	100.0%	0.0%	100.0%	0.0%	4.1 min	$0.213	turn limit
GLM 5.3	64.9%	24.1%	79.6%	1.5%	18.5%	6.8 min	$1.165	completed
GLM 5.3 Flash	45.2%	45.0%	81.6%	6.8%	67.0%	9.0 min	$0.144	turn limit
MiniMax M3	0.0%	100.0%	0.0%	100.0%	0.0%	5.5 min	$0.111	turn limit

No model demonstrated a reliable complete base-and-wall takeoff in this bounded test. GLM 5.3 Flash had the strongest wall-run F1 at 81.6%, but its base quantity error was 45.0%. GLM 5.3's wall quantity was only 1.5% off, yet just 18.5% of its predicted wall area had both a matching run and the right height. A close total can hide compensating errors.

How the runs worked

• Published opentakeoff-mcp@0.9.96, a fresh isolated session for every trial, with clean source copies and known scales. The approved answers were hidden from the models. No Spline connection or original-job writes; source and reference hashes were checked afterward.

• The custom harness supplied the same SFG tracing rules and allowlisted MCP tool schemas. Models could inspect positioned text, vectors, schedules, and plan crops; commit/edit polygons, cuts, lines, or surfaces as appropriate; audit overlays; and export their takeoffs. One-click tracing was disabled. There was no shell, arbitrary file reader, answer access, or assisting vision model.

• Floor scope: net interior floor-envelope geometry, including closets/restrooms/circulation and floor under furnishings; exclude walls, columns, shafts, exterior areas and stair voids. Follow interior wall faces, handle door openings and returns, and avoid gaps/overlaps. Finish classification, pricing, waste, attic stock and repeated-unit multipliers were not scored.

• Base/wall scope: physically located installed runs, door gaps excluded, wall-specific heights, and the same already-settled estimating directions. No new decisions were applied to a live job.

• Same per-trial limits: 24 model turns, 180 tool calls, 32,768 output tokens per response, requested medium reasoning, temperature 0.2, and a $1.50 soft cost cutoff. No new model turn after 720 seconds. In-flight responses could finish past the time or cost cutoff. The HTTP timeout was a 360-second socket-inactivity timeout, not a hard wall-clock deadline; two transient-error retries were allowed. Reported times are actual end-to-end trial times, including provider waits, MCP work and exports, excluding time waiting in the local queue.

• Transport limits: up to 1,000 vector segments and 65,000 text characters per tool response; the six most recent image attachments were retained. Text-only models received positioned text/vectors, without images. Eight floor workers and four supplemental workers ran concurrently. Provider routing and latency were not held fixed.

What the metrics mean

• Floor IoU: area of the intersection divided by the area of the union of model and approved floor geometry. Holes and deductions are accounted for. Overlapping model shapes are unioned rather than counted twice.

• Area error: absolute difference in net floor area divided by approved net area. Each sheet's percentage error is averaged equally.

• Boundary F1: harmonic mean of predicted-perimeter precision and reference-perimeter recall within one physical inch. This checks edge placement more strictly than broad area overlap.

• Base/wall run F1: the analogous length-based match within one physical inch. Wall quantity also depends on height. Correct-height wall area is the share of predicted wall-tile area on an approved run with height within one inch of the reference height; it is not a completeness measure.

Limits, costs and reproducibility

These are previously reviewed private jobs, not a claim of an unseen dataset. There is one corrected-protocol trial per model/sheet, no repeat-run confidence interval, and only one supplemental base/wall case. Equal time/turn/cost allowances do not equalize model capability or provider latency. The results apply to this prompt, tool access, provider routing and run allowance.

The final floor trials total $15.451 in response-reported API cost; the base/wall trials total $4.342. An earlier pilot with a 6,144-token output cap stopped on truncated responses. It was excluded and the retained trials were rerun with the corrected 32,768-token/continuation protocol. The pilot's separately recorded cost was $5.362. These sums come from response usage records, not an invoice reconciliation.

All 40 saved floor scales matched the supplied scales. Scoring checks covered identity, missing drawings, shifted geometry with unchanged area, holes plus deductions, accepted line reproduction, and deliberately incorrect wall heights. Every nonempty final takeoff had a marked PDF that rendered successfully. Those checks validate the scoring/export path; they do not turn a model's drawing into an approved takeoff.

The source plans, approved geometry, job identities, raw tool transcripts and private takeoff exports are not being published. The anonymized scalar records below let anyone reproduce the tables and inspect sheet-to-sheet variation. The end-to-end geometry benchmark cannot be independently rerun from this public post without the private inputs.

Exact OpenRouter model IDs

• moonshotai/kimi-k3 (vision)

• qwen/qwen3.8-2.4t-a95b (text)

• qwen/qwen3.8-flash (vision)

• deepseek/deepseek-v4-pro-0813 (text)

• deepseek/deepseek-v4.1-flash (vision)

• z-ai/glm-5.3 (text)

• z-ai/glm-5.3-flash (vision)

• minimax/minimax-m3 (vision)

Anonymized result records (JSON)

{"date":"2026-10-03","mcp_version":"0.9.96","floor_trials":[{"model":"deepseek/deepseek-v4-pro-0813","sheet":"F01","iou":0.4441107687000775,"boundary_f1_1in":0,"area_error_pct":125.16905026352394,"seconds":239.39760850000312,"cost_usd":0.11775069504,"floor_shapes":21,"status":"completed"},{"model":"deepseek/deepseek-v4-pro-0813","sheet":"F02","iou":0.9778369716012822,"boundary_f1_1in":0.004341423746378519,"area_error_pct":0.7488471922992703,"seconds":391.5447200410126,"cost_usd":0.20819949216,"floor_shapes":13,"status":"completed"},{"model":"deepseek/deepseek-v4-pro-0813","sheet":"F03","iou":0.614028603104879,"boundary_f1_1in":0,"area_error_pct":62.85886275385697,"seconds":467.0306451659999,"cost_usd":0.21275726208,"floor_shapes":1,"status":"completed"},{"model":"deepseek/deepseek-v4-pro-0813","sheet":"F04","iou":0.4978115956366477,"boundary_f1_1in":0.0008183780034435324,"area_error_pct":67.82610624427123,"seconds":416.2228799170116,"cost_usd":0.14961733248,"floor_shapes":14,"status":"completed"},{"model":"deepseek/deepseek-v4-pro-0813","sheet":"F05","iou":0.6019950031159133,"boundary_f1_1in":0.33211489689887697,"area_error_pct":34.703741773773686,"seconds":262.41360912501113,"cost_usd":0.19283198880000002,"floor_shapes":29,"status":"completed"},{"model":"deepseek/deepseek-v4.1-flash","sheet":"F01","iou":0.9543319997771165,"boundary_f1_1in":0.004561732219209453,"area_error_pct":3.6729199542940334,"seconds":503.9789294580114,"cost_usd":0.1628152944,"floor_shapes":15,"status":"completed"},{"model":"deepseek/deepseek-v4.1-flash","sheet":"F02","iou":0.9436377671712237,"boundary_f1_1in":0.006636149831464228,"area_error_pct":2.888904758896947,"seconds":326.6217637080117,"cost_usd":0.27203014799999997,"floor_shapes":8,"status":"completed"},{"model":"deepseek/deepseek-v4.1-flash","sheet":"F03","iou":0.8771534140961924,"boundary_f1_1in":0.017465251414317835,"area_error_pct":5.15492542134953,"seconds":335.6815867080004,"cost_usd":0.22172043600000002,"floor_shapes":12,"status":"completed"},{"model":"deepseek/deepseek-v4.1-flash","sheet":"F04","iou":0.0,"boundary_f1_1in":0,"area_error_pct":100.0,"seconds":738.513347250002,"cost_usd":0.023636496,"floor_shapes":0,"status":"time_limit"},{"model":"deepseek/deepseek-v4.1-flash","sheet":"F05","iou":0.8933288688497737,"boundary_f1_1in":0.4597955818580338,"area_error_pct":2.7174052917309326,"seconds":407.22824154098635,"cost_usd":0.0765157368,"floor_shapes":26,"status":"completed"},{"model":"minimax/minimax-m3","sheet":"F01","iou":0.14901875808510015,"boundary_f1_1in":0.004374528216539147,"area_error_pct":84.61944308188212,"seconds":261.39004645799287,"cost_usd":0.22972781999999997,"floor_shapes":1,"status":"turn_limit"},{"model":"minimax/minimax-m3","sheet":"F02","iou":0.0,"boundary_f1_1in":0,"area_error_pct":100.0,"seconds":857.8875772499887,"cost_usd":0.32201561,"floor_shapes":0,"status":"turn_limit"},{"model":"minimax/minimax-m3","sheet":"F03","iou":0.0,"boundary_f1_1in":0,"area_error_pct":100.0,"seconds":498.624578500021,"cost_usd":0.23880474000000002,"floor_shapes":0,"status":"turn_limit"},{"model":"minimax/minimax-m3","sheet":"F04","iou":0.0,"boundary_f1_1in":0,"area_error_pct":100.0,"seconds":1077.6867192080244,"cost_usd":0.16284481,"floor_shapes":0,"status":"time_limit"},{"model":"minimax/minimax-m3","sheet":"F05","iou":0.0,"boundary_f1_1in":0,"area_error_pct":100.0,"seconds":612.8149634579895,"cost_usd":0.22017076,"floor_shapes":0,"status":"turn_limit"},{"model":"moonshotai/kimi-k3","sheet":"F01","iou":0.9343332014069585,"boundary_f1_1in":0.007593544440362428,"area_error_pct":4.780794869948527,"seconds":270.1091958330071,"cost_usd":0.8364228,"floor_shapes":7,"status":"completed"},{"model":"moonshotai/kimi-k3","sheet":"F02","iou":0.948668580510107,"boundary_f1_1in":0.007706646807812959,"area_error_pct":4.017922133257721,"seconds":329.35069229101646,"cost_usd":1.163175,"floor_shapes":7,"status":"completed"},{"model":"moonshotai/kimi-k3","sheet":"F03","iou":0.8707769733081789,"boundary_f1_1in":0.15878544847094617,"area_error_pct":10.799230526164463,"seconds":529.9409723750141,"cost_usd":1.1111536800000001,"floor_shapes":8,"status":"turn_limit"},{"model":"moonshotai/kimi-k3","sheet":"F04","iou":0.8980133398294864,"boundary_f1_1in":0.40075703560531717,"area_error_pct":0.7636543716801504,"seconds":233.26239891699515,"cost_usd":1.5182721000000001,"floor_shapes":10,"status":"sheet_cost_limit"},{"model":"moonshotai/kimi-k3","sheet":"F05","iou":0.8993407792562986,"boundary_f1_1in":0.16908233726111566,"area_error_pct":2.7090074788416016,"seconds":727.0297916249838,"cost_usd":1.3007583600000001,"floor_shapes":25,"status":"completed"},{"model":"qwen/qwen3.8-2.4t-a95b","sheet":"F01","iou":0.0,"boundary_f1_1in":0,"area_error_pct":100.0,"seconds":837.1146793329972,"cost_usd":0.594436,"floor_shapes":0,"status":"time_limit"},{"model":"qwen/qwen3.8-2.4t-a95b","sheet":"F02","iou":0.0,"boundary_f1_1in":0,"area_error_pct":100.0,"seconds":966.7624881250085,"cost_usd":0.5871040000000001,"floor_shapes":0,"status":"time_limit"},{"model":"qwen/qwen3.8-2.4t-a95b","sheet":"F03","iou":0.0,"boundary_f1_1in":0,"area_error_pct":100.0,"seconds":1399.7952198330022,"cost_usd":0.687936,"floor_shapes":0,"status":"time_limit"},{"model":"qwen/qwen3.8-2.4t-a95b","sheet":"F04","iou":0.0,"boundary_f1_1in":0,"area_error_pct":100.0,"seconds":990.2368782919948,"cost_usd":0.786588,"floor_shapes":0,"status":"time_limit"},{"model":"qwen/qwen3.8-2.4t-a95b","sheet":"F05","iou":0.0,"boundary_f1_1in":0,"area_error_pct":100.0,"seconds":864.8237349159899,"cost_usd":0.38627799999999995,"floor_shapes":0,"status":"time_limit"},{"model":"qwen/qwen3.8-flash","sheet":"F01","iou":0.0,"boundary_f1_1in":0,"area_error_pct":100.0,"seconds":860.173312540981,"cost_usd":0.084053028,"floor_shapes":0,"status":"time_limit"},{"model":"qwen/qwen3.8-flash","sheet":"F02","iou":0.8799927800580001,"boundary_f1_1in":0.02412268858448681,"area_error_pct":10.878468289064013,"seconds":766.2640332090086,"cost_usd":0.08883659,"floor_shapes":5,"status":"time_limit"},{"model":"qwen/qwen3.8-flash","sheet":"F03","iou":0.8545403073031758,"boundary_f1_1in":0.09221846121904773,"area_error_pct":11.370017001044447,"seconds":673.2481209999823,"cost_usd":0.083402666,"floor_shapes":9,"status":"turn_limit"},{"model":"qwen/qwen3.8-flash","sheet":"F04","iou":0.0,"boundary_f1_1in":0,"area_error_pct":100.0,"seconds":776.3663354169985,"cost_usd":0.061969746,"floor_shapes":0,"status":"time_limit"},{"model":"qwen/qwen3.8-flash","sheet":"F05","iou":0.2810369335263254,"boundary_f1_1in":0.20017602724611594,"area_error_pct":69.50753127705345,"seconds":745.7944382500136,"cost_usd":0.07758628,"floor_shapes":11,"status":"time_limit"},{"model":"z-ai/glm-5.3","sheet":"F01","iou":0.6006969288264488,"boundary_f1_1in":0.012452196572932131,"area_error_pct":41.13012501942016,"seconds":161.11519575002603,"cost_usd":0.6310118,"floor_shapes":27,"status":"completed"},{"model":"z-ai/glm-5.3","sheet":"F02","iou":0.8912970127068952,"boundary_f1_1in":0.004655811338171526,"area_error_pct":6.3005005956847615,"seconds":217.51601895899512,"cost_usd":0.6103746400000001,"floor_shapes":14,"status":"completed"},{"model":"z-ai/glm-5.3","sheet":"F03","iou":0.9333620293155774,"boundary_f1_1in":0.09570720708752986,"area_error_pct":1.4054283149223479,"seconds":131.33003916702,"cost_usd":0.46123744,"floor_shapes":1,"status":"completed"},{"model":"z-ai/glm-5.3","sheet":"F04","iou":0.480617755060339,"boundary_f1_1in":0.2141355217068607,"area_error_pct":100.7419084567879,"seconds":200.8765668749984,"cost_usd":0.44990051999999997,"floor_shapes":21,"status":"completed"},{"model":"z-ai/glm-5.3","sheet":"F05","iou":0.7664360979220572,"boundary_f1_1in":0.4445616432033014,"area_error_pct":18.655084923338418,"seconds":221.2007902500045,"cost_usd":0.6410550800000001,"floor_shapes":35,"status":"completed"},{"model":"z-ai/glm-5.3-flash","sheet":"F01","iou":0.9151189149239038,"boundary_f1_1in":0.0028788124548035513,"area_error_pct":5.419832146286741,"seconds":448.4796685829933,"cost_usd":0.09069321817,"floor_shapes":15,"status":"completed"},{"model":"z-ai/glm-5.3-flash","sheet":"F02","iou":0.0,"boundary_f1_1in":0,"area_error_pct":100.0,"seconds":160.45991441601655,"cost_usd":0.12632749000000001,"floor_shapes":0,"status":"turn_limit"},{"model":"z-ai/glm-5.3-flash","sheet":"F03","iou":0.0,"boundary_f1_1in":0,"area_error_pct":100.0,"seconds":769.4688550419814,"cost_usd":0.09667637235,"floor_shapes":0,"status":"time_limit"},{"model":"z-ai/glm-5.3-flash","sheet":"F04","iou":0.860630372424114,"boundary_f1_1in":0.18564868703646498,"area_error_pct":5.81253169097887,"seconds":389.60746862500673,"cost_usd":0.04945371,"floor_shapes":11,"status":"completed"},{"model":"z-ai/glm-5.3-flash","sheet":"F05","iou":0.8121753960143276,"boundary_f1_1in":0.22677420939263424,"area_error_pct":10.835691613832596,"seconds":346.112674709002,"cost_usd":0.11454477,"floor_shapes":27,"status":"turn_limit"}],"base_wall_scope_scores":[{"model":"z-ai/glm-5.3","scope":"base","run_f1_1in":0.6493476266745348,"quantity_error_pct":24.05074600196548,"seconds":407.0272966659977,"cost_usd":1.1654498800000002,"status":"completed"},{"model":"z-ai/glm-5.3","scope":"wall_tile","run_f1_1in":0.7963470929414697,"quantity_error_pct":1.5164467951383127,"height_correct_predicted_area_fraction":0.18493224240922204,"seconds":407.0272966659977,"cost_usd":1.1654498800000002,"status":"completed"},{"model":"qwen/qwen3.8-flash","scope":"base","run_f1_1in":0,"quantity_error_pct":100.0,"seconds":787.0259336669988,"cost_usd":0.09167715600000001,"status":"time_limit"},{"model":"qwen/qwen3.8-flash","scope":"wall_tile","run_f1_1in":0,"quantity_error_pct":100.0,"height_correct_predicted_area_fraction":0,"seconds":787.0259336669988,"cost_usd":0.09167715600000001,"status":"time_limit"},{"model":"qwen/qwen3.8-2.4t-a95b","scope":"base","run_f1_1in":0,"quantity_error_pct":100.0,"seconds":771.8264243340236,"cost_usd":0.757906,"status":"time_limit"},{"model":"qwen/qwen3.8-2.4t-a95b","scope":"wall_tile","run_f1_1in":0,"quantity_error_pct":100.0,"height_correct_predicted_area_fraction":0,"seconds":771.8264243340236,"cost_usd":0.757906,"status":"time_limit"},{"model":"moonshotai/kimi-k3","scope":"base","run_f1_1in":0,"quantity_error_pct":100.0,"seconds":610.2490477079991,"cost_usd":1.586843322,"status":"sheet_cost_limit"},{"model":"moonshotai/kimi-k3","scope":"wall_tile","run_f1_1in":0,"quantity_error_pct":100.0,"height_correct_predicted_area_fraction":0,"seconds":610.2490477079991,"cost_usd":1.586843322,"status":"sheet_cost_limit"},{"model":"z-ai/glm-5.3-flash","scope":"base","run_f1_1in":0.4516749181055883,"quantity_error_pct":45.028887102057844,"seconds":540.1213092080143,"cost_usd":0.14368460000000002,"status":"turn_limit"},{"model":"z-ai/glm-5.3-flash","scope":"wall_tile","run_f1_1in":0.8157821200192182,"quantity_error_pct":6.754202980375484,"height_correct_predicted_area_fraction":0.6702205643939494,"seconds":540.1213092080143,"cost_usd":0.14368460000000002,"status":"turn_limit"},{"model":"deepseek/deepseek-v4.1-flash","scope":"base","run_f1_1in":0,"quantity_error_pct":100.0,"seconds":243.48362320798333,"cost_usd":0.21320703600000002,"status":"turn_limit"},{"model":"deepseek/deepseek-v4.1-flash","scope":"wall_tile","run_f1_1in":0,"quantity_error_pct":100.0,"height_correct_predicted_area_fraction":0,"seconds":243.48362320798333,"cost_usd":0.21320703600000002,"status":"turn_limit"},{"model":"minimax/minimax-m3","scope":"base","run_f1_1in":0,"quantity_error_pct":100.0,"seconds":330.8593024580041,"cost_usd":0.11055231999999998,"status":"turn_limit"},{"model":"minimax/minimax-m3","scope":"wall_tile","run_f1_1in":0,"quantity_error_pct":100.0,"height_correct_predicted_area_fraction":0,"seconds":330.8593024580041,"cost_usd":0.11055231999999998,"status":"turn_limit"},{"model":"deepseek/deepseek-v4-pro-0813","scope":"base","run_f1_1in":0.5323785968716975,"quantity_error_pct":18.889037791476703,"seconds":720.61874854099,"cost_usd":0.27270035232,"status":"completed"},{"model":"deepseek/deepseek-v4-pro-0813","scope":"wall_tile","run_f1_1in":0.5040756236107724,"quantity_error_pct":5.350082686174713,"height_correct_predicted_area_fraction":0.30039598028937103,"seconds":720.61874854099,"cost_usd":0.27270035232,"status":"completed"}],"base_wall_cost_note":"Two scope rows describe one trial; do not double-count runtime/cost.","scope":"five anonymous private floor sheets; one separate six-sheet base/wall source packet","inputs":"Private; source plans and reference geometry are not included."}

Reproduce the floor summary arithmetic

Copy the JSON block above into anonymized-results.json, then run this with Python 3 (standard library only):

import json, statistics
from collections import defaultdict
data = json.load(open("anonymized-results.json"))
groups = defaultdict(list)
for row in data["floor_trials"]:
    groups[row["model"]].append(row)
for model, rows in sorted(groups.items()):
    assert len(rows) == 5
    print(model,
          "IoU %", round(100 * statistics.mean(r["iou"] for r in rows), 1),
          "boundary F1 %", round(100 * statistics.mean(r["boundary_f1_1in"] for r in rows), 1),
          "area error %", round(statistics.mean(r["area_error_pct"] for r in rows), 1),
          "median min", round(statistics.median(r["seconds"] for r in rows) / 60, 1),
          "USD", round(sum(r["cost_usd"] for r in rows), 3))

The next comparison I would find useful is better completion discipline and accurate wall-face placement under a clearly stated allowance. If you work on model/tool integration, which part of this protocol would you change and measure next?