fix(evals): persist eval name on edit so renames actually stick - #1850
fix(evals): persist eval name on edit so renames actually stick#1850onatozmenn wants to merge 2 commits into
Conversation
EditAndRunUserEvalView accepted a ame in the request serializer but never assigned it, so the only way to change an eval's name was to delete it and add it again, which re-ran the eval over every row. The rename now goes through validate_eval_name and the same per-dataset uniqueness check AddUserEvalView uses, and it carries over to the eval column and its reason column, which both snapshot the name at creation time. The save_as_template branch is untouched since ame means the new template there. Blank or unchanged names are no-ops. Frontend: the Name field in the eval picker was hard-disabled in edit mode. It is editable now, with the existing length validation applied in both modes. Closes future-agi#1769
|
Two process questions, neither blocking on my side. The CLA bot never showed up on this PR, so I left that box unchecked. I looked for it and could not find a CLA workflow in Also, the only check that ran here is |
|
Good catch, you were right. I moved the rename validation and uniqueness check ahead of all transactional writes, including version creation. I also strengthened the collision regression test to verify that both the version count and pinned version remain unchanged after a rejected edit. The full |
|
Verified the new commit (439c0bd) against the diff — this addresses the transactional leak correctly:
One tiny nit (non-blocking): Nothing further from my side — the fix looks mergeable pending maintainer review. Thanks for the quick turnaround! |
Summary
An eval's name could not be changed once it was added to a dataset.
EditAndRunUserEvalViewacceptsnamein its request serializer and the frontend sends it on every edit, but the view never assigned it, so the request came back200 OKwhile the name stayed the same. The Name field in the eval picker was also hard-disabled in edit mode. The only workaround was to delete the eval and add it again, which re-ran it over every row and threw away the existing results.This makes the rename actually work, end to end.
Linked issues
Closes #1769
Linear:
Type of change
1) What changes were done
A. Backend: persist the name on edit
EditAndRunUserEvalView.postnow assignsnameto theUserEvalMetricwhen one is sent and it differs from the current name.validate_eval_name, the same helperUserEvalSerializer.validate_nameuses when an eval is created, so the rules do not differ between create and rename.AddUserEvalViewuses (name+organization+dataset,deleted=False), excluding the eval being edited, and returns the existingEVAL_NAME_EXISTSerror on a collision.nameon every save.save_as_templatebranch is untouched. Therenameis the name of the newEvalTemplate, not the instance.B. Backend: keep the grid columns in sync
_rename_eval_columnshelper renames the eval column and its reason column, which both snapshot the eval name when they are created (see theColumn.objects.createcalls around lines 7619 and 8446). Without this the rename would land on the instance but the grid header would still show the old name.{eval}-reasonfor datasets and{eval}-{column}-reasonfor experiments, so the helper swaps the name prefix instead of rebuilding the whole string.get_or_createkeyed partly onf"{eval_metric.name}-reason", so renaming afterwards would have created a duplicate reason column under the new name.C. Frontend: unblock the Name field
EvalPickerConfigFullhaddisabled={isEditMode}on the Name input with a comment saying the name belongs to theUserEvalMetricinstance rather than the template. That is exactly the name this PR makes editable, so the flag is gone.!isEditModebefore because the field could not be typed in.resolvedNamealready resolved to the picker'sevalNamestate in edit mode, andEvaluationDraweralready put it in the POST payload.2) Why the changes were done
validate_eval_nameand the existing uniqueness filter keeps rename and create consistent. Adding a separate rule set for renames would let a name exist that could never have been created.post_savesignal onUserEvalMetric, because the eval column and reason column are already reconciled inside this same view and this keeps the whole rename in one transaction.3) Tests written + scenarios each covers
model_hub/tests/test_evaluation_api.py—TestEditAndRunUserEvalViewtest_edit_persists_new_name— a rename is written to the instance instead of being dropped.test_edit_renames_eval_and_reason_columns— the eval column and its-reasoncolumn follow the rename.test_edit_rejects_name_taken_by_another_eval— a collision with another eval on the same dataset returns 400 and leaves the name alone.test_edit_rejects_invalid_name_format— a name with spaces is rejected, matching the create path.test_edit_without_name_keeps_existing_name— an emptynamein the payload does not blank the eval out.test_edit_with_unchanged_name_is_not_a_collision— resending the eval's own name does not trip the uniqueness check against itself.One existing test changed:
test_edit_and_run_user_eval_successsent"name": "Updated Eval". That name has a space and a capital letter, so it could never have been created throughadd_user_evalin the first place, and it now hits the shared validator. It sendsupdated-evalinstead. The assertions are unchanged.4) How to run / test
UI steps (manual)
5) Screenshots / recordings
I did not attach UI screenshots. I could not bring the full compose stack up locally, so the frontend side of this is a code change I reasoned through rather than one I clicked through. Happy to redo it with screenshots if you would rather see that before merging.
6) Edge cases & considerations
validate_eval_namewas enforced. Only the incoming name is validated, so those evals can still be renamed to something valid. They are just not silently rewritten.allow_blank=Trueand the drawer always sendsname, so a blank value keeps the current name rather than wiping it.save_as_template. Skipped entirely, otherwise saving a template would rename the instance as a side effect.-sourceid-{id}suffix and name reason columns{eval}-{column}-reason, so the helper filters and rewrites for that shape as well as the dataset one.run=falseand never ran, there is no column to rename yet. The filters just match nothing and the instance rename still applies.7) Pre-existing issues (NOT introduced by this PR)
black --checkandisort --check-onlyalready fail onmodel_hub/views/develop_dataset.pyandmodel_hub/tests/test_evaluation_api.pyondev. I matched the surrounding style rather than reformatting the files, since that would have buried the change in a few hundred unrelated lines.eslintreports areact-hooks/exhaustive-depswarning on the effect atEvalPickerConfigFull.jsx:272. It is ondevtoo and is unrelated to the lines I touched.8) Architectural / important decisions
validate_eval_nameinstead of a rename-specific rule set: a renamed eval should not be able to hold a name thatadd_user_evalwould have rejected.transaction.atomic()block.get_or_creatematches on the reason column name, so ordering it the other way would create a duplicate column.Checklist
bin/testpasses locally (backend)yarn test:runpasses locally (frontend)yarn contracts:checkpasses if API surface changedTwo notes on the boxes I left unchecked. I could not run the frontend suite or
contracts:checkon Windows:yarn installdies building the nativecanvasdependency, so I only linted the file I changed. The API surface is unchanged, no serializer fields were added or removed, so I do not think contracts need regenerating, but say the word if you want me to confirm. I do not have a Linear account, so I could not link the issue there. I will sign the CLA when the bot prompts.Investigated with AI assistance.