shorten error string, more enhancements needed

[aseembits93]

TODOS

1. unique message per test function and not per invocation id
2. shorten very long output values with "..."

PR Type

Enhancement

---

Description

• Add pytest error shortening utility

• Integrate shortening into test comparison

• Preserve non-LSP exit behavior unchanged

---

Diagram Walkthrough

flowchart LR
  U["code_utils.shorten_pytest_error()"] -- "extract concise lines" --> S["shortened pytest error"]
  E["equivalence.compare_test_results()"] -- "apply to candidate error" --> S
  E -- "apply to original error" --> S

Loading

File Walkthrough

Relevant files

Enhancement

code_utils.py Add pytest error shortening utility                                            codeflash/code_utils/code_utils.py Introduce shorten_pytest_error helper. Extracts E>/> lines via multiline regex. Leaves exit_with_message behavior unchanged. +4/-0      equivalence.py Integrate error shortening into comparison flow                    codeflash/verification/equivalence.py Import and use shorten_pytest_error. Shorten candidate and original pytest errors. Apply only when errors are present. +5/-1

---

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.

---

Codeflash Bot seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[github-actions]

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 2 🔵🔵⚪⚪⚪

🧪 No relevant tests

🔒 No security concerns identified

⚡ Recommended focus areas for review Possible Issue The regex in shorten_pytest_error only captures lines starting with E or > followed by spaces; this may miss other informative pytest lines (e.g., AssertionError: lines without leading markers, diff blocks, or traceback context) and could over-truncate valuable context. Validate that the shortening still preserves sufficient information for comparisons. def shorten_pytest_error(pytest_error_string: str) -> str: return "\n".join(re.findall(r"^[E>] +(.*)$", pytest_error_string, re.MULTILINE)) Behavior Change Applying shorten_pytest_error before comparing candidate vs. original errors changes matching semantics; ensure downstream consumers and any equality/substring logic expect shortened strings and that output shown to users remains helpful. if cdd_pytest_error: cdd_pytest_error = shorten_pytest_error(cdd_pytest_error) original_pytest_error = ( original_test_failures.get(original_test_result.id.test_fn_qualified_name(), "") if original_test_failures else "" ) if original_pytest_error: original_pytest_error = shorten_pytest_error(original_pytest_error)

[github-actions]

PR Code Suggestions ✨

Explore these optional code suggestions:

Category	Suggestion	Impact
Possible issue	Fix variable name and optimize regex This will raise a NameError because pytest_error_string is referenced but not defined; the parameter is named pytest_error_string. Use the parameter consistently. Also, precompile the regex at module scope to avoid recompilation on each call if this runs often. codeflash/code_utils/code_utils.py [365-366] +_PYTEST_ERR_LINE_RE = re.compile(r"^[E>] +(.*)$", re.MULTILINE) + def shorten_pytest_error(pytest_error_string: str) -> str: - return "\n".join(re.findall(r"^[E>] +(.*)$", pytest_error_string, re.MULTILINE)) + return "\n".join(_PYTEST_ERR_LINE_RE.findall(pytest_error_string)) Suggestion importance[1-10]: 9 __ Why: The function references pytest_error_string but the parameter is named pytest_error_string in the suggestion text but actually is pytest_error_string? In the PR code it's pytest_error_string used in body while the parameter is pytest_error_string? The code uses an undefined pytest_error_string, which would raise NameError; fixing it is critical. Precompiling the regex is a valid optimization aligned with the change.	High
General	Preserve error when shortening empties shorten_pytest_error can return an empty string; downstream consumers may expect non-empty messages. Guard against converting non-empty errors into empty strings by falling back to the original error if shortening yields empty output. codeflash/verification/equivalence.py [61-69] if cdd_pytest_error: - cdd_pytest_error = shorten_pytest_error(cdd_pytest_error) + shortened = shorten_pytest_error(cdd_pytest_error) + cdd_pytest_error = shortened or cdd_pytest_error ... if original_pytest_error: - original_pytest_error = shorten_pytest_error(original_pytest_error) + shortened = shorten_pytest_error(original_pytest_error) + original_pytest_error = shortened or original_pytest_error Suggestion importance[1-10]: 5 __ Why: Guarding against an empty result from shorten_pytest_error is reasonable and low-risk, preserving information. It's a minor robustness improvement without changing behavior otherwise.	Low

[KRRT7]

maybe change the var name? it's no longer the original pytest error if it's been shortened

[KRRT7]

codeflash will optimize this away or should've