⚡️ Speed up function format_numeric_fields by 8%

[codeflash-ai]

📄 8% (0.08x) speedup for format_numeric_fields in datacompy/spark/helper.py

⏱️ Runtime : 397 milliseconds → 366 milliseconds (best of 5 runs)

📝 Explanation and details

The optimization achieves an 8% speedup through two key changes that reduce Python overhead during DataFrame column processing:

1. Set-based membership testing: Changed numeric_types from a list to a set. The original code performed c[1] not in numeric_types checks 229 times according to the profiler. List membership testing is O(n) while set membership is O(1), making lookups significantly faster when checking data types.

2. Generator expression instead of list building: Replaced the explicit loop that builds a fixed_cols list with a generator expression that lazily computes column transformations. This eliminates the overhead of:

• Multiple list.append() calls (229 total appends in the original)
• Intermediate list storage
• The explicit loop structure

The profiler results show the optimization's impact clearly - the original code spent significant time in the loop structure (lines with c[1] not in numeric_types, fixed_cols.append() calls), while the optimized version consolidates all the column processing logic into a single generator that's consumed directly by df.select().

The generator approach is particularly effective here because PySpark's select() method can consume the generator directly without materializing an intermediate list, reducing memory pressure and eliminating Python-level iteration overhead. This optimization is most beneficial for DataFrames with many columns, where the membership testing and list building costs accumulate significantly.

✅ Correctness verification report:

Test	Status
⚙️ Existing Unit Tests	✅ 7 Passed
🌀 Generated Regression Tests	🔘 None Found
⏪ Replay Tests	🔘 None Found
🔎 Concolic Coverage Tests	🔘 None Found
📊 Tests Coverage	100.0%

⚙️ Existing Unit Tests and Runtime

Test File::Test Function	Original ⏱️	Optimized ⏱️	Speedup
test_spark/test_helper.py::test_format_numeric_fields	5.81ms	5.19ms	11.9%✅

To edit these changes git checkout codeflash/optimize-format_numeric_fields-mi5xgu5t and push.