fix: prevent environment pollution when importing pyspark

[sam-goodwin]

Closes: #1551

• Change catch to a finally to ensure environment is always popped when probing for pyspark.pandas in external_config.py
• Keep track of dirty state and only pop environment variables that were modified

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 83.07%. Comparing base (4df61da) to head (01e0cea).
Report is 51 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff             @@
##             main    #1552       +/-   ##
===========================================
- Coverage   94.29%   83.07%   -11.22%     
===========================================
  Files          91      111       +20     
  Lines        7024     8191     +1167     
===========================================
+ Hits         6623     6805      +182     
- Misses        401     1386      +985     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[cosmicBboy]

thanks for the fix!