Fix memory leak in forked worker caused by unreleased file descriptor locks

[wjddn279]

Problem?

We conducted an investigation to resolve the continuous memory growth in the Local Executor's forked processes, following the previous memory spike fix.

Using memray to profile the Local Executor's forked processes, we confirmed that memory was steadily increasing. Specifically, logging-related memory was growing, and this was traced to the process of writing to local files.
memray-flamegraph-output-2026-04-11 13:17:18.516888.html

Cause?

Looking at, full_path.touch(new_file_permissions) conf.get("logging", "file_task_handler_new_file_permissions", fallback="0o664") in flamegraph, I noticed that basic string objects were not being properly garbage collected, which led us to suspect unreleased references.

Upon examining self._lock = _get_lock_for_file(self._file) in the memory-increasing area, I found that file descriptors were being cached as locks in a dictionary with no corresponding release mechanism. This prevented the file descriptors from being properly released, and consequently, the parent Process objects also retained references and were never garbage collected.

Solution

After adding code to manually release these references, we re-ran the profiling and confirmed that the related objects were being properly freed.
memray-flamegraph-output-2026-04-12 13:14:12.556546.html

PS

The remained parsed = [sys.intern(str(x)) for x in rel.split(sep) if x and x != '.']issue will be addressed in a separate PR.

---

Was generative AI tooling used to co-author this PR?

• Yes (please specify the tool below)

---

• Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
• For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
• When adding dependency, check compliance with the ASF 3rd Party License Policy.
• For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

[eladkal]

importing private modules from upstream library is probably not something we want to do

[wjddn279]

I've made pr for handling it in structlog library hynek/structlog#806.
I can't confirm it could be accepted and merged, but there is no other way to resolving this issue

[ashb]

Yeah, we shouldn't really do this. If we correctly close/delete the structlog objects it should handle this for us.

Doing it this way feels liek papering over the problem?