perf: reduce import overhead on background jobs

[ankush]

Avoids 7MB of overhead and cleanup costs for each background job.

Memory profile of worker horse doing frappe.ping

After

Also, several other modules are loaded instantly on new jobs. In worker pool we can save on this CPU cost by preloading (paying shared memory cost). This increases memory usage of 8 workers by 20MB but reduces import overheads on each job by almost 60-70%

---

performing 100 job (frappe.ping) - good enough proxy for overheads.

Before:

λ bench worker-pool --num-workers=8 --burst > /dev/null  
8.67s user 5.18s system 491% cpu 2.815 total

After:

λ time bench worker-pool --num-workers=8 --burst > /dev/null
1.71s user 2.56s system 179% cpu 2.376 total

performing 100 job (frappe.email.queue.flush) - realistic work

Before:

λ time bench worker-pool --num-workers=8 --burst > /dev/null
26.98s user 11.11s system 597% cpu 6.378 total

After:

λ time bench worker-pool --num-workers=8 --burst > /dev/null
8.61s user 4.73s system 510% cpu 2.613 total

[ankush]

Analyzing typical imports that we can save for worker-pool:

• safe_exec - 6mb
• requests - 5mb
• PIL - 1.5mb
• filetype - 1.4mb
• SQLParse - 2mb
• bs4 - 4.3mb
• pypdf - 5.2

[ankush]

Some python shenanigans, if I explicitly import frappe in local scope then it works fine.

λ bench worker-pool
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/home/ankush/benches/develop/apps/frappe/frappe/utils/bench_helper.py", line 128, in <module>
    main()
  File "/home/ankush/benches/develop/apps/frappe/frappe/utils/bench_helper.py", line 34, in main
    FrappeCommandGroup(commands=commands)(prog_name="bench")
  File "/home/ankush/benches/develop/env/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ankush/benches/develop/env/lib/python3.11/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/home/ankush/benches/develop/env/lib/python3.11/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ankush/benches/develop/env/lib/python3.11/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ankush/benches/develop/env/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ankush/benches/develop/env/lib/python3.11/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ankush/benches/develop/apps/frappe/frappe/commands/scheduler.py", line 225, in start_worker_pool
    start_worker_pool(
  File "/home/ankush/benches/develop/apps/frappe/frappe/utils/background_jobs.py", line 329, in start_worker_pool
    if frappe._tune_gc:
       ^^^^^^
UnboundLocalError: cannot access local variable 'frappe' where it is not associated with a value