fix: stale frappe.local

[ankush]

• web.error.log shows that the web worker is failing to connect to the DB server.
• The error message indicates web workers are not using site config at all and using the default username without a password for connecting.
• Restarting the web workers fixed the issue.

[2023-04-09 08:12:31 +0000] [101928] [ERROR] Error handling request /api/method/ping
Traceback (most recent call last):
  File "/home/frappe/frappe-bench/env/lib/python3.7/site-packages/gunicorn/workers/sync.py", line 136, in handle
    self.handle_request(listener, req, client, addr)
  File "/home/frappe/frappe-bench/env/lib/python3.7/site-packages/gunicorn/workers/sync.py", line 179, in handle_request
    respiter = self.wsgi(environ, resp.start_response)
  File "/home/frappe/frappe-bench/env/lib/python3.7/site-packages/werkzeug/local.py", line 231, in application
    return ClosingIterator(app(environ, start_response), self.cleanup)
  File "/home/frappe/frappe-bench/env/lib/python3.7/site-packages/werkzeug/wrappers/base_request.py", line 237, in application
    resp = f(*args[:-2] + (request,))
  File "/home/frappe/frappe-bench/apps/frappe/frappe/app.py", line 97, in application
    for after_request_task in frappe.get_hooks("after_request"):
  File "/home/frappe/frappe-bench/apps/frappe/frappe/__init__.py", line 1363, in get_hooks
    hooks = _dict(cache().get_value("app_hooks", load_app_hooks))
  File "/home/frappe/frappe-bench/apps/frappe/frappe/utils/redis_wrapper.py", line 84, in get_value
    val = generator()
  File "/home/frappe/frappe-bench/apps/frappe/frappe/__init__.py", line 1337, in load_app_hooks
    for app in [app_name] if app_name else get_installed_apps(_ensure_on_bench=True):
  File "/home/frappe/frappe-bench/apps/frappe/frappe/__init__.py", line 1291, in get_installed_apps
    installed = json.loads(db.get_global("installed_apps") or "[]")
  File "/home/frappe/frappe-bench/apps/frappe/frappe/database/database.py", line 884, in get_global
    return self.get_default(key, user)
  File "/home/frappe/frappe-bench/apps/frappe/frappe/database/database.py", line 888, in get_default
    d = self.get_defaults(key, parent)
  File "/home/frappe/frappe-bench/apps/frappe/frappe/database/database.py", line 905, in get_defaults
    defaults = frappe.defaults.get_defaults(parent)
  File "/home/frappe/frappe-bench/apps/frappe/frappe/defaults.py", line 89, in get_defaults
    globald = get_defaults_for()
  File "/home/frappe/frappe-bench/apps/frappe/frappe/defaults.py", line 238, in get_defaults_for
    as_dict=1,
  File "/home/frappe/frappe-bench/apps/frappe/frappe/database/database.py", line 161, in sql
    self.connect()
  File "/home/frappe/frappe-bench/apps/frappe/frappe/database/database.py", line 85, in connect
    self._conn = self.get_connection()
  File "/home/frappe/frappe-bench/apps/frappe/frappe/database/mariadb/database.py", line 85, in get_connection
    local_infile=frappe.conf.local_infile,
  File "/home/frappe/frappe-bench/env/lib/python3.7/site-packages/pymysql/connections.py", line 353, in __init__
    self.connect()
  File "/home/frappe/frappe-bench/env/lib/python3.7/site-packages/pymysql/connections.py", line 633, in connect
    self._request_authentication()
  File "/home/frappe/frappe-bench/env/lib/python3.7/site-packages/pymysql/connections.py", line 907, in _request_authentication
    auth_packet = self._read_packet()
  File "/home/frappe/frappe-bench/env/lib/python3.7/site-packages/pymysql/connections.py", line 725, in _read_packet
    packet.raise_for_error()
  File "/home/frappe/frappe-bench/env/lib/python3.7/site-packages/pymysql/protocol.py", line 221, in raise_for_error
    err.raise_mysql_exception(self._data)
  File "/home/frappe/frappe-bench/env/lib/python3.7/site-packages/pymysql/err.py", line 143, in raise_mysql_exception
    raise errorclass(errno, errval)
pymysql.err.OperationalError: (1045, "Access denied for user 'frappe'@'blablah.compute.internal' (using password: NO)")

It seems any exception thrown in finally block prevent locals from releasing, which means subsequent requests can end up serving with incorrect locals.

• Non-httpexceptions don't trigger release_locals https://github.com/pallets/werkzeug/blob/main/src/werkzeug/local.py#L221
• This "cache" prevents frappe.init from re-initing:

  frappe/frappe/__init__.py

  Lines 187 to 188 in 6522021

  	if getattr(local, "initialised", None):
  	return

Full context: https://gameplan.frappe.cloud/g/frappe-cloud/projects/86/discussion/2250/

Credit: @adityahase for finding root cause.

[codecov]

Codecov Report

Merging #20695 (782828f) into develop (76e576c) will increase coverage by 0.00%.
The diff coverage is 75.00%.

Additional details and impacted files

@@           Coverage Diff            @@
##           develop   #20695   +/-   ##
========================================
  Coverage    63.92%   63.92%           
========================================
  Files          758      758           
  Lines        68755    68761    +6     
  Branches      6208     6208           
========================================
+ Hits         43952    43956    +4     
- Misses       21279    21281    +2     
  Partials      3524     3524           

Flag	Coverage Δ
server	68.84% <75.00%> (-0.02%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.