Failing to set client_options for ElasticSearch backend

[wilbrdt]

Bug Report

Expected behavior/code
Using the --es-client-options to set a new CA certificate for ElasticSearch backend should not raise an error.

Actual Behavior
Fatal error :

2023-01-06 18:32:04,775 INFO     ralph.cli Running API server on 0.0.0.0:8100 with es backend
2023-01-06 18:32:04,776 INFO     ralph.cli Do not use runserver in production - start production servers through a process manager such as gunicorn/supervisor/circus.
INFO:     Will watch for changes in these directories: ['/app']
INFO:     Loading environment from '/tmp/tmp6uie9pai'
INFO:     Uvicorn running on http://0.0.0.0:8100 (Press CTRL+C to quit)
INFO:     Started reloader process [1] using WatchFiles
Traceback (most recent call last):
  File "/usr/local/bin/ralph", line 33, in <module>
    sys.exit(load_entry_point('ralph-malph', 'console_scripts', 'ralph')())
  File "/usr/local/bin/ralph", line 25, in importlib_load_entry_point
    return next(matches).load()
  File "/usr/local/lib/python3.9/importlib/metadata.py", line 86, in load
    module = import_module(match.group('module'))
  File "/usr/local/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 850, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/app/src/ralph/__main__.py", line 23, in <module>
    cli.cli()
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/app/src/ralph/cli.py", line 568, in runserver
    uvicorn.run(
  File "/usr/local/lib/python3.9/site-packages/uvicorn/main.py", line 564, in run
    ChangeReload(config, target=server.run, sockets=[sock]).run()
  File "/usr/local/lib/python3.9/site-packages/uvicorn/supervisors/basereload.py", line 45, in run
    for changes in self:
  File "/usr/local/lib/python3.9/site-packages/uvicorn/supervisors/basereload.py", line 64, in __next__
    return self.should_restart()
  File "/usr/local/lib/python3.9/site-packages/uvicorn/supervisors/watchfilesreload.py", line 85, in should_restart
    changes = next(self.watcher)
  File "/usr/local/lib/python3.9/site-packages/watchfiles/main.py", line 119, in watch
    with RustNotify([str(p) for p in paths], debug, force_polling, poll_delay_ms, recursive) as watcher:
FileNotFoundError: Permission denied (os error 13) about ["/app/k3d-storage/2d37968c-2b27-4271-a05d-53e62419dfba"]
Process SpawnProcess-1:
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/usr/local/lib/python3.9/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.9/site-packages/uvicorn/_subprocess.py", line 76, in subprocess_started
    target(sockets=sockets)
  File "/usr/local/lib/python3.9/site-packages/uvicorn/server.py", line 60, in run
    return asyncio.run(self.serve(sockets=sockets))
  File "/usr/local/lib/python3.9/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "uvloop/loop.pyx", line 1517, in uvloop.loop.Loop.run_until_complete
  File "/usr/local/lib/python3.9/site-packages/uvicorn/server.py", line 67, in serve
    config.load()
  File "/usr/local/lib/python3.9/site-packages/uvicorn/config.py", line 474, in load
    self.loaded_app = import_from_string(self.app)
  File "/usr/local/lib/python3.9/site-packages/uvicorn/importer.py", line 21, in import_from_string
    module = importlib.import_module(module_str)
  File "/usr/local/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 850, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/app/src/ralph/api/__init__.py", line 5, in <module>
    from ralph.conf import settings
  File "/app/src/ralph/conf.py", line 302, in <module>
    settings = Settings()
  File "pydantic/env_settings.py", line 39, in pydantic.env_settings.BaseSettings.__init__
  File "pydantic/main.py", line 342, in pydantic.main.BaseModel.__init__
pydantic.error_wrappers.ValidationError: 1 validation error for Settings
BACKENDS -> DATABASE -> ES -> CLIENT_OPTIONS
  value is not a valid dict (type=type_error.dict)

Steps to Reproduce
bin/ralph runserver -b es --es-client-options ca_certs=toto

Environment

• Ralph version: master
• Platform: Linux 62-Ubuntu SMP Tue Nov 22 19:54:14 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

[jmaupetit]

Looks like it only occurs while running the runserver command. It seems related to the way we serialize the settings in a temporary file. As an example, let's use the following command:

$ bin/ralph runserver -b es --es-client-options verify_certs=True

Output (truncated) logs:

2023-01-08 21:37:03,644 INFO     ralph.cli Running API server on 0.0.0.0:8100 with es backend
2023-01-08 21:37:03,644 INFO     ralph.cli Do not use runserver in production - start production servers through a process manager such as gunicorn/supervisor/circus.
INFO:     Will watch for changes in these directories: ['/app']
INFO:     Loading environment from '/tmp/tmpmd28cd30'
INFO:     Uvicorn running on http://0.0.0.0:8100 (Press CTRL+C to quit)
INFO:     Started reloader process [1] using WatchFiles
Process SpawnProcess-1:
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
  [...]
  File "/app/src/ralph/api/__init__.py", line 5, in <module>
    from ralph.conf import settings
  File "/app/src/ralph/conf.py", line 302, in <module>
    settings = Settings()
  File "pydantic/env_settings.py", line 39, in pydantic.env_settings.BaseSettings.__init__
  File "pydantic/main.py", line 342, in pydantic.main.BaseModel.__init__
pydantic.error_wrappers.ValidationError: 1 validation error for Settings
BACKENDS -> DATABASE -> ES -> CLIENT_OPTIONS
  value is not a valid dict (type=type_error.dict)

If we take a look at the temporary settings:

$ bin/compose exec app cat /tmp/tmpmd28cd30
RALPH_RUNSERVER_BACKEND=es
RALPH_BACKENDS__DATABASE__ES__CLIENT_OPTIONS={'verify_certs': True}
RALPH_BACKENDS__DATABASE__MONGO__COLLECTION=marsha
RALPH_BACKENDS__DATABASE__MONGO__DATABASE=statements
RALPH_BACKENDS__DATABASE__MONGO__CONNECTION_URI=mongodb://mongo:27017/
RALPH_BACKENDS__DATABASE__ES__OP_TYPE=index
RALPH_BACKENDS__DATABASE__ES__INDEX=statements
RALPH_BACKENDS__DATABASE__ES__HOSTS=http://elasticsearch:9200

The RALPH_BACKENDS__DATABASE__ES__CLIENT_OPTIONS environment variable is not properly parsed.