Replies: 6 comments 6 replies
-
|
Update https://gist.github.com/dalf/b3728182ef69f855c9103db6e31f38ed With indexes, SQLite is actually only 10 to 20 time slower than a memory access, and with a small cache is can be similar. Of course, it does not solve the statistics issue (shared the stats between the workers). ping @return42 [EDIT] Anyway, if the memory is issue, the translation loads about 60MB in RAM (checked |
Beta Was this translation helpful? Give feedback.
-
|
@dalf I'm sorry for the delay, I did not had much time this weekend and just tried to finish some small task .. for this topic I need more time .. and next week I have a lot of work to do .. give me some time and I will come back to this discussion / many thanks for your thoughts and analysis 👍 |
Beta Was this translation helpful? Give feedback.
-
I think access to individual records is not quite as critical since we need access to those records relatively infrequently (we will rarely do a SQL-SELECT on these tables). I would be more concerned with a SQL solution that initializing the SQL databases (connect and first query) takes more time than loading a python module with a big dictionary in. Do you have any experience with this? |
Beta Was this translation helpful? Give feedback.
-
Are you refereeing to
|
Beta Was this translation helpful? Give feedback.
-
Yes, plus a first query .. I assume connection is (due to lazy evaluation) ultra fast but when the first SQL-SELECT is done on a table, the table needs to be loaded .. and this might take more time compared to read the python module by the python interpreter.
Yes, build time does not care, we store the SQLite DB in the repo (will replace the python modules from /data). |
Beta Was this translation helpful? Give feedback.
-
|
In the gist, the max time in the benchmark is 0,2 ms. When the server starts, it can send a dummy request to avoid the initialization overhead of the first request. |
Beta Was this translation helpful? Give feedback.
-
|
UPDATE: I must think about it again, may I still have an error in my thoughts about the process model and process flow ... give me some more time, then I'll get back to you ... as soon as I have sorted my thoughts :)
I'm unsure about the process flow, especial when wsgi (walkers) came into play .. You mean the uwsgi prepare the workers and those workers waiting for a request .. I'm right so far? .. and when worker is build by the wsgi server, we can do a dummy SQL-SELECT on the tables? In the process flow I have in mind there is no wsgi server .. and what I asking myself is: how long would it take to read the python modules compared to a connect and single SQL-SELECT .. I don't know if I could express myself clearly, I want to achieve a performance gain, which is also possible without an additional component like a WSGI server. |
Beta Was this translation helpful? Give feedback.
-
|
What I see:
DATA_CONNECTION = None
def data_initialize():
global DATA_CONNECTION
# Read-only connection :
# * SQLite does not support write from multi-processes (multiple processes can read the database).
# * and most importantly : the webapp must not modify data.db. It is updated once per month by the `searxng_extra`
DATA_CONNECTION = sqlite3.connect('file:data.db?mode=ro', uri=True)
# https://phiresky.github.io/blog/2020/sqlite-performance-tuning/
DATA_CONNECTION.executescript("""
pragma temp_store = memory;
pragma mmap_size = 30000000000;
""")
# TODO : send a dummy request to initialize the database, if this decrease the overhead of the following requests.
def fetch_engine_description(engine, language):
res = DATA_CONNECTION.execute("SELECT description, source FROM engine_descriptions WHERE engine=? AND language=?", (engine, language))
result = res.fetchone()
if result is None:
return None, None
return result[0], result[1]==> outside the |
Beta Was this translation helpful? Give feedback.
-
|
Actually, The line that allocates memory: Line 157 in 11c0651 The purpose of this code is to do that: So, one idea is to store these values in |
Beta Was this translation helpful? Give feedback.
-
wow thats much 👍 .. may we have a chance to remove that monkey patching of flask_babel Line 134 in 11c0651 Form the .. Lines 37 to 41 in 11c0651 we can at least remove
|
Beta Was this translation helpful? Give feedback.
-
This is unrelated to my comment. I can't say if this is possible. I've just focused on |
Beta Was this translation helpful? Give feedback.
-
|
The "issue" with sqlite is readability: it requires an external program to read the content compare to json. We can keep an SQL dump in the repository: it won't be use, but it makes the PRs easy to compare. SQLite supports multiple tables per database, in the case of SearXNG it can make sense to create one database file per data type, so we keep the update workflow. |
Beta Was this translation helpful? Give feedback.
-
For reference :
sqlite is available in all Python standard version, multiple workers can access the same read-only database. It would decrease the memory footprint by ~ 10 MB per worker according to
tracemalloc.The drawback is the access time : about 300 nanosec instead of few nanosec. A small cache can help., and it would not be a problem for
engine_descriptions.jsonfor example.external_bangs.jsonmight be different since there are a lot of access and they need to be fast.Beta Was this translation helpful? Give feedback.
All reactions