Reduce high memory usage

[Ruriko]

Is it normal for the python script to use a lot of memory cause it currently uses about 1gb ram. Is there anyway to reduce it?

ps aux | grep python | awk '{sum=sum+$6}; END {print sum/1024 " MB"}'
1034.71 MB

ps aux  | grep python
root       307  0.0  0.2  31852  5980 ?        Ss   Aug03   0:00 /usr/bin/python3 /usr/bin/networkd-dispatcher --run-startup-triggers
root       465  0.0  0.2 110588  6404 ?        Ssl  Aug03   0:00 /usr/bin/python3 /usr/share/unattended-upgrades/unattended-upgrade-shutdown --wait-for-signal
root       570  1.0 41.2 1954168 1051908 ?     Ssl  Aug03  32:23 python3 /var/www/api.adoreanime.com/htdocs/HibiAPI/main.py
root     43690  0.0  0.0   9032   664 pts/0    S+   06:36   0:00 grep --color=auto python

[mnixry]

I think this is caused by the caching feature of HibiAPI.

By default, HibiAPI turns on the default caching feature. However, in the case of poor cache hit rate, this can cause a lot of useless memory usage. This is especially true in the case of multiple workers and no Redis, because the default memory cache of aiocache caches one copy per worker process, which cannot be solved at this time.

So for your current problem, we have the following solutions:

1. Turn off the caching option in general.yml, which will disable caching globally

• HibiAPI/hibiapi/configs/general.yml

  Lines 33 to 36 in d997c5a

  	cache:
  	enabled: true # 设置是否启用缓存
  	ttl: 3600 # 缓存默认生存时间, 单位为秒
  	uri: "memory://" # 缓存URI

2. For low hit rate requests, you can add a Cache-Control: no-store request header to skip the data caching
3. Use Redis as an external cache or reduce the number of workers (since for most requests, increasing the number of workers does not improve performance)

Due to my poor algorithmic knowledge, I haven't found a better asynchronous caching method that is easy to implement and scalable, perhaps you can help us improve the current caching algorithm (e.g. using LRU or a better algorithm)?

[Ruriko]

How do I set to use redis as the external cache?

[mnixry]

How do I set to use redis as the external cache?

Install aioredis<=2.0.0, and use redis://... URL in caching settings.

[mnixry]

Now, HibiAPI supports the diskcache backend, which means we can use the cache in the disk instead of consuming a large amount of memory. You can reference to cashews document to set up this.