Username log filter causes database lock when a database exception is raised #1705
Comments
|
Someone else has this issue over on Discord, but we were never able to figure out what causes it. It seems to be failing somewhere in the update check. Can you change the update check interval back to the default of 6 hours or even increase it to 24 hours? |
JonnyWong16
added
status:need-more-info
topic:server
and removed
status:awaiting-triage
labels
|
Thanks for answer. ok, i set the interval up to 24h. I'll see if it helps. |
|
Mine was set to 1h, which I don't remember doing, but I've set it to 24h. I'll let you know what I see. |
|
Yes, mine was set to 1h too, and i don't did it. |
|
I have similar issue. I'm running Tautulli in Docker on Unraid with x86 CPU. Since I'm running Tautulli in Docker, I will completely disable update check. |
I have disabled update check, but Tautilli is still hangs. Logs: |
|
It has nothing to do with the update check. So I'm back to having no idea what is causing it. |
|
Is there any additional kind of logging or anything that we can enable to help troubleshoot? |
|
I think this is started after latest update to 2.9.5. Before that I have not any problem with Tautulli. |
|
I have the same problem, but it seems I also can trigger the problem by browsing the play history. Once I hit the play history the GUI hangs at "fetching rows" and the terminal outputs If I don't touch the GUI Tautulli goes unresponsive after a while with the folling messages in the terminal: Problem started arising around 2.9.4(docker container), have been running for years without a single hickup before that.
Git Branch: master |
|
If you are going to contribute to this issue then you need to provide all the info in the issues template. Saying "me too" does not contribute to identifying the issue at all.
|
|
Tautulli Version: 2.9.5 |
|
I have to say, for me, since i set the update check interval to 24h, the problem is gone now for 2 days. |
|
Tautulli Version: 2.9.5 |
|
Although completely disabling of update checking isn't help at all, set interval to 24h seems to be a working solution. Got almost 48 hours of uptime, it's record for now. |
|
Set to 24h isn't the solution. I runs for 2 days and then it stops working again. |
|
My workaround for now:
|
|
Apparently it hang 3 weeks ago with "skipped: maximum number of running instances reached" but I just noticed yesterday. Tautulli Version: 2.9.5 |
I have the same settings - update check set to 24h, container restarting every night (docker backups). Tautulli still hangs, but much less often. |
|
Same issue for me, running from docker (lscr.io/linuxserver/tautulli:latest) |
|
@JonnyWong16 any update on this? After latest update it got much worse - Tautulli hangs 3-4 times a day, setting update check interval to 24h isn't helping at all. I'm using Tautulli for more than 4 years, this is the only issue that I have. Is there are additional steps which we can follow to help you determine and fix issue? |
|
I still have absolutely no idea. |
|
This is started after v2.9.5 update. Is there are something that can cause this? Some kind of race condition, thread problems, etc. |
|
Try the following tests. Test 1: New Tautulli database with existing settings.
Test 2: Existing Tautulli database with default settings.
Test 3: New Tautulli database and default settings.
This might help narrow down if it is a specific setting or database issue. |
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as off-topic.
|
Hi, This happens on all our new Tautulli installations, any update on this ? |
|
No one has responded to my suggested tests from 3 weeks ago. |
|
Hi all. I found this thread for what might be a related reason. I have noticed that this hanging happens pretty much consistently when I am doing a mass import. If I get a whole new show, and I import them season by season, eventually Tautulli just craps out. I have a pastebin of the logs while Tautulli is unresponsive and it was in the middle of adding heaps of episodes. Here are the logs themselves: I am also using Docker, running on Unraid 6.10.1 |
Can you perform the tests that swift outlined? |
Hi, I performed your tests and Test 1 & Test 3 containers have not crashed but Test 2 container (existing database with default settings) has crashed in the same way as before. Logs from this container here - https://pastebin.com/DQGE4FPE |
|
Thanks for testing. Can you try repairing your database and testing again? Go through the steps in this FAQ even if it says your database isn't corrupted. https://github.com/Tautulli/Tautulli/wiki/Frequently-Asked-Questions#the-long-way |
|
Ok, I've done that. Will let you know how it goes. |
|
I am going to make a copy of my DB and try the repair as well. Will let you know how it goes. |
|
Also for Test 2, does it make a difference if you import your database using the "Merge" method vs the "Overwrite" method? |
I've created another instance using "Overwrite" and will monitor. |
|
The Test 2 with the imported database using "Merge" has crashed in the same way (I think). Logs here The "Overwrite" imported database instance is currently still up, will continue to monitor. Test 1 and Test 3 instances still running OK. |
|
The repair of the DB seems to have helped. My Tautulli container ran for a little over 15 hours before my daily backup script ran. Will see how it continues to run today. |
|
The Test 2 "Overwrite" container has crashed. Logs here |
|
Just wanted to report that, after the DP repair, everything seems to be working as expected and the container is now running as it was before the issue started. |
|
This is happening every time a collection of episodes are added. I'm going to run the database repair steps detailed here and see what happens. The result is that Tautulli completely hangs and stops responding or working. Highly annoying 😆 |
|
Have been facing the same issue for the past week so decided to spend some time this past weekend on figuring out the cause.
The actual issue is the logging which causes a deadlock when The specific deadlock happened here: Line 448 in 746d449 Line 130 in 746d449 Line 684 in 746d449 Disabling the username filtering stopped the lockups and allowed me to figure out the cause of the fatal errors, which were happening due to duplicate inserts in the DB, probably due to episodes from the same show getting processed by different threads: TVMaze DuplicateTMDB DuplicateImage Hash Duplicate |
|
Thank you for figuring this out. This also explains why manually repairing the database fixes it because going through the steps corrects the unique keys. Now I know exactly what to fix. |
JonnyWong16
changed the title
|
This bug has been fixed and will be available in the next release of Tautulli. This issue will be automatically closed once the update is available. |
|
Excellent, thanks. Perhaps, the temporary caches like the id/image lookups, usernames etc. can all be moved to a separate store (like https://pypi.org/project/diskcache/, specifically: https://grantjenks.com/docs/diskcache/tutorial.html#index) to reduce the chances of these kinds of locks ever happening in the future? Let me know if you want me to create a separate FR for this. |
Describe the Bug
I have tautulli in a docker container on a raspberry running.
It runs smooth for a while, but after round about a day it stops responding with this message in log.
After restarting the docker container it runs for a while again.
Steps to Reproduce
no steps, its doing by itself
Expected Behavior
running smoothly :)
Screenshots
No response
Relevant Settings
latest docker container: tautulli/tautulli:latest
Git Branch: master
Git Commit Hash: 9a4ca69590bacbbc5ec0076e078b9e38aea85e35
Configuration File: /config/config.ini
Database File: /config/tautulli.db | Status
Log File: /config/logs/tautulli.log
Backup Directory: /config/backups
Cache Directory: /config/cache
Export Directory: /config/exports
Newsletter Directory: /config/newsletters
Arguments: ['--datadir', '/config']
Platform: [Docker] Linux 5.10.103-v7+ (#1529 SMP Tue Mar 8 12:21:37 GMT 2022 - Debian GNU/Linux 11 bullseye)
System Timezone: UTC (UTC+0000)
Python Version: 3.9.10 (main, Jan 29 2022, 08:34:32) [GCC 10.2.1 20210110]
Tautulli Version
V2.9.5
Git Branch
master
Git Commit Hash
9a4ca69
Platform and Version
Docker
Python Version
3.9.10
Browser and Version
every Browser
Link to Logs
https://gist.github.com/ManuelW77/1c5f712f1b627dd2852d60946eff24c0
The text was updated successfully, but these errors were encountered: