LogSpam + possible speedups #543
Comments
|
Some update to this: heap size use is set to 1200 so UMS allocates something less to this value. Scanning library generated 1,1GB DB file so it seems it failed due to insufficient memory. Unfortunately I don't have logs because parents restarted computer so it was overwritten 👎 |
|
@ExSport if the DB is that big it's probably a bug in how we store DB information, so logs will be useful |
|
@SubJunk I will try to record somehow but somewhere in the middle of scanning log had few hundreds of MegaBytes. I think most of the place in db are thumbs? Will try to disable them because scanning e.g. (never counted them) 50'000 pics and 7'000 videos should be the culprit of huge db (if thumbs are stored) and also FFmpeg spamming is also culprit of huge log when scanning files :-) |
|
@ExSport If you want to have a look at the DB, I recommend DbVisualizer. Thumbs are being stored in the THUMB column of the FILES table. |
|
@taconaut Thanks for tip! I tried shortly and I see DB with "INFORMATION_SCHEMA" and "PUBLIC" (empty) schema. I tried to find what you mentioned and also some filenames, codecs etc. but without success. I know MS SQL but here I was little lost, will try look again later. |
|
@taconaut You mentioned FILES...I see this in file c:\ProgramData\UMS\database\ medias.trace.db: Seems not standard behavior. |
|
@ExSport I think the trace.db file contains the DB logs. Btw in mlx thumbs and covers are stored as you propose in the file system with the path in the DB and I haven't seen any issues with DB size. |
|
@taconaut Thanks! Replacing h2.jar with the latest one from [http://www.h2database.com/html/main.html] fixed it! Now I can debug DB more easily 👍 |
|
@ExSport Cool, you're welcome :) |
|
Other tips for speed improvements: |
|
@SubJunk It seems UMS doesn't use multithreading for FFMPEG. By default -threads param is 1 (minimally for MPEG2 output) so CPU usage is about 40% and video stutter. It seems for mpeg2 output -slices param by default is not definitely 1 from my tests so it can be omitted perhaps but not -threads as it generates 38fps for tested video (1080p) for first 1min where there is no "action" so later in movie CPU is not fully utilized so realtime encoding is impossible. Adding -threads 4 to command line generates about 70% CPU and 66fps what is much better for realtime encoding. |
|
@ExSport FFmpeg shouldn't use the -threads param by default, since FFmpeg is good at automatically detecting threads. So that's either a UMS bug or a setting you forgot about, is it the same with a clean install? |
|
@SubJunk Check the latest FFmpeg code/discussion/documentation and you will find threads=1 by default. All my testing were from command line (x64) and I am sure for mpeg2 output when threads param is missing it is used as 1 by default. I searched for default settings and found some older posts where FFmpeg is detecting threads automatically [-threads 0] and also slices but it also depends on the output format as mpeg2 can be different from x264 etc. In another discussion there is info that by default it is nr. of cores minus 1 so maybe my dual core with multithreading = 4 threads are count as 2-1=1. But latest posts talk about default settings -threads 1 what also shows my own tests. |
|
@ExSport tested current implementation and FFmpeg uses both my CPU cores on about 65% and the rest is for Java and other processes. The -threads in the command line are not specified. But it is an old notebook with Intel T5500, 1.6 MHz CPU. EDIT: changing - threads to 4, 6 makes no difference. For me the CPU is always utilized on 100% until the cache is not full. Than it drops to about 25%. EDIT2: FFmpeg is compiled with --disable-w32threads but I am running UMS on the 32-bit Windows. So I have no idea how it works. |
|
Tests from yesterday where one ffmpeg64 never took more than 50% CPU without -threads param and 70% with -threads 4/8 (two ffmpeg were able to utilize whole CPU): I repeated these tests 20x with consistent results but today I don't understand it but omitting -threads param behaves like -threads 0 [auto] which is about 7fps slower than 4 or 8 (auto is not so optimal). -loglevel verbose & debug (funny that adding debug level, FFmpeg shows correct fps hidden by default but for other file it shows 0 so not useful): Verbose: MediaInfo: It seems auto threading is not working correctly at all circumstances but forcing -threads 4 fixed slow encoding in 100%. |
|
@ExSport maybe it is the ffmpeg64 bug. I tested the 32-bit version. I will test it on my PC with 64-bit Windows. |
|
@valib Maybe it is also connected to specifics of tested file. I will try other files later. Now I tested 32bit ffmpeg with similar results. Using -threads 4 or 8 and -slices 32 makes encoding fast as possible (same for 32/64 version). |
|
@ExSport I did testing a while ago and found it was fastest without threads specified. I'm not even sure if the -threads option does anything now anyway, since the FFmpeg documentation lists its only possible value as "auto". |
|
@SubJunk I suppose it is also not consistent due to different files. There is no auto value string as FFmpeg forum say and also my testing with -threads auto ignores it as wrong argument. Auto is represented by -threads 0 but as I said, for tested file it was inconsistent and also always slower. Also users in discussion writes it differs for different files. Maybe my behavior is also due wrong fps detection so outputting 50fps can have different behavior as with 25fps and also different for x264 and mpeg2 encoding. I also tested another file with correct 25fps output, non interlaced mkv file and CPU utilization with default settings (without threads param) was much better so as I said, testing only one type of file is not relevant. |
|
Some update about using -r -threads -slices? 👍 |
|
Old issue, it should not be relevant anymore... |
v5.1.3 Build: 987e16d (2015-04-27)
File is ok on renderer so making possible/configurable to not hide similar files will be great. Why switching to ffmpeg for picture?
What's this spam? I thought it is related to XBOX only:-)
When creating thumbs I spotted messages like "Consider increasing the value for the 'analyzeduration' and 'probesize' options"
By default first one is 5000000=5s and other one is 5000000bytes (plus -stats is on by default). What about to speed up things and use 0 and 32 for max speed?
Also creating one thumb for test mkv file generates 12KB of spam in log. What about to use -v quiet/fatal/... to reduce it?
What about to use all three/four params not for thumbs generation only? It should be useful for "maybe" faster rewinding, start of file transcoding, cleaner logs, etc...
These are some from my spots in log when I started scanning my shared files.
By the way scanning is still in progress and DB size is already 750MB. What exactly is stored there? When file is updated, old record is updated or new one added with not deleting old one?
What about the DB access. Is it fully cached in memory or how queries are done (to disk or memory)? Useful to know how big heap memory should be set.
Thanks!
ExSport
The text was updated successfully, but these errors were encountered: