playout stops every few days #100
Comments
|
This issues relate to the engine. I need the ffplayout.yml and full logs, as Attachments, or code blocks. Logs also from day before. |
|
Loggin on the day was not set to debug so only shows the error i posted. ffplayout.yml: helptext: Sometimes it can happen, that a file is corrupt but still playable,
this can produce an streaming error over all following files. The only way
in this case is, to stop ffplayout and start it again here we only say it
can stop, the starting process is in your hand. Best way is a systemd serivce
on linux. 'stop_threshold' stop ffplayout, if it is async in time above this
value.
stop_on_error: false
stop_threshold: 11
mail:
helptext: Send error messages to email address, like missing playlist; unvalid
json format; missing clip path. Leave recipient blank, if you don't need this.
'mail_level' can be WARNING or ERROR.
subject: Playout Error
smpt_server: mail.example.org
smpt_port: 587
sender_addr: ffplayout@example.org
sender_pass: abc123
recipient: null
mail_level: ERROR
logging:
helptext: Logging to file, if 'log_to_file' False log to console. 'backup_count'
says how long log files will be saved in days. Path to /var/log/ only if you
run this program as deamon. 'log_level' can be DEBUG, INFO, WARNING, ERROR.
'ffmpeg_level' can be INFO, WARNING, ERROR.
log_to_file: true
backup_count: 7
log_path: /var/log/ffplayout/
log_level: DEBUG
ffmpeg_level: ERROR
processing:
helptext: Default processing, for all clips that they get prepared in that way,
so the output is unique. 'aspect' must be a float number. 'logo' is only used
if the path exist. 'logo_scale' scale the logo to target size, leave it blank
when no scaling is needed, format is 'number:number', for example '100:-1'
for proportional scaling. With 'logo_opacity' logo can become transparent.
With 'logo_filter' 'overlay=W-w-12:12' you can modify the logo position. With
'use_loudnorm' you can activate single pass EBU R128 loudness normalization.
'loud_*' can adjust the loudnorm filter. 'output_count' sets the outputs for
the filtering, > 1 gives the option to use the same filters for multiple outputs.
This outputs can be taken in 'ffmpeg_param', names will be vout2, vout3; aout2,
aout2 etc.
width: 1024
height: 576
aspect: 1.778
fps: 25
add_logo: true
logo: /media/video/logo/logo.png
logo_scale: null
logo_opacity: 0.7
logo_filter: overlay=W-w-30:30
add_loudnorm: true
loud_I: -18
loud_TP: -1.5
loud_LRA: 11
output_count: 1
playlist:
helptext: Set 'playlist_mode' to 'False' if you want to play clips from the 'storage'
section. Put only the root path here, for example '/playlists' subfolders
are readed by the script. Subfolders needs this structur '/playlists/2018/01'
(/playlists/year/month). 'day_start' means at which time the playlist should
start, leave day_start blank when playlist should always start at the begin.
'length' represent the target length from playlist, when is blank real length
will not consider.
playlist_mode: true
path: /opt/playlists
day_start: 00:00:00
length: '24:00:00'
storage:
helptext: Play ordered or ramdomly files from path. 'filler_clip' is for fill
the end to reach 24 hours, it will loop when is necessary. 'extensions' search
only files with this extension. Set 'shuffle' to 'True' to pick files randomly.
path: /media/video/video files
filler_clip: /media/video/loop/
extensions:
- .mp4
- .mkv
shuffle: true
text:
helptext: Overlay text in combination with libzmq for remote text manipulation.
On windows fontfile path need to be like this 'C\:/WINDOWS/fonts/DejaVuSans.ttf'.
In a standard environment the filter drawtext node is Parsed_drawtext_2. 'over_pre'
if True text will be overlay in pre processing. Continue same text over multiple
files is in that mode not possible. 'text_from_filename' activate the extraction
from text of a filename. With 'style' you can define the drawtext parameters
like position, color, etc. Post Text over API will override this. With 'regex'
you can format file names, to get a title from it.
add_text: true
over_pre: false
bind_address: 127.0.0.1:5555
fontfile: /usr/share/fonts/truetype/dejavu/DejaVuSans.ttf
text_from_filename: false
style: x=(w-tw)/2:y=(h-line_h)*0.9:fontsize=24:fontcolor=#ffffff:box=1:boxcolor=#000000:boxborderw=4
regex: ^(.*)_
out: |
Then maybe is better that we wait, until the error comes again. Then I need both logs, from day before and current day. For simulating day transitions you could also play with time_machine script. But you need to have a bit knowledge about the app structure. |
|
I found this. if client does playlist 24 hours and clicks the loop function (where t auto completes playlist) then it does not give issue (so far) the other weird parts are
|
|
happened again day before log: New day log:
log after restart |
|
is this issue resolved with installer v3.2.0? |
|
Until I can reproduce or validate the error, I can not fix it. More up I ask for both full logs, not only a snipped. Private data you can hide in it, but I need the rest. And when you have not a really good reason for it, then let |
|
Ok, I found the bug, you don't have to send the logs any more. |
|
Logs show nothing out of the norm much The one thing i noted not sure it it changes things. During the day we added files to the playlist to make it a full 24 hours and saved it. its why it played till mid night no issue. nor did the loop file play in absence of playlist Could it be video file issue that does not play? or character in file name eg the period in the file below? [2021-06-19 23:59:23,254] [WARNING] Clip length is not in time, new duration is: 36.75 |
|
Bug should be fixed in version v3.4.0 and updated in installer. |
|
To avoid me updating incorrectly and creating bugs that waste time. I want to be sure update done correctly to avoid creating bugs that do not need to be there |
|
Normally you can delete the folder (/opt/ffplayout_engine) and run the updated installer. |
|
ok so ONLY engine in this case right? if it were GUI would be its folder in /opt then as well right? |
|
No the GUI is under /var/www. Is not an official upgrade method, but I think it should work, without guaranty. |
|
what is the BEST way to upgrade to avoid creating bugs |
|
update applied by removing /opt/ffplayout_engine and then downloading installer 3.2.0 zip and installing. |
|
playout and streaming stoped again [2021-07-03 23:56:53,261] [ERROR] Playlist not long enough! playlist was short of 24 hours but at 00:00:00 it did not start new playlist and restart streaming output until i had to manually click the YELLOW reload icon then it started again |
|
I was not able to reproduce it. But maybe you can help me debugging it. You can clone the master repo to the machine where you get this error.
|
|
I am waiting to confirm if again.
May be an idea to hard code the system to fill with black if loop file is missing. Then the new days playlist load will work to engine (as currently engine stops so when new list loaded engine does nothing) |
|
No, normally all these behaviours should be catched. If not, the fault is somewhere else, we have to find out now. |
|
Ok seems the issue before was happening because the loop file specified the file name and was set to shuffle. with the loop file specified and shuffle unticked. it has not presented this issue again yet again. so may be related to issue with loop file triggering a stop of everything |
|
I really don't think so. To verify I have also try to reproduce it, but I was not able. The shuffle mode also affect only the folder mode, not playlist mode. The easiest would be, when you follow the steps from before. Because your last error was on 2021-07-03, it is better to use that situation as a starting point. I guess with loop file you mean filler_clip? |
|
Ok i will try clone the machine for testing (i dont want to risk breaking a test system we know is not developing these bugs anymore) Yes loop = filler_clip |
Yes that would be nice. |
|
also what CPU and RAM is best for test machine so i can be sure its not part of the issues. and here is the last 24 hour ram use as well However i started to see this error daily which breaks output briefly. [2021-07-14 06:42:16,160] av_interleaved_write_frame(): Cannot allocate memory running UDP for local testing rather than pushing RTMP |
|
I would say that the system specs are enough, but I have not much experience with UDP. When you don't need a output you can use the null mode, as I wrote before. Or you try multicast like: |
|
This showed on decoder logs today [2021-07-19 00:29:04,901] av_interleaved_write_frame(): Broken pipe |
|
For issues related to ffmpeg I can not give support. |
|
the bug was back now log was [2021-07-20 23:51:34,945] [ERROR] Playlist not long enough! Decode error log: The issue was the loop video (filler) did not play to fill in the short |
i realize that but i am just making you aware of the bug found as it may have bearings on future faults being caused |
|
This morning again Decoder log: and the GUI logs up and does not load well after loading decoder log (that takes long) have to reload whole page Playout log: Encode log error: when the pipe broke streaming output and playout stopped and no filler played either. system has been up and running 20 days now and seems to have this error more and more the longer it runs. ram use below CPU use below You can also see lan use when it dropped and stopped streaming (red) and then GREEN spikes when it was loading logs on gui slowly |
|
I think at the moment we mixing different issues, that gets complicated for later tracking. Let's try to stay fist on the main issue. You should also let: stop_on_error: true
stop_threshold: 11on default. Most time it will not help you, when you change this settings.Your playout log show very high delta values, that means that something is not right with your file, or playlist, or ffmpeg can not handle the file correctly. |
|
these last few days are a follow on on the original issue. ok that may be part of it mine was set The high delta might that be caused by files with differing frame rate and resolutions (i noted my test video files seem to be a mixed bag of formats) |
|
Different frame rates and resolutions doesn't impact duration and so on has no influence on the delta value. |
|
I applied the above. Playout logs showed: then it showed it started the stream output, then the internal lossless encode then again [2021-07-26 06:42:11,275] [DEBUG] delta: 868.997552 and it goes in a loop again doing all the same over and over The only way i could restart it was to untick Stop_on_error and then restart playout. (it was 10 min before i had to force it to restart) usually when it stops i see CPU drop. but during this CPU usage stayed at the usual amount it is when running normally. system up for 24 days straight. not sure if things may be creeping in with system running so long possibly hence why does not show up on localized simulation tests just noted DECODER log files are 117MiB currently and last logs some files around 50MiB |
Just for the record: this errors are happen when ffmpeg tries to decode AV1 with a bad decoder. I would recommend to compile ffmpeg with libdav1d. |
|
HI AV1 codecs not used at all. I did a check to be sure. also finding server "leaks" memory and either freezes or needs to be rebooted every 30-40 days or so.
|
Maybe now not anymore, but here it was ;-)
I have not see it before, but I will pay attention to it the next weeks. When you see it again, please try to figure out, which component will produce this: the python script it self, or one of the both ffmpeg instances. |
|
I'm following this keenly as I would like to restart using this system. |
|
Since some weeks I have upgrade our production system to the latest version. I see also that the engine restarted sometimes, but in my case there is a error with a specific video file. It has a different duration, so it hits the threshold. A memory leak I could not see until now. |
|
HI, has this been resolved. Soak testing server shows consistant issues on old ver every 45 days and locks machine up still. For best testing whats best way to update the testing server other and full new install? |
|
To update you can pass |
|
please can you explain how to correctly fully use the --update command. |
|
The problem is, I don't know, what are your current version, so is difficult to guarantee that everything works like it should. The update process will not update the ffplayout.yml config. I would recommend to run the update process on a test system and see if everything goes smooth. When you only need a newer engine, you can replace that one alone. Here was the same situation. But I do not recommend to use the version from master, this will have many changes in the next days. |
|
installer 3.2.0 was used and updated manually last by removing /opt/ffplayout_engine and runing the zip what is the correct use or update pricess |
Yes that you can do so. Only take care that you create the virtual environment again.
This question I don't understand. |
|
please can you clarify the correct way to use the --update command. want to update to latest version to see that all bugs gone. 45 days again and seeing.
I suspected before that low resources may have contributed to this so i increases resources, but no change. confirming not a resource issue |
|
I would recommend:
|
|
will give it a run and revert. Thanks a million any news on timeline for GUI for live yet? |
|
I need to figure out how I will continue with the GUI. It uses Nuxt 2.* with bootstrap 4, and until now there is no way to upgrade to Nuxt 3 and bootstrap 5. For adding new features, it makes no sense, and getting better performance is not possible. I have to wait until they support a clean upgrade path, or I have to rewrite the hole page. But when I have to rewrite, I'm not sure if I want to still stick on Nuxt, or I use something else, like svelte. |
|
Yeah make sense to upgrade rather but makes alot more work but gives more flexibility and options (without the gun to your head waiting for them) |
|
I'm closing here. Feel free to open a new issue when something is happen. |
Installer 3.1.0 installed
playout works well.
BUT 2x now have see it has issues loading playlist. basically every 2 days now.
Playout streaming output STOPS. BUT GUI shows play GREEN
logs show
[2021-06-13 00:00:00,125] [ERROR] Playlist not long enough!
[2021-06-13 00:00:00,135] [ERROR] Playlist is not long enough: 86338.86 seconds needed
[2021-06-13 00:00:00,135] [ERROR] File not exist: None
I check the playlist it exists and is 24 hours long (slightly over)
also the loop file does not play when playlist is too short or missing.
I have the file set to the mapped video drive
/media/povideo/loop/loop.mp4
(.mp4 .mkv SHUFFLE)
The video file is valid and read write access is the same as the rest of the video files.
Also logging not logging for encode but all the rest logs.
Tonight when the issue showed i used the YELLOW Restart playout 3x
On the 3rd time the playlist file started playing (and the error no longer showed in log)
CPU remains constant at 40%
RAM:
Total: 3.9GiB
Used: 2.2GiB
Free: 129.5MiB
Cached: 968.0MiB
Am posting here as its the installer used. not sure where the issue may be api, engine etc
also not sure if this is a known bug fixed with latest installer. Rather than assume i posted here
The text was updated successfully, but these errors were encountered: