Memory usage issue #5
Comments
|
ffmpeg isn't supported anymore, please use avconv. make sure you are processing frames in real-time, the capture just piles up frames if they aren't processed fast enough. |
|
I must still have something incorrect... its memory usage still vastly surpassed the 1GB mark. This is the command that I am using to capture my live stream: Right now the recordings are being saved as one minute clips on ram disk for later processing. For testing I have them immediately deleted with a cron every minute until I can get this memory issue under control. Am I leaving something out? |
|
I have tried many different iterations of the command that I posted the other day but I still cannot get this to stop the continuous growth. What do you mean "processing frames in real-time"? |
|
Unfortunately I can't provide a solution either -- I can just leave a "mee too" here and share some experience. I had the same problem: memory usage just grew over time. During testing, I also tried sending the data to /dev/null which did work for me. So my conclusion was that the encoder was too slow (for whatever reason -- it wasn't maxing out the CPUs). If sending to /dev/null exhibits the same problem for you, it might either be a different issue or a performance problem as well (but even without an encoder). If the problem itself (encoding is delayed somewhere) cannot be solved, maybe it would be useful to drop frames in bmdcapture. I made some attempts to do so, but failed to drop video and audio frames properly, which led to problems in the encoder (waiting for the related audio frame which would never be sent because it was dropped...). I have no idea how easy or hard it would be to implement that. I don't know which libav version you are using, but it might be possible, that some dev version leaked memory (didn't check), so using a release version might be a good idea. (In my setup I just had to do live streaming for a specific time frame; just upping the bmdcapture memory limit worked for me because of that) |
|
On 23/05/12 12:06, Christian Hoffmann wrote:
I'll try to implement something more graceful but IF the problem is slow
libav is testrun on valgrind in our regression checking system, so it is
lu Luca Barbato |
|
The cards being used are the DeckLink SDI and the DeckLink Duo. I don't think that the machine is having a problem processing frames because it is a server class box with a quad core and 6GB of ram and 1TB of HD space. |
This is bmdcapture running for about 20 hours with the following command: It appears that the memory issue is still there. |
|
On 24/05/12 07:21, thenexxuz wrote:
I'd need a valgrind run, looks like there is a memory leak. lu Luca Barbato |
|
I used the following command: The output is listed here http://pastebin.com/u1kCbAH8 |
|
I have the same problem with latest bmdcapture, libav 9_beta1 and BlackMagic SDK961. Still not exist solution for this problem ? |
|
Coworkers have a workaround... cron restarts the capture process after about 9 to 13 hours. We have safeguards in place in case if it wants to restart during a scheduled time then it waits... we do lose about 3 minutes for the restart. |
|
Any news regarding this issue? |
|
I'm trying my best to track possible memory leaks and plug what valgrind reports. I also integrated in capture support in libav (check my local bmd branch and libbmd) to compare two different implementations of the same concept. I'm running some tests as this week to compare the result. Could you please report me exactly what you are doing when the problem appears? if you just write to /dev/null what happens? |
ghost
assigned 
|
I have the same issue. I compiled bmdtool with libavformat from avconv and ffmpeg. It is the same, the memory usage increases continuously. It is then impossible to use bmdtool for 24/7. After 2 hours, |
|
ffmpeg is not supported. If you are capturing use my bmd branch from libav On Fri, Apr 19, 2013 at 11:29 AM, ralawa notifications@github.com wrote:
|
|
Where can we found your "bmd branch from libav tree"? Thanks. |
|
https://github.com/lu-zero/libav/tree/bmd On Fri, Apr 19, 2013 at 2:03 PM, ralawa notifications@github.com wrote:
|
|
nut is the problem, for me works avi instead nut without memory leaking 2013/4/19 Luca Barbato notifications@github.com
|
|
On Fri, Apr 19, 2013 at 3:21 PM, digitalx notifications@github.com wrote:
I'll triplecheck that, thanks for the pointer. |
|
Yes, bmdcapture works fine with avi. |
|
On Tue, Apr 23, 2013 at 3:17 PM, ralawa notifications@github.com wrote:
Avi is horrible, I'll try to fix nut later if the problem is there... may I |
|
Yes, I'm volonteer. |
|
I can confirm ralawas test results . With nut memory usage is increasing. With avi memory usage is low and stable but audio-/videosync issue starts. I would also test new variants. |
|
Could you please use massif (part of the valgrind tool collection) and provide the ms_print results? |
|
I've started valgrind. How many hours should I run the tool? |
|
Once it starts filling up
|
|
here is the link to my result. |
|
and here the result of a 12 hour capture process. |
|
Hi guys - i seem to have the same issue with memory usage. Using latest Ubuntu 12.04 LTS with latest libav compiled from git and latest Blackmagic driver+sdk (9.7.1) with Blackmagic Intensity Pro. Memory usage is growing - after about 28hrs process bmdcapture dies with Out of memory and the following syslog message. Machine has 4GB ram and its an i5 processor |
|
The issue is that nut is a raw format and takes up a lot of space. If your input is something like ffmpeg, it will decode the file very quickly and fill up your pipe or buffer space (RAM). bmdplay is a simple encoder which isn't using much CPU and designed to play back in realtime. The result is a backlog of data in the buffer since the decoder is running really fast. For ffmpeg you can fix this by using the -re flag to do realtime encoding. This prevents buffer fill-up by only encoding at real time. |
|
Im have tried both ffmpeg and avconv using the -re option, and it still leaks memory. What I am running is a Decklink Quad >> bmdcapture >> avconv >> RTMP out. With nut, I get a memory leak, and with avi or asf I get an AV Sync issue. |
|
Sorry, I was referring to bmdplay. For bmdcapture I had to eliminate the On Thu, Nov 6, 2014 at 4:24 PM, Klindg notifications@github.com wrote:
|
|
Hey Rokclimb15... Currently this is what I am running... ./bmdcapture -C 3 -m 9 -V 4 -A 2 -F nut -f pipe:1 | avconv -y -re -r 29.97 -i - -codec:v libx264 -preset veryfast -bf 0 -qmax 24 -g 60 -b:video 4000k -aspect 16:9 -s 1280x720 -codec:a libfdk_aac -b:audio 128k -f flv rtmp://localhost/live/live Your suggesting... ./bmdcapture -C 3 -m 9 -V 4 -A 2 -F nut -f pipe:1 | avconv -y -r 29.97 -i - -codec:v libx264 -preset veryfast -bf 0 -qmax 24 -g 60 -b:video 4000k -aspect 16:9 -s 1280x720 -codec:a libfdk_aac -b:audio 128k -buffersize 4000000000 -f flv rtmp://localhost/live/live ?? This will need to run 24/7. Thanks |
|
Yes, drop the -re since it can underflow the buffer and crash. I don't do On Thu, Nov 6, 2014 at 4:36 PM, Klindg notifications@github.com wrote:
|
|
You can compile latest ffmpeg master with decklink |
|
Rokclimb15, Up and running minus the -re and with a bufsize set. Thanks for the input, Ill see if this works for me. Amnonbb, I have a couple of test boxes here to work with, and did read something about that elsewhere. Thanks for the link, Ill get that spun up on one of my other test servers. Appreciate it guys, Ill update with my results. |
|
Hi Rokclimb15, Unfortunately, this caused a memory leak and an AVSync issue. Amnonbb, I have not had a chance to try ffmpeg with decklink yet. I was trying to compile ffmpeg with decklink support, via https://trac.ffmpeg.org/wiki/CompilationGuide/Centos, and also https://www.ffmpeg.org/ffmpeg-devices.html section 3.21, but am uncertain as to what I need to add with --extra-cflags and --extra-ldflags. cflags should point to decklinkSDK include directory I believe, but Im unsure as to ldflags... I may be over thinking/going down the wrong path here, and I dont want to fill up this thread with non BMDtools related questions, but would it be possible to private chat about this, or continue this via Email? Very interested as you say you have a working solution thats running 24/7. Thanks! |
|
No problem. |
|
Pass to bmdcapture |
|
Meanwhile I'm adding encoding support in bmdcapture so you might not need to pass to avconv for the simplest usage patterns =) |
|
Hi Lu-zero, Sounds good on the encoding support. Im in the middle of reconfiguring the server I am testing with. Ill let you know how -o strict=experimental:syncpoints=none goes once I get that done. Thanks! |
|
Hi Lu-zero, Im unable to get this to work. Not sure if I missed something, but here is what I am getting... ./bmdcapture -C 3 -m 9 -I 1 -F nut -o strict=experimental:syncpoints=none -f test.dat Just returns the -h output. using the -I option returns /bmdcapture: invalid option -- 'I' I can still pull the SDI feed without the -o option via... I think I may be missing something here. I pulled down the latest bmdtools via... Of course, trying to pipe it to avconv gives the pipe:: Invalid data found when processing input error. Thanks! |
|
are you sure
and
I forgot a |
|
Thanks for the quick response Lu-zero! All set now, up and running. Will update the thread with my results. Again, Appreciate the quick response. |
|
Now it should work as expected. please tell me if it does not. Sadly I have |
|
Hi lu-zero, I am using your fix strict=experimental:syncpoints=none. In a single pipe this fix completly the memory leak. But now I am trying to tee the pipe, see: Only the first pipe is being filled (the tee pipe), the second pipe seems to not receive any data. PS: without the syncpoints option the pipe works normally. |
|
FFmpeg is not supported to begin with, the line reported seems to missing a I hope to have a little time to check if it happens with avconv. Sadly I hadn't had time to push some updates to libbmd and bmdtools I was planning to do this December due some emergencies =/ I guess I should start populating the wiki with examples and scripts on how to do something like that. |
|
I am using the latest avconv from git (not ffmpeg, it is just a naming convention in my bash functions, sry). |
|
just reading it again: tee is getting is stdout captured twice and that won't work. in order to do something like that you should do: I tested it locally and works as expected. |
|
Hi lu-zero I'm using bmdcapture to make multiple SDI stream recordings of uncompressed audio/video and I'm seeing a memory leak. When the tool exits it would appear that a copy of the full recording exists in cache. When I run the tool a subsequent time (with the same parameters) then the memory is freed when avio_open() is being called. That said, in the event that the filename parameter does change in a subsequent execution of the tool then the cached file is not overwritten and yet more cache gets eaten up by the tool. Has anyone else noticed this issue? Is there a possible solution to this memory leak? --thanks |
|
Linux uses a cache to speed up the io On Wed, Jan 28, 2015 at 10:19 PM, IDCmbray notifications@github.com wrote:
|
|
OK, I was afraid that was the case. Because of the way my scripts work my I end up with a full 16GB of RAM being used up as cache which causes problems in the processing that must be done on the SDI recordings. I added a hack into my script where I explicitly clear the cache (echo 3 > /proc/sys/vm/drop_caches) between recordings thus freeing up the memory required for the processing but I was hoping for a "cleaner" solution to this problem. --thanks for the reply :) |
|
You could use a swapfile as big as twice your ram or tune the the vm to On Thu, Jan 29, 2015 at 7:54 PM, IDCmbray notifications@github.com wrote:
|
|
Hm, why should specific options (for O_DIRECT or something) or special tuning be necessary at all? Filling the free RAM with buffer caches should not have any impact on performance or RAM accessible to other processes. @IDCmbray, what kind of processing problems are you seeing? If some tool bases its mode of operation on the "cached" memory value, this sounds wrong to me. If it just tries to allocate memory, Linux should flush these buffer caches as-needed without you having to worry about or am I missing something? |
|
@lu-zero That would be awesome and greatly appreciated! Thanks! @hoffie When re-reading my previous post I realized it may be a little (and by that I mean a lot) misleading. The system I have implemented is for testing a hardware encoder. Basically I'm using a Decklink Duo to output a test pattern over one SDI port of the Decklink, passing that through a hardware encoder who's output is then looped back to the Decklink where I make the recording. The issue I'm seeing is that when the RAM gets eaten up by cache the bmdplay tool begins to fail by either not playing audio and/or playing choppy video (presumably because the pts are not being properly processed). I believe the playback becomes choppy because the playback is always the first process to start on the machine it is therefore always the oldest data and therefore first to be overwritten when new memory is required. The bad playback resulting from the overfilled RAM is what affects my processing. It should be noted that on a typical run through a test sequence I would be obtaining upwards or 1000 recordings of raw SDI each of at least 30 seconds... hence 16 GB of RAM is quickly filled. |
|
With the latest version of bmdcapture and the recommended -o option, there seems no memory leak to be anymore. I have used nut container with no problem so far. |
|
Could you please paste how are you using it? On Mon, Feb 23, 2015 at 11:43 AM, felixdreinulldrei <
|
|
for longtime testing i'm using this line with two blackmagic studio2 cards: |
I am trying to use bmdcapture to pipe to an instance of ffmpeg segmented streaming for 24/7 recording but I keep running into the same issue, running out of memory. The server it is running on has 6GB of ram but after about 2 and a half days the ram fills up completely and bmdcapture dies then the ram is cleared. I have confirmed that piping bmdcapture to /dev/null still fills up the ram.
Are there any suggestions?
--- Want to back this issue? **[Post a bounty on it!](https://www.bountysource.com/issues/942865-memory-usage-issue?utm_campaign=plugin&utm_content=tracker%2F256764&utm_medium=issues&utm_source=github)** We accept bounties via [Bountysource](https://www.bountysource.com/?utm_campaign=plugin&utm_content=tracker%2F256764&utm_medium=issues&utm_source=github).The text was updated successfully, but these errors were encountered: