Memory leak on long video (200 clips) #72

srameshr · 2020-09-20T15:33:10Z

Generating a video from clips with the kenBurns model of width 720 fails in between when running on an AWS machine with sufficient memory.

Using xvfb-run -s '-screen 0 1024x768x24 -ac +extension GLX +render -noreset' node index.js fails with an out of memory error if the video is 720P in width.

Error:

LLVM ERROR: Unable to allocate section memory

The text was updated successfully, but these errors were encountered:

mifi · 2020-09-23T12:04:02Z

Does it work on other AWS EC2 machines?

srameshr · 2020-09-23T12:52:41Z

Its does not work on any AWS or DO machines. I tried with a 32 GB RAM as well and increased the swap size too.
My local mac has a 16 GB memory and it works perfectly fine for any length of the video.

mifi · 2020-09-23T16:38:39Z

Which OS are you running? Could you try newest Ubuntu?

srameshr · 2020-09-23T17:15:25Z

I tried Fedora, Ubuntu & CentOS. Fails on all the distros.

mifi · 2020-09-24T09:46:16Z

how do you install it? do you use npm i -g editly? which version of node.js? headless machine or x11?

srameshr · 2020-09-24T09:53:11Z

I install it by running npm install as editly is a dependency in our project. Node 14.X and NPM 6.X. Here are the steps required to install all the dependencies required to run editly on AWS linux / centos distros. Ignore the indexes, they are off.

 	1 sudo yum update -y
	2  export LC_CTYPE="en_US.UTF-8"
	3  sudo yum install gcc-c++ make
        4  curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.34.0/install.sh | bash
	6  . ~/.nvm/nvm.sh
	7 nvm install node

       14  npm -v
       15  node- v
       cd ~
        sudo yum install mesa-libGL -y
	sudo yum install -y Xvfb
	sudo yum install -y mesa-dri-drivers
	 sudo yum install fontconfig freetype freetype-devel fontconfig-devel libstdc++ -y
   19  cd /usr/local/bin
   21  sudo mkdir ffmpeg
   22  cd /usr/local/bin/ffmpeg
   23  sudo wget https://johnvansickle.com/ffmpeg/releases/ffmpeg-release-amd64-static.tar.xz
   27  sudo tar -xf ffmpeg-release-amd64-static.tar.xz
   30  sudo ln -sfn /usr/local/bin/ffmpeg/ffmpeg-*-amd64-static/ffmpeg /usr/bin/ffmpeg
   31  sudo ln -sfn /usr/local/bin/ffmpeg/ffmpeg-*-amd64-static/ffprobe /usr/bin/ffprobe
   32  cd ~
  
   # read here https://aws.amazon.com/premiumsupport/knowledge-center/ec2-memory-swap-file/
   swapon -s

   sudo dd if=/dev/zero of=/swapfileone bs=128M count=256
   sudo chmod 600 /swapfileone
   sudo mkswap /swapfileone
   sudo swapon /swapfileone

srameshr · 2020-09-24T17:36:19Z

@mifi Tried it and it broke again! Fails around 73% of completion when the video width is 1280 PX and fails around 96% of completion when the video width trying to construct is 720 PX

 72% Done with transition, switching to next transitionFromClip (53)
createFrameSource image clip 54 layer 0
createFrameSource image-overlay clip 54 layer 1
createFrameSource news-title clip 54 layer 2
createFrameSource subtitle clip 54 layer 3
 72%  72%  72%  72%  72%  72%  72%  72%  72%  73%  73%

LLVM ERROR: Unable to allocate section memory
    at afterWriteDispatched (internal/stream_base_commons.js:156:25)
    at writeGeneric (internal/stream_base_commons.js:147:3)
    at Socket._writeGeneric (net.js:784:11)
    at Socket._write (net.js:796:8)
    at writeOrBuffer (_stream_writable.js:352:12)
    at Socket.Writable.write (_stream_writable.js:303:10)
    at /root/video-generator/node_modules/editly/index.js:509:49
    at new Promise (<anonymous>)

mifi · 2020-09-24T20:56:08Z

Could you share your json spec?

srameshr · 2020-09-25T01:40:02Z

const editlySpec = {
				outPath: path.resolve(__dirname, `../output/${fileName}`),
				width: parseInt(process.env.VIDEO_WIDTH, 10),
				height: parseInt(process.env.VIDEO_HEIGHT, 10),
				defaults: {
					duration: SLIDE_DURATION,
					transition: {
					duration: 0.5,
					name: 'fade',
					},
					layer: {
						fontPath: path.resolve(__dirname, `../assets/fonts/CircularStd-Bold.ttf`),
					},
	
				},
				audioFilePath: path.resolve(__dirname, `../assets/music/song.mp3`),
				
				clips: clips.map((clip, index) => {
					return {
						duration: SLIDE_DURATION,
						layers: [{
							type: 'image',
							path: path.resolve(__dirname, '../tmp/' + clip.path),
							zoomDirection: index % 2 === 0 ? 'in' : 'out',
						}, 
						{
							type: 'image-overlay',
							path: path.resolve(__dirname, '../assets/images/logo_transparent.png'),
							position: {  x: 0.99, y: 0.05, originX: 'right', originY: 'top' },
							width: 0.10,
							height: 0.10,
						}, 
						{
							type: 'news-title',
							text: clip.name
						}, {
							type: 'subtitle',
							text: clip.caption,
							backgroundColor: 'rgba(0,0,0,0.5)'
						}]
					}
				})
			}

mifi · 2020-09-25T09:48:35Z

How many clips is it? Could you post the output if you run with verbose: true ? Also are you running newest version of editly?

cat node_modules/editly/package.json

srameshr · 2020-09-25T13:44:36Z

"version": "0.6.3"
There are around 200 clips, with each clip duration being 10 seconds.

mifi · 2020-09-27T10:47:24Z

Ok, it does sound like a memory leak then. I've never tried such a long video with 200 clips. Gonna have to find where in the code it's leaking. I'm guessing it's not related to AWS, and it will probably fail on another computer too.

srameshr · 2020-09-27T12:00:33Z

It works on my mac with 500 clips.

mifi · 2020-09-27T12:02:02Z

Does it have the same amount of memory? And is it the same input/output resolution on the test on your mac?

srameshr · 2020-09-27T13:29:42Z

It has half the memory of my AWS machine. Input is same on both the machines.

mifi · 2020-10-05T11:08:38Z

Maybe the AWS machine doesn't have hardware acceleration, while the Mac has. Then maybe in node-canvas there is some memory leak only with software acceleration.

srameshr · 2020-10-14T16:09:14Z

@mifi Anything on this? Tried the latest version. Fails with the same error.

mifi · 2020-10-14T16:59:58Z

Could you run editly with the verbose flag and share the full "calculated" spec that gets printed in the beginning?

ariel-frischer · 2020-10-28T02:08:32Z

I might be having a similar issue, I was using c5d.2xlarge aws ec2 with elastic beanstalk (Docker running on 64bit Amazon Linux 2/3.0.3). I only had 15 clips but each clip has like a video + an overlayed alpha channel .mov with custom text animations. My issue is that it just stops on the 3rd to last clip, with no error message or anything, I'm assuming ec2/node could be running out of memory or there is a memory leak. Editly actually finished creating the file, but doesn't render the last 3 clips I input. I will test using a larger ec2 instance and relay my results.

ffmpeg version 3.4.8-0ubuntu0.2 Copyright (c) 2000-2020 the FFmpeg developers
  built with gcc 7 (Ubuntu 7.5.0-3ubuntu1~18.04)
  configuration: --prefix=/usr --extra-version=0ubuntu0.2 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --enable-gpl --disable-stripping --enable-avresample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librubberband --enable-librsvg --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-omx --enable-openal --enable-opengl --enable-sdl2 --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libopencv --enable-libx264 --enable-shared
  libavutil      55. 78.100 / 55. 78.100
  libavcodec     57.107.100 / 57.107.100
  libavformat    57. 83.100 / 57. 83.100
  libavdevice    57. 10.100 / 57. 10.100
  libavfilter     6.107.100 /  6.107.100
  libavresample   3.  7.  0 /  3.  7.  0
  libswscale      4.  8.100 /  4.  8.100
  libswresample   2.  9.100 /  2.  9.100
  libpostproc    54.  7.100 / 54.  7.100

-i myVideo.mp4 -t 8.828571428571427 -vf fps=30,scale=(iw*sar)*max(1920/(iw*sar)\,1080/ih):ih*max(1920/(iw*sar)\,1080/ih),crop=1920:1080 -map v:0 -vcodec rawvideo -pix_fmt rgba -f image2pipe -

ffmpeg cli:

ffmpeg -f rawvideo -vcodec rawvideo -pix_fmt rgba -s 1920x1080 -r 30 -i - -stream_loop -1 -i soundTrack.mp3 -shortest -map 0:v:0 -map 1:a:0 -acodec aac -b:a 128k -vf format=yuv420p -vcodec libx264 -profile:v high -preset:v medium -crf 18 -movflags faststart -y /backend/tmp/vayn_in_home_article_INITIATE_AI_d8becfe3-485b-4cb2-ac7f-59e080af1db8.mp4

ariel-frischer · 2020-10-28T02:26:08Z

Update: Improving ec2 instance to c5d.9xlarge seemed to have no effect, same issue occurs. If I run this same editly render on my local windows machine it works :/ no error message to figure out how to fix is very stressful.

ariel-frischer · 2020-10-28T02:48:21Z

I noticed in my verbose logs ffmpeg exits (0 code), but then tries to write another frame, then cleanup, then closes the videos it was supposed to write... You can see it here:

srameshr · 2020-10-28T05:21:53Z

@ariel-frischer Are those stock footages from pexel.com?

ariel-frischer · 2020-10-28T05:23:32Z

@ariel-frischer Are those stock footages from pexel.com?

Yea mostly... why?

srameshr · 2020-10-28T05:25:13Z

@ariel-frischer I am building a tool for YouTubers that picks up videos from pexels.com and was just curious about how others are using this tool. Are you sharing what you are working on in public?

ariel-frischer · 2020-10-28T05:28:16Z

@ariel-frischer I am building a tool for YouTubers that picks up videos from pexels.com and was just curious about how others are using this tool. Are you sharing what you are working on in public?

I'm making a text to video platform, I guess your right I could hide the urls, didn't see much of the harm though.

srameshr · 2020-10-28T05:30:52Z

Sent you a mail.

ariel-frischer · 2020-10-28T05:34:34Z

Sent you a mail.

? I didn't seem to get anything...

srameshr · 2020-10-28T05:38:31Z

The one with gmail.com or .edu? I sent it to gmail.

ariel-frischer · 2020-10-28T05:39:29Z

The one with gmail.com or .edu? I sent it to gmail.

Gmail is better I just updated it.

ariel-frischer · 2020-10-28T07:38:46Z

Update: Somehow my editly version was not on the latest version 💯 things should be working now as expected.

ariel-frischer · 2020-10-28T23:09:09Z

Ok, I've figured out the issue, I noticed editly just stopped at around the time my audio file finished, even though loopAudio was set to true. The actual issue was that my FFMPEG version 3.4.8-0ubuntu0.2 was not high enough so there could have been some bugs with stream loop -1. My new ffmpeg version 4.3.1 seemed to solve this issue, all the clips are now rendering correctly with looped audio.

mifi added the Cloud label Sep 23, 2020

srameshr mentioned this issue Sep 25, 2020

Error: gl returned null (missing dependencies) #33

Open

mifi added the bug Something isn't working label Sep 27, 2020

mifi changed the title ~~LLVM ERROR: Unable to allocate section memory - On AWS M5.X large machines with~~ Memory leak on long video (200 clips) Sep 27, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Memory leak on long video (200 clips) #72

Memory leak on long video (200 clips) #72

srameshr commented Sep 20, 2020 •

edited

Loading

mifi commented Sep 23, 2020

srameshr commented Sep 23, 2020 •

edited

Loading

mifi commented Sep 23, 2020

srameshr commented Sep 23, 2020

mifi commented Sep 24, 2020

srameshr commented Sep 24, 2020 •

edited

Loading

srameshr commented Sep 24, 2020

mifi commented Sep 24, 2020

srameshr commented Sep 25, 2020

mifi commented Sep 25, 2020 •

edited

Loading

srameshr commented Sep 25, 2020

mifi commented Sep 27, 2020

srameshr commented Sep 27, 2020

mifi commented Sep 27, 2020

srameshr commented Sep 27, 2020 •

edited

Loading

mifi commented Oct 5, 2020

srameshr commented Oct 14, 2020

mifi commented Oct 14, 2020

ariel-frischer commented Oct 28, 2020 •

edited

Loading

ariel-frischer commented Oct 28, 2020 •

edited

Loading

ariel-frischer commented Oct 28, 2020

srameshr commented Oct 28, 2020

ariel-frischer commented Oct 28, 2020

srameshr commented Oct 28, 2020

ariel-frischer commented Oct 28, 2020

srameshr commented Oct 28, 2020

ariel-frischer commented Oct 28, 2020

srameshr commented Oct 28, 2020

ariel-frischer commented Oct 28, 2020

ariel-frischer commented Oct 28, 2020

ariel-frischer commented Oct 28, 2020

Memory leak on long video (200 clips) #72

Memory leak on long video (200 clips) #72

Comments

srameshr commented Sep 20, 2020 • edited Loading

mifi commented Sep 23, 2020

srameshr commented Sep 23, 2020 • edited Loading

mifi commented Sep 23, 2020

srameshr commented Sep 23, 2020

mifi commented Sep 24, 2020

srameshr commented Sep 24, 2020 • edited Loading

srameshr commented Sep 24, 2020

mifi commented Sep 24, 2020

srameshr commented Sep 25, 2020

mifi commented Sep 25, 2020 • edited Loading

srameshr commented Sep 25, 2020

mifi commented Sep 27, 2020

srameshr commented Sep 27, 2020

mifi commented Sep 27, 2020

srameshr commented Sep 27, 2020 • edited Loading

mifi commented Oct 5, 2020

srameshr commented Oct 14, 2020

mifi commented Oct 14, 2020

ariel-frischer commented Oct 28, 2020 • edited Loading

ariel-frischer commented Oct 28, 2020 • edited Loading

ariel-frischer commented Oct 28, 2020

srameshr commented Oct 28, 2020

ariel-frischer commented Oct 28, 2020

srameshr commented Oct 28, 2020

ariel-frischer commented Oct 28, 2020

srameshr commented Oct 28, 2020

ariel-frischer commented Oct 28, 2020

srameshr commented Oct 28, 2020

ariel-frischer commented Oct 28, 2020

ariel-frischer commented Oct 28, 2020

ariel-frischer commented Oct 28, 2020

srameshr commented Sep 20, 2020 •

edited

Loading

srameshr commented Sep 23, 2020 •

edited

Loading

srameshr commented Sep 24, 2020 •

edited

Loading

mifi commented Sep 25, 2020 •

edited

Loading

srameshr commented Sep 27, 2020 •

edited

Loading

ariel-frischer commented Oct 28, 2020 •

edited

Loading

ariel-frischer commented Oct 28, 2020 •

edited

Loading