write_videofile results in "No such file or directory: OSError" on AWS Lambda instance

[matthewdowney]

Hi there,

I've been using moviepy with great results so far, but this issue has me confused. I'm trying to compile a few images to an mp4 video on an AWS lambda instance. I've run this code successfully on my CentOS server, and have only encountered this issue after migrating to AWS lambda.

My code is really simple:

render_path = "/tmp/%s.mp4" % id # AWS Lambda provides write access to the /tmp directory
image_clips = list()
for image in image_list:
    path = BytesIO(urllib.urlopen(image['url']).read())
    im = array(PilImage.open(path)) # load image as numpy array
    image_clips.append(ImageClip(im).set_duration(1.75)) 
print "Rendering video to %s" % render_path
clip = concatenate(image_clips)
clip.write_videofile(render_path, fps = 25, audio=False, threads=4)

This worked fine on CentOS. However, on AWS it gave me a huge stack trace that ended in

  File "/var/task/moviepy/video/VideoClip.py", line 339, in write_videofile
    ffmpeg_params=ffmpeg_params)
  File "/var/task/moviepy/video/io/ffmpeg_writer.py", line 200, in ffmpeg_write_video
    ffmpeg_params=ffmpeg_params)
  File "/var/task/moviepy/video/io/ffmpeg_writer.py", line 132, in __init__
    self.proc = sp.Popen(cmd, **popen_params)
  File "/usr/lib64/python2.7/subprocess.py", line 710, in __init__
    errread, errwrite)
  File "/usr/lib64/python2.7/subprocess.py", line 1335, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory

I assumed it was an ffmpeg issue, so I got an ffmpeg binary and put it in the lambda deploy root, which contains the main script & dependencies. Then, I added some code to copy ffmpeg to /tmp/ and chmod it to executable, for good measure, before setting it as the IMAGEIO_FFMPEG_EXE environmental variable to point to this path:

# Get the path of the ffmpeg binary that we've uploaded & copy it to tmp
ffmpeg_path = os.path.join(os.getcwd(), 'ffmpeg', 'ffmpeg')
shutil.copy(ffmpeg_path, "/tmp/ffmpeg")
ffmpeg_path = "/tmp/ffmpeg"
os.environ['IMAGEIO_FFMPEG_EXE'] = ffmpeg_path
print "Set ffmpeg path to %s" % ffmpeg_path
print "Contents of /tmp:"
for f in os.listdir('/tmp/'):
    print f # To make sure ffmpeg actually got copied over (it did)

# Make sure it's executable
st = os.stat(ffmpeg_path)
os.chmod(ffmpeg_path, st.st_mode | stat.S_IEXEC)

But I still encountered the same error :( At that point, I modified moviepy/video/io/ffmpeg_writer.py to display the command it was about to invoke, hoping to suss out an error that way:

#ffmpeg_writer.py
. . .
        print "cmd: %s\nparams: %s" % (cmd, popen_params) # I added this
        self.proc = sp.Popen(cmd, **popen_params) # This line was causing the error
. . . 

Which gave me:

cmd: ['ffmpeg', '-y', '-loglevel', 'error', '-f', 'rawvideo', '-vcodec', 'rawvideo', '-s', '1920x1080', '-pix_fmt', 'rgb24', '-r', '25.00', '-i', '-', '-an', '-vcodec', 'libx264', '-preset', 'medium', '-threads', '4', '-pix_fmt', 'yuv420p', u'/tmp/1018.mp4']
params: {'stdin': -1, 'stderr': -1, 'stdout': <open file '/dev/null', mode 'wb' at 0x7f7bea086c90>}

Is there anything here that sticks out as the culprit? To me the lack of STDIN appears out of the ordinary, but I don't know enough about the inner workings of moviepy to say. Any insight here is much appreciated!

For the sake of completeness, here's the full output & stack trace that I got from calling the lambda:

[WARNING] 2016-01-30T01:32:13.272Z Warning: could not find imageio's ffmpeg executable: [Errno 30] Read-only file system: '/var/tmp/.imageio'
START RequestId: 483fb2dd-c6f1-11e5-be3a-55de705bb1b2 Version: $LATEST
Set ffmpeg path to /tmp/ffmpeg
Contents of /tmp:
ffmpeg
Rendering video to /tmp/1018.mp4
[MoviePy] >>>> Building video /tmp/1018.mp4
[MoviePy] Writing video /tmp/1018.mp4
cmd: ['ffmpeg', '-y', '-loglevel', 'error', '-f', 'rawvideo', '-vcodec', 'rawvideo', '-s', '1920x1080', '-pix_fmt', 'rgb24', '-r', '25.00', '-i', '-', '-an', '-vcodec', 'libx264', '-preset', 'medium', '-threads', '4', '-pix_fmt', 'yuv420p', u'/tmp/1018.mp4']
params: {'stdin': -1, 'stderr': -1, 'stdout': <open file '/dev/null', mode 'wb' at 0x7f7bea086c90>}
[Errno 2] No such file or directory: OSError
Traceback (most recent call last):
  File "/var/task/lambda_function.py", line 64, in lambda_handler
    clip.write_videofile(render_path, fps = 25, audio=False, threads=4)
  File "<decorator-gen-51>", line 2, in write_videofile
  File "/var/task/moviepy/decorators.py", line 54, in requires_duration
    return f(clip, *a, **k)
  File "<decorator-gen-50>", line 2, in write_videofile
  File "/var/task/moviepy/decorators.py", line 137, in use_clip_fps_by_default
    return f(clip, *new_a, **new_kw)
  File "<decorator-gen-49>", line 2, in write_videofile
  File "/var/task/moviepy/decorators.py", line 22, in convert_masks_to_RGB
    return f(clip, *a, **k)
  File "/var/task/moviepy/video/VideoClip.py", line 339, in write_videofile
    ffmpeg_params=ffmpeg_params)
  File "/var/task/moviepy/video/io/ffmpeg_writer.py", line 200, in ffmpeg_write_video
    ffmpeg_params=ffmpeg_params)
  File "/var/task/moviepy/video/io/ffmpeg_writer.py", line 132, in __init__
    self.proc = sp.Popen(cmd, **popen_params)
  File "/usr/lib64/python2.7/subprocess.py", line 710, in __init__
    errread, errwrite)
  File "/usr/lib64/python2.7/subprocess.py", line 1335, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory

Duration: 453.22 ms Billed Duration: 500 ms     Memory Size: 1024 MB    Max Memory Used: 146 MB 

[dkarchmer]

See #238
Without that change, I don't believe it is possible to tell moviepy to use an already installed ffmpeg during run time.

[Zulko]

What about:

from moviepy.config import change_settings
change_settings({"FFMPEG_BINARY": "/path/to/ffmpeg"})

If this doesn't work, I merged @dkarchmer 's pull request so now you should be qble to set q FFMPEG_BINARY environment variable.

[matthewdowney]

@Zulko Perfect, that's exactly what I needed. You sir, are a gentleman and a scholar. Thank you very much!