ElasticTranscoder.createJob not working? #455

Closed
grvhi opened this Issue Jan 3, 2015 · 5 comments

Comments

Projects
None yet
4 participants
@grvhi

grvhi commented Jan 3, 2015

I'm using Lambda to trigger video transcoding tasks when a video file is uploaded to an S3 bucket. My Lambda function works up to the point of triggering the ET job. This is my Lambda function:

var AWS = require('aws-sdk');
var s3 = new AWS.S3({apiVersion: '2006-03-01'});
var eltr = new AWS.ElasticTranscoder({
    apiVersion: '2012-09-25',
    region: 'us-east-1'
});
var pipelineId = 'MYPIPELINEID';
var webPreset = '1351620000001-100070';

exports.handler = function(event, context) {
   var bucket = event.Records[0].s3.bucket.name;
   var key = event.Records[0].s3.object.key;
   s3.getObject({
       Bucket:bucket,
       Key:key
     },
      function (err, data) {
        if (err) {
           console.log('error getting object ' + key + ' from bucket ' + bucket +
               '. Make sure they exist and your bucket is in the same region as this function.');
           context.done('ERROR', 'error getting file' + err);
        }
        else {
          if (data.Metadata.filetype == 'video') {
            console.log('Found new video: ' + key + ', sending to ET');
            sendVideoToET(key);
          } else {
            console.log('Upload ' + key + 'was not video');
            console.log(JSON.stringify(data.Metadata));
          }
           context.done(null,'');
        }
      }
   );
};

function sendVideoToET(key){
  console.log('Sending ' + key + ' to ET');
  var params = {
    PipelineId: pipelineId,
    OutputKeyPrefix: '',
    Input: {
      Key: key,
      FrameRate: 'auto',
      Resolution: 'auto',
      AspectRatio: 'auto',
      Interlaced: 'auto',
      Container: 'auto'
    },
    Output: {
      Key: key + '__mp4',
      ThumbnailPattern: 'thumbs-{count}',
      PresetId: webPreset,
      Rotate: 'auto'
    }
  };
  eltr.createJob(params, function (err, data) {
    if (err) {
      console.log('Failed to send new video ' + key + ' to ET');
      console.log(err);
      console.log(err.stack)
    } else {
      console.log(data);
    }
  });
}

In the Lambda console output, the line Sending KEY to ET appears, so I know my function is being called. However, I don't get any further output in the logs after that line, so none of the console.log() lines are being called; no error, no stack trace, no data output at all.

There's a good chance I'm doing simple stupid, but I would be very grateful if someone could point me in the right direction!

@lsegal

This comment has been minimized.

Show comment
Hide comment
@lsegal

lsegal Jan 3, 2015

Contributor

There are two reasons why this command might be failing:

  1. It looks like you immediately call context.done() after kicking off the ET job. When Lambda receives context.done() it will likely be killing the process, but this is happening before the createJob has completed, since that call is asynchronous. It would explain why the synchronous parts of that function work (the console.log) but not the async part-- in other words, the process is being killed while you are waiting for the HTTP connection to be made. To solve this you should move the context.done() callback to the createJob() callback-- and also remember to keep it in the error condition of your initial call.
  2. This one is less important, but it will affect performance (and might be why the function is timing out)-- you are using s3.getObject(), which downloads the entire object contents as part of the call (since it is a GET). For videos, this is likely a large download and will take quite a long time, (and cost extra for the compute time needed). Given that you aren't using the contents in this function, I would recommend using s3.headObject(), which does a HEAD request grabbing all the metadata and other headers without accessing the body contents. That will save on time and costs.

Hope that helps,

Contributor

lsegal commented Jan 3, 2015

There are two reasons why this command might be failing:

  1. It looks like you immediately call context.done() after kicking off the ET job. When Lambda receives context.done() it will likely be killing the process, but this is happening before the createJob has completed, since that call is asynchronous. It would explain why the synchronous parts of that function work (the console.log) but not the async part-- in other words, the process is being killed while you are waiting for the HTTP connection to be made. To solve this you should move the context.done() callback to the createJob() callback-- and also remember to keep it in the error condition of your initial call.
  2. This one is less important, but it will affect performance (and might be why the function is timing out)-- you are using s3.getObject(), which downloads the entire object contents as part of the call (since it is a GET). For videos, this is likely a large download and will take quite a long time, (and cost extra for the compute time needed). Given that you aren't using the contents in this function, I would recommend using s3.headObject(), which does a HEAD request grabbing all the metadata and other headers without accessing the body contents. That will save on time and costs.

Hope that helps,

@lsegal lsegal added the Question label Jan 3, 2015

@grvhi

This comment has been minimized.

Show comment
Hide comment
@grvhi

grvhi Jan 5, 2015

@lsegal - thank you very much. Both 1. and 2. are absolutely correct!

grvhi commented Jan 5, 2015

@lsegal - thank you very much. Both 1. and 2. are absolutely correct!

@grvhi grvhi closed this Jan 5, 2015

@mortenlyager

This comment has been minimized.

Show comment
Hide comment
@mortenlyager

mortenlyager Feb 23, 2015

@Erve1879 - Can you post a working copy of your code?

@Erve1879 - Can you post a working copy of your code?

@DigitalFrontiersMedia

This comment has been minimized.

Show comment
Hide comment
@DigitalFrontiersMedia

DigitalFrontiersMedia Oct 28, 2015

@Erve1879 Thanks for posting the gist of the working copy but it appears to be 404 now. Would it be possible to post it back again? I'm interested in how you use s3.headObject() with this. Thank you!

@Erve1879 Thanks for posting the gist of the working copy but it appears to be 404 now. Would it be possible to post it back again? I'm interested in how you use s3.headObject() with this. Thank you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment