aws-sdk mediaconvert createJob(params) failing with MissingRequiredParameter errors

[spouzols]

Hi. I'm opening this topic as a regression issue, as we didn't find a reference to it, on this repo or the AWS forums, except maybe this tweet: https://twitter.com/shinichy/status/1029908422181838848

We hit a regression with the MediaConvert createJob(params) function in aws-sdk, in an AWS lambda context in Node.js 6.10 environment. Our lambda had been running for 4 months untouched and triggers a simple transcoding job creation when a movie file is received on a source s3 bucket, and suddenly a transcoding job failed to be created on August 16. The previous successful lambda created a job on August 14. The createJob() is called essentially with a template (which also hasn't changed for about 4 months), and a few additional parameters specific to the environment of the lambda, and the source object on s3.

The error was :

Unable to transcode input at s3://<bucket>/<filename>.mp4 due to an error: MultipleValidationErrors: There were 4 validation errors:
* MissingRequiredParameter: Missing required key 'Outputs' in params.Settings.OutputGroups[0]
* MissingRequiredParameter: Missing required key 'Type' in params.Settings.OutputGroups[0].OutputGroupSettings
* MissingRequiredParameter: Missing required key 'MinSegmentLength' in params.Settings.OutputGroups[0].OutputGroupSettings.HlsGroupSettings
* MissingRequiredParameter: Missing required key 'SegmentLength' in params.Settings.OutputGroups[0].OutputGroupSettings.HlsGroupSettings

We were using the default aws-sdk version provided in the lambda node environment so didn't have immediately a version to point to, but we pinpointed the behavior change to the 2.251.1 version of aws-sdk. (Note that the current AWS doc at https://docs.aws.amazon.com/lambda/latest/dg/current-supported-versions.html claims the 2.249.1 is currently deployed.)

Here is a simpler version of the lambda code, with the same createJob() :

const AWS = require('aws-sdk');

const MEDIA_CONVERT_ENDPOINT = process.env.MEDIA_CONVERT_ENDPOINT;
const MEDIA_CONVERT_ROLE_ARN = process.env.MEDIA_CONVERT_ROLE_ARN;
const JOB_TEMPLATE = process.env.JOB_TEMPLATE;
const OUTPUT_LOCATION = process.env.OUTPUT_LOCATION;

const mediaConvert = new AWS.MediaConvert({endpoint: MEDIA_CONVERT_ENDPOINT});

exports.handler = function(event, context, callback) {
  const srcBucket = event.Records[0].s3.bucket.name;
  const srcKey = decodeURIComponent(event.Records[0].s3.object.key.replace(/\+/g, ' '));
  const fileInput = `s3://${srcBucket}/${srcKey}`;

  const params = {
    JobTemplate: JOB_TEMPLATE,
    Role: MEDIA_CONVERT_ROLE_ARN,
    Settings: {
      Inputs: [
        {
          FileInput: fileInput,
        },
      ],
    },
  };

  if (OUTPUT_LOCATION) {
    params.Settings.OutputGroups = [
      {
        OutputGroupSettings: {
          HlsGroupSettings: {
            Destination: OUTPUT_LOCATION,
          },
        },
      },
    ];
  }

  mediaConvert.createJob(params).promise()
  .then(() => {
    console.log('Transcode job created.');
    callback(null, 'message');
  })
  .catch((err) => {
    console.error(`Unable to transcode input at s3://${srcBucket}/${srcKey} due to an error: ${err}`);
    callback(null, 'message');
  });
};

The template itself does contain the necessary parameters cited (Outputs, Type, MinSegmentLength, SegmentLength) and this is the crux of the matter, those fields seem to have been added to the required parameters expected locally by the aws-sdk client while this wasn't the case before.

Template JSON:

{
  "Queue": "arn:aws:mediaconvert:region:account:queues/Default",
  "Name": "templateName",
  "Settings": {
    "OutputGroups": [
      {
        "CustomName": "",
        "Name": "HLS",
        "Outputs": [
          {
            "Preset": "h264-240p",
            "NameModifier": "-240p"
          },
          {
            "Preset": "h264-360p",
            "NameModifier": "-360p"
          },
          {
            "Preset": "h264-480p",
            "NameModifier": "-480p"
          },
          {
            "Preset": "h264-720p",
            "NameModifier": "-720p"
          },
          {
            "Preset": "h264-1080p",
            "NameModifier": "-1080p"
          }
        ],
        "OutputGroupSettings": {
          "Type": "HLS_GROUP_SETTINGS",
          "HlsGroupSettings": {
            "ManifestDurationFormat": "INTEGER",
            "SegmentLength": 6,
            "TimedMetadataId3Period": 10,
            "CaptionLanguageSetting": "OMIT",
            "Destination": "s3://<bucket>/",
            "TimedMetadataId3Frame": "PRIV",
            "CodecSpecification": "RFC_4281",
            "OutputSelection": "MANIFESTS_AND_SEGMENTS",
            "ProgramDateTimePeriod": 600,
            "MinSegmentLength": 0,
            "DirectoryStructure": "SINGLE_DIRECTORY",
            "ProgramDateTime": "EXCLUDE",
            "SegmentControl": "SEGMENTED_FILES",
            "ManifestCompression": "NONE",
            "ClientCache": "ENABLED",
            "StreamInfResolution": "INCLUDE"
          }
        }
      }
    ],
    "AdAvailOffset": 0,
    "Inputs": [
      {
        "AudioSelectors": {
          "Audio Selector 1": {
            "Offset": 0,
            "DefaultSelection": "DEFAULT",
            "ProgramSelection": 1
          }
        },
        "VideoSelector": {
          "ColorSpace": "FOLLOW"
        },
        "FilterEnable": "AUTO",
        "PsiControl": "USE_PSI",
        "FilterStrength": 0,
        "DeblockFilter": "DISABLED",
        "DenoiseFilter": "DISABLED",
        "TimecodeSource": "EMBEDDED"
      }
    ]
  }
}

Additional checks :

• This is still failing today, and we checked the latest 2.295.0 version with the same error
• Creating a transcode job from the AWS console from the template itself does work

Workarounds :

• As a workaround we currently build with the 2.250.1 version packaged inside the lambda
• Using a getJobTemplate() and merging it to createJob() with a complete JSON should work

To be clear the current AWS SDK documentation (https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/MediaConvert.html#createJob-property) does now mention those fields as required, but the aws-sdk changelog doesn't mention this change in 2.251.1 and only details added transcoding features.

[chrisradek]

@spouzols
Thanks for reporting this. I'll get in touch with the service team to find out why these fields were marked as required. We sometimes allow parameters to be marked required when they weren't originally for the scenario where the service would return a validation error if the parameter was missing, but you've shown that's not the case here.

[chrisradek]

@spouzols
This should now be fixed with version 2.297.0 of the SDK. Thanks for bringing the issue to our attention!

[spouzols]

@chrisradek
Thank you. We checked our use case with fix version 2.297.0 packaged in lambda and it's working.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs and link to relevant comments in this thread.