AWS Kinesis Firehose ErrorOutputPrefix

[majindageta]

This is a Bug Report

Description

• What went wrong? Creating a Firehose stream with custom prefix generates an error because AWS needs a ErrorOutputPrefix, adding the ErrorOutputPrefix generate an error in serveless because the key is not recognized.
• What did you expect should have happened? Recognize the ErrorOutputPrefix key
• What was the config you used?

this is a sample code i used to create the stream:

      Type: AWS::KinesisFirehose::DeliveryStream
      Properties:
        DeliveryStreamName: hello-workd
        S3DestinationConfiguration: #using ExtendedS3DestinationConfiguration too
          BucketARN:
            Fn::Join:
            - ''
            - - 'arn:aws:s3:::'
              - Ref: s3bucketName
          BufferingHints:
            IntervalInSeconds: "60"
            SizeInMBs: "1"
          CompressionFormat: "UNCOMPRESSED"
          Prefix: "!{timestamp:yyyy'-'MM'-'dd}/"
          ErrorOutputPrefix: "!{firehose:error-output-type}/!{timestamp:yyyy'-'MM'-'dd}/"
          RoleARN: { Fn::GetAtt: [ FirehoseToS3Role, Arn ] }

• What stacktrace or error message from your provider did you see?

An error occurred: stream - Ths supplied prefix(es) do not satisfy the following constraint: ErrorOutputPrefix cannot be null or empty when Prefix contains expressions (Service: AmazonKinesisFirehose; Status Code: 400; Error Code: InvalidArgumentException; Request ID: ec424f7e-a722-08c5-b96b-24425795edb8).

An error occurred: SgtvWwwLogsStream - Encountered unsupported property PrefixError

Additional Data

• Serverless Framework Version you're using: 1.40.0
• Operating System: MacOs 10.14.4

This is my first time opening an issue, hope everything is fine.
Thank you very much!

[dwoodruff-cricut]

I have spent an entire day trying to work around this issue also. I'm trying to do an update. It states that the field should be setup here in the cli reference. Even when I try it in boto3 as documented here, it still does not work. It is also documented here in the online documentation. None of these approaches work

[JonahAcquah]

Hope this is not too late. I finally got to the bottom of this. The AWS-SDK version used by the Lambda runtime is 2.290.0, certainly true for nodejs 8.10. This SDK version is missing the "ErrorOutputPrefix" key in the member list used for validation and hence the error message. To get around this issue, avoid trying to set this up in Cloudformation directly, you'll have no control over what SDK version is used. Use the Cloudformation "Custom Resources" facility to delegate the responsibility of creating the "Prefix" and "ErrorOutputPrefix" to your Nodejs (or whatever framework you're using), and force nodejs to use the latest version AWS-SDK by including the latest version in your deployment package. If this is not an option, then you'll just have to wait for AWS to upgrade the lambda AWS-SDK for whatever language/framework you're using. Hope this helps.

Cloudformation custom resource doc: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/template-custom-resources.html

[JonahAcquah]

Sample code to update Kinesis firehose via the custom resources nodejs function:

firehose.updateDestination({
CurrentDeliveryStreamVersionId: YOUR_VERSION_ID,
DeliveryStreamName: YOUR_DELIVERY_STREAM_NAME,
DestinationId: YOUR_STREAM_DESTINATION_ID,
ExtendedS3DestinationUpdate: {
Prefix: "YOUR_PREFIX",
ErrorOutputPrefix: "YOUR_ERROR_OUTPUT_PREFIX"
}
})

[dwoodruff-cricut]

Ah, so forcing the inclusion of the latest sdk along side of the cloud formation deploy may work. I presume I could do this in a python script, but your suggestion for a node script may also work. Ideally I'm trying to remove any human intervention when creating our firehose in a production setting and use the error prefix. I hope to try our your suggestion in the next few weeks. I have moved on to some other priorities at the moment.

[majindageta]

Thank you JonahAcquah for the sample.
This can be a partial solution for those who have this problem.

[Marzocchi666]

Hi, im trying to deploy also a kinesis firehose via CloudFormation using troposphere library, and i cannot use ErrorOutputPrefix, it states the following : "AttributeError: S3DestinationConfiguration object does not support attribute ErrorOutputPrefix"
Could not find any solution to this, and also the aws documentation [] (https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-kinesisfirehose-deliverystream-s3destinationconfiguration.html) does not mention anything about it, based on what is in the docs, it should work, but it does not, or maybe im doing something wrong, which i do not know what yet :(

[majindageta]

Hi Marzocch666 you can try #6015 (comment) solution by JonahAcquah.

Unfortunetly there is no fix for this

[Marzocchi666]

Hi Marzocch666 you can try #6015 (comment) solution by JonahAcquah.

Unfortunetly there is no fix for this

Thanks for reply!