API Gateway -> Kinesis Firehose - New Line Issue

[cwbeck]

For bug reports:

• What went wrong?

When using Kinesis Firehose with API Gateway, every variation of trying to add a new line fails when trying to write to S3 despite the docs suggesting this should work: http://docs.aws.amazon.com/firehose/latest/APIReference/API_PutRecord.html

Data looks like this instead: -

{"a":"foo","ip":"...","ua":"...","stage":"test"}\n{"a":"foo","ip":"...","ua":"...","stage":"test"}

• What did you expect should have happened?

Output of data in S3 should have new lines... (where \n is an actual new line!)

{"a":"foo","ip":"...","ua":"...","stage":"test"}
{"a":"foo","ip":"...","ua":"...","stage":"test"}
{"a":"foo","ip":"...","ua":"...","stage":"test"}
{"a":"foo","ip":"...","ua":"...","stage":"test"}

• What was the config you used?

#set ($payload = $input.path('$.d'))
#set ($payload.ip = $context.identity.sourceIp)
#set ($payload.ua = $context.identity.userAgent)
#set ($payload.stage = $context.stage)
{
    "DeliveryStreamName": "market",
    "Record": {
        "Data": "$util.base64Encode($input.json('$.d'))$util.base64Encode('\n')"
    }
}

• What stacktrace or error message from your provider did you see?

No parse errors.

[cwbeck]

Update:

Tried various ways to get this to work, instead just used the base64encoded value. Seems a bit of a messy way of doing it however...

#set ($payload = $input.path('$.d'))
#set ($payload.ip = $context.identity.sourceIp)
#set ($payload.ua = $context.identity.userAgent)
#set ($payload.stage = $context.stage)
#set ($dataString = "$input.json('$.d')")
{
    "DeliveryStreamName": "market",
    "Record": {
        "Data": "$util.base64Encode($dataString)Cg=="
    }
}

[StoneCypher]

I have this same issue, using the demo data producer in the firehose setup wizard

So do several people on StackOverflow

To reproduce, just set up a standard kinesis firehose setup, and follow the instructions given for testing with demo data. The demo producer will write JSON records in series with no delimiter of any kind, as so:

{"TICKER_SYMBOL":"AMZN","SECTOR":"TECHNOLOGY","CHANGE":10.49,"PRICE":775.14}{"TICKER_SYMBOL":"XTC","SECTOR":"HEALTHCARE","CHANGE":-1.11,"PRICE":113.19}{"TICKER_SYMBOL":"QXZ","SECTOR":"FINANCIAL","CHANGE":-1.64,"PRICE":216.29}

The Redshift consumer will then fail, the reason given being 1214 | Delimiter not found.

So Amazon's own example doesn't work, on these grounds.

[pmuens]

Closing this for now since it's AWS and not Serverless related.

Please post the solution for this here so that others who stumble upon this can read how to resolve it. And feel free to continue the discussion here...

[StoneCypher]

Oh wow, I forgot all about this.

1. You need to add FORMAT AS JSON 's3://yourbucketname/aJsonPathFile.txt'
2. You need to create a JSON path file at the above address because the default data producer produces upper-case column names, which redshift cannot consume

[pmuens]

Thanks for the update on this @StoneCypher 👍

[cheptsov]

@cwbeck Did you find any other ways to add new line delimiters? Last Friday AWS released a new update to some regions which broke your solution. Adding Cg== now doesn't help. It throws a SerializationException now in most cases if you add Cg==.

For example, this code doesn't work anymore:

{
"DeliveryStreamName": "fus-bear-csv-dev",
"Records": [
{
"Data": "$util.base64Encode('a')Cg=="
}
]
}

Method response body after transformations: {"__type":"SerializationException"}

[StoneCypher]

@pmuens - i just saw this, a little over a year later

you're welcome

[leigeber]

In case it helps the only thing that worked for me was the final answer here, actually adding a literal newline in a string variable and passing that through.

https://stackoverflow.com/questions/41159139/how-to-add-a-newline-in-a-mapping-template

[cheptsov]

@leigeber
I just got the same answer from their support. Thanks. I checked and it works. At the same time it's a pity that it's not documented.

[soobrosa]

@leigeber @cheptsov is it still working for you? tried to reproduce to no avail -- maybe I'm not getting right which newline do you mean - can you please help me clarifying what exactly should I do?

#set( $key = $context.identity.apiKey )
#set( $keyname = "apiKey" )
#set( $traceidval = $input.params().get("header").get("X-Amzn-Trace-Id"))#set( $bodyname = "body" )
#set( $traceid = "traceid")
#set( $body = $input.body )
#set( $quote = '"' )
#set( $b64 = $util.base64Encode("{$quote$keyname$quote:$quote$key$quote,$quote$traceid$quote:$quote$traceidval$quote,$quote$bodyname$quote:$body}") )
{
"DeliveryStreamName": "XstreamX",
 "Record": {
   "Data": "$b64"
 }
}

[leigeber]

@soobrosa The key is to have an actual newline in the string before encoding. Something like the following.

#set( $b64 = $util.base64Encode("{$quote$keyname$quote:$quote$key$quote,$quote$traceid$quote:$quote$traceidval$quote,$quote$bodyname$quote:$body}
") )

[soobrosa]

@leigeber thanks so much for helping me so fast -- it fixed the problem 👍