Regression: Event is not UTF-8 encoded when testing lambda locally #2494
Labels
Comments
|
I can reproduce this issue. I will investigate it. |
|
I had a same issue like you. Encrypted Json CreatorEncrypted Your example json filelambda_function.py
|
|
I can confirm the same error whenever the body contains any character that cannot be encoded into Latin-1. As already mentioned, the issue is related to the |
|
Thanks for reporting this! As @eduardomourar mentioned, the issue is |
hoffa
added
the
stage/waiting-for-release
|
This has been fixed in 1.17.0: % sam --version
SAM CLI, version 1.17.0
% sam local invoke HelloWorldFunction -e events/event.json
Invoking app.lambda_handler (python3.8)
Skip pulling image and use local one: amazon/aws-sam-cli-emulation-image-python3.8:rapid-1.17.0.
Mounting /Users/rehnc/Desktop/tmp/sam-app/.aws-sam/build/HelloWorldFunction as /var/task:ro,delegated inside runtime container
START RequestId: c6276e1b-60ec-46a2-9e6c-ddc361ba2fc1 Version: $LATEST
END RequestId: c6276e1b-60ec-46a2-9e6c-ddc361ba2fc1
REPORT RequestId: c6276e1b-60ec-46a2-9e6c-ddc361ba2fc1 Init Duration: 0.13 ms Duration: 86.39 ms Billed Duration: 100 ms Memory Size: 128 MB Max Memory Used: 128 MB
{"statusCode": 200, "body": "{\"message\": \"hello world\"}"} |
hoffa
removed
the
stage/waiting-for-release
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Description:
Passing a non-Latin body to
sam local invokeproduces aUnicodeEncodeError.My guess is the changes from #2281 might be involved (cc @awood45).
Steps to reproduce:
Unfortunately I haven't yet had time to produce a minimal repo but the command line I am using is:
Event JSON file:
{ "Records": [ { "messageId": "19dd0b57-b21e-4ac1-bd88-01bbb068cb78", "receiptHandle": "MessageReceiptHandle", "body": "[{\"type\":\"kanji\",\"deleted\":false,\"value\":{\"r\":{\"py\":[\"ya4\"],\"on\":[\"ア\"],\"kun\":[\"つ.ぐ\"],\"na\":[\"や\",\"つぎ\",\"つぐ\"]},\"m\":{\"en\":[\"Asia\",\"rank next\",\"come after\",\"-ous\"],\"fr\":[\"Asie\",\"suivant\",\"sub-\",\"sous-\"],\"es\":[\"pref. para indicar\",\"venir después de\",\"Asia\"],\"pt\":[\"Ásia\",\"próxima\",\"o que vem depois\",\"-ous\"]},\"rad\":{\"x\":7,\"nelson\":1},\"refs\":{\"nelson_c\":43,\"nelson_n\":81,\"halpern_njecd\":3540,\"halpern_kkd\":4354,\"halpern_kkld\":2204,\"halpern_kkld_2ed\":2966,\"heisig\":1809,\"heisig6\":1950,\"gakken\":1331,\"oneill_names\":525,\"oneill_kk\":1788,\"moro\":{\"vol\":1,\"page\":525,\"ref\":\"272\"},\"henshall\":997,\"sh_kk\":1616,\"sh_kk2\":1724,\"jf_cards\":1032,\"tutt_cards\":1092,\"kanji_in_context\":1818,\"kodansha_compact\":35,\"maniette\":1827,\"skip\":\"4-7-1\",\"sh_desc\":\"0a7.14\",\"four_corner\":\"1010.6\",\"deroo\":3273,\"conning\":545},\"misc\":{\"sc\":7,\"gr\":8,\"freq\":1509,\"jlpt\":1,\"kk\":25,\"jlptn\":1},\"c\":\"亜\",\"comp\":\"⼀⼁⼝\"}},{\"type\":\"kanji\",\"deleted\":false,\"value\":{\"r\":{\"py\":[\"ya1\"],\"on\":[\"ア\",\"アク\"],\"kun\":[\"おし\"]},\"m\":{\"en\":[\"mute\",\"dumb\"]},\"rad\":{\"x\":30},\"refs\":{\"nelson_c\":939,\"nelson_n\":795,\"heisig\":2958,\"heisig6\":2964,\"moro\":{\"vol\":2,\"page\":1066,\"ref\":\"3743\"},\"skip\":\"1-3-7\",\"sh_desc\":\"3d8.3\",\"four_corner\":\"6101.7\"},\"misc\":{\"sc\":10,\"kk\":15},\"c\":\"唖\",\"comp\":\"⼀⼁⼝⺉\"}},{\"type\":\"kanji\",\"deleted\":false,\"value\":{\"r\":{\"py\":[\"wa2\"],\"on\":[\"ア\",\"アイ\",\"ワ\"],\"kun\":[\"うつく.しい\"],\"na\":[\"い\"]},\"m\":{\"en\":[\"beautiful\"]},\"rad\":{\"x\":38},\"refs\":{\"nelson_n\":1213,\"halpern_kkd\":419,\"halpern_kkld_2ed\":313,\"heisig\":2200,\"heisig6\":2313,\"moro\":{\"vol\":3,\"page\":707,\"ref\":\"6262\"},\"skip\":\"1-3-6\",\"sh_desc\":\"3e6.5\",\"four_corner\":\"4441.4\"},\"misc\":{\"sc\":9,\"gr\":9,\"kk\":15},\"c\":\"娃\",\"comp\":\"⼟⼥\"}},{\"type\":\"kanji\",\"deleted\":false,\"value\":{\"r\":{\"py\":[\"a1\",\"e1\",\"a5\",\"a2\",\"a4\"],\"on\":[\"ア\",\"オ\"],\"kun\":[\"おもね.る\",\"くま\"],\"na\":[\"ほとり\",\"あず\",\"あわ\",\"おか\",\"きた\",\"な\"]},\"m\":{\"en\":[\"Africa\",\"flatter\",\"fawn upon\",\"corner\",\"nook\",\"recess\"],\"fr\":[\"flatter\",\"flagorner\",\"Afrique\",\"coin\",\"recoin\"],\"es\":[\"A fonética\",\"África\",\"meandro\",\"adular\",\"halagar\"]},\"rad\":{\"x\":170},\"refs\":{\"nelson_c\":4985,\"nelson_n\":6435,\"halpern_njecd\":346,\"halpern_kkd\":408,\"halpern_kkld\":256,\"halpern_kkld_2ed\":305,\"heisig\":1295,\"heisig6\":1391,\"gakken\":1515,\"oneill_names\":569,\"moro\":{\"vol\":11,\"page\":798,\"ref\":\"41599\"},\"sh_kk\":2258,\"maniette\":1304,\"skip\":\"1-3-5\",\"sh_desc\":\"2d5.6\",\"four_corner\":\"7122.0\",\"conning\":819},\"misc\":{\"sc\":8,\"gr\":9,\"freq\":1126,\"jlpt\":1,\"kk\":15,\"jlptn\":1},\"c\":\"阿\",\"comp\":\"⼅⼝⻖\"}},{\"type\":\"kanji\",\"deleted\":false,\"value\":{\"r\":{\"py\":[\"ai1\"],\"on\":[\"アイ\"],\"kun\":[\"あわ.れ\",\"あわ.れむ\",\"かな.しい\"]},\"m\":{\"en\":[\"pathetic\",\"grief\",\"sorrow\",\"pathos\",\"pity\",\"sympathize\"],\"fr\":[\"pitoyable\",\"peine\",\"chagrin\",\"pitié\",\"pathétique\",\"compatir\"],\"es\":[\"compasión\",\"lástima\",\"miseria\",\"piedad\",\"pena\",\"compadecerse de\"],\"pt\":[\"patético\",\"pesar\",\"pena\",\"emoção\",\"compaixão\",\"solidariesar\"]},\"rad\":{\"x\":30,\"nelson\":8},\"refs\":{\"nelson_c\":304,\"nelson_n\":791,\"halpern_njecd\":2068,\"halpern_kkd\":2568,\"halpern_kkld\":1310,\"halpern_kkld_2ed\":1781,\"heisig\":401,\"heisig6\":428,\"gakken\":1670,\"oneill_kk\":1249,\"moro\":{\"vol\":2,\"page\":997,\"ref\":\"3580\"},\"henshall\":998,\"sh_kk\":1675,\"sh_kk2\":1790,\"jf_cards\":1131,\"tutt_cards\":1239,\"kanji_in_context\":1804,\"kodansha_compact\":327,\"maniette\":408,\"skip\":\"2-2-7\",\"sh_desc\":\"2j7.4\",\"four_corner\":\"0073.2\",\"deroo\":465,\"conning\":1860},\"misc\":{\"sc\":9,\"gr\":8,\"freq\":1715,\"jlpt\":1,\"kk\":3,\"jlptn\":1},\"c\":\"哀\",\"comp\":\"⼇⼝⾐\"}},{\"type\":\"kanji\",\"deleted\":false,\"value\":{\"r\":{\"py\":[\"ai4\"],\"on\":[\"アイ\"],\"kun\":[\"いと.しい\",\"かな.しい\",\"め.でる\",\"お.しむ\",\"まな\"],\"na\":[\"あ\",\"あし\",\"え\",\"かな\",\"なる\",\"めぐ\",\"めぐみ\",\"よし\",\"ちか\"]},\"m\":{\"en\":[\"love\",\"affection\",\"favourite\"],\"fr\":[\"amour\",\"affection\",\"favori\"],\"es\":[\"amor\",\"afecto\",\"favorito\"],\"pt\":[\"amor\",\"afeição\",\"favorito\"]},\"rad\":{\"x\":61,\"nelson\":87},\"refs\":{\"nelson_c\":2829,\"nelson_n\":1927,\"halpern_njecd\":2492,\"halpern_kkd\":3133,\"halpern_kkld\":1606,\"halpern_kkld_2ed\":2191,\"heisig\":737,\"heisig6\":796,\"gakken\":436,\"oneill_names\":2018,\"oneill_kk\":456,\"moro\":{\"vol\":4,\"page\":1123,\"ref\":\"10947\"},\"henshall\":417,\"sh_kk\":259,\"sh_kk2\":259,\"sakade\":339,\"jf_cards\":545,\"henshall3\":441,\"tutt_cards\":602,\"crowley\":268,\"kanji_in_context\":1079,\"kodansha_compact\":790,\"maniette\":745,\"skip\":\"2-4-9\",\"sh_desc\":\"4i10.1\",\"four_corner\":\"2024.7\",\"deroo\":2067,\"skip_miss\":[{\"misclass\":\"posn\",\"ref\":\"2-1-12\"},{\"misclass\":\"posn\",\"ref\":\"2-6-7\"}],\"conning\":778},\"misc\":{\"sc\":13,\"gr\":4,\"freq\":640,\"jlpt\":2,\"kk\":7,\"jlptn\":3},\"c\":\"愛\",\"comp\":\"受⽖⼍⼡⼼\"}},{\"type\":\"kanji\",\"deleted\":false,\"value\":{\"r\":{\"py\":[\"ai1\",\"ai2\"],\"on\":[\"アイ\"],\"kun\":[\"ひら.く\"]},\"m\":{\"en\":[\"approach\",\"draw near\",\"push open\"],\"es\":[\"empujar\",\"acercarse\"]},\"rad\":{\"x\":64},\"refs\":{\"nelson_c\":1910,\"nelson_n\":2160,\"halpern_kkd\":510,\"halpern_kkld_2ed\":383,\"heisig\":2248,\"heisig6\":1310,\"moro\":{\"vol\":5,\"page\":229,\"ref\":\"12082\"},\"sh_kk2\":1101,\"skip\":\"1-3-7\",\"sh_desc\":\"3c7.12\",\"four_corner\":\"5303.4\",\"deroo\":1363,\"conning\":2247},\"misc\":{\"sc\":10,\"gr\":8,\"freq\":2258,\"kk\":2},\"c\":\"挨\",\"comp\":\"⼛⽮⺘\"}},{\"type\":\"kanji\",\"deleted\":false,\"value\":{\"r\":{\"py\":[\"yan3\"],\"on\":[\"オウ\",\"アイ\"],\"kun\":[\"あい\"]},\"m\":{\"en\":[\"good-looking\",\"quiet\"]},\"rad\":{\"x\":38},\"refs\":{\"nelson_n\":1208,\"oneill_names\":1091,\"moro\":{\"vol\":3,\"page\":696,\"ref\":\"6242\"},\"skip\":\"1-3-6\",\"sh_desc\":\"3e6.7\",\"four_corner\":\"4846.1\"},\"misc\":{\"sc\":9,\"kk\":15},\"c\":\"姶\",\"comp\":\"⼀⼝⼥𠆢\"}},{\"type\":\"kanji\",\"deleted\":false,\"value\":{\"r\":{\"py\":[\"feng2\"],\"on\":[\"ホウ\"],\"kun\":[\"あ.う\",\"むか.える\"],\"na\":[\"あい\",\"おう\"]},\"m\":{\"en\":[\"meeting\",\"tryst\",\"date\",\"rendezvous\"],\"es\":[\"encuentro\",\"cita\",\"encuentro casual\",\"encontrarse\",\"reunirse\",\"citarse\",\"verse por casualidad\"]},\"rad\":{\"x\":162},\"refs\":{\"nelson_c\":4694,\"nelson_n\":6054,\"halpern_kkd\":4002,\"halpern_kkld_2ed\":2774,\"heisig\":2417,\"heisig6\":2497,\"oneill_names\":1516,\"moro\":{\"vol\":11,\"page\":75,\"ref\":\"38901X\"},\"skip\":\"3-3-7\",\"sh_desc\":\"2q7.15\",\"four_corner\":\"3730.4\",\"deroo\":2555,\"skip_miss\":[{\"misclass\":\"stroke_diff\",\"ref\":\"3-4-7\"}],\"conning\":1379},\"misc\":{\"sc\":11,\"gr\":9,\"freq\":2116,\"kk\":15},\"c\":\"逢\",\"comp\":\"⼁⼆⼡⻍\",\"var\":[\"162-nyou-2\"]}},{\"type\":\"kanji\",\"deleted\":false,\"value\":{\"r\":{\"py\":[\"kui2\"],\"on\":[\"キ\"],\"kun\":[\"あおい\"],\"na\":[\"まもる\",\"け\"]},\"m\":{\"en\":[\"hollyhock\"],\"es\":[\"malva real\",\"malva loca\"]},\"rad\":{\"x\":140},\"refs\":{\"nelson_c\":3990,\"nelson_n\":5120,\"halpern_njecd\":2317,\"halpern_kkd\":2906,\"halpern_kkld\":1493,\"halpern_kkld_2ed\":2018,\"heisig\":2391,\"heisig6\":2472,\"oneill_names\":1723,\"moro\":{\"vol\":9,\"page\":806,\"ref\":\"31458X\"},\"sh_kk\":2222,\"skip\":\"2-3-9\",\"sh_desc\":\"3k9.17\",\"four_corner\":\"4443.0\"},\"misc\":{\"sc\":12,\"gr\":9,\"freq\":2122,\"jlpt\":1,\"kk\":15,\"jlptn\":1},\"c\":\"葵\",\"comp\":\"⼃⼆⼈⼤⽨⺾\"}}]", "attributes": { "ApproximateReceiveCount": "1", "SentTimestamp": "1523232000000", "SenderId": "123456789012", "ApproximateFirstReceiveTimestamp": "1523232000001" }, "messageAttributes": { "Type": { "dataType": "String", "stringValue": "kanji" }, "Timestamp": { "dataType": "Number", "stringValue": "1590969632292" } }, "md5OfBody": "7b270e59b47ff90a553787216d55d91d", "eventSource": "aws:sqs", "eventSourceARN": "arn:aws:sqs:us-west-2:xxxxx", "awsRegion": "us-west-2" }, ] }Observed result:
Stack trace
As best I can tell, the issue appears to be the event is passed to
requests.post(as introduced by 57ef284) where it is assumed to be encoded as Latin-1.Expected result:
This used to work without error (until presumably GitHub updated the version of sam-cli deployed on their Actions image).
Additional environment details (Ex: Windows, Mac, Amazon Linux etc)
sam --version: Whatever is currently deployed on GitHub actionsThe text was updated successfully, but these errors were encountered: