Connection failure when using SDKv2 within API Gateway Lambda from SAM

[MaxOrelus]

Describe the bug

I've developed a serverless api function that uses the iot SDK to connect to mqtt and publish a message. During development the functionality works as it's suppose to, but once it's deployed into AWS there seems to be some network issues with the SDK. The MQTT is not able to establish a connection when deployed within AWS.

Expected Behavior

It should be able to connect to the MQTT client and publish a message to a topic.

Current Behavior

START RequestId: b34092c7-e644-419b-b5a3-6a7608e40354 Version: $LATEST
--
2023-01-09T15:58:30.162Z	b34092c7-e644-419b-b5a3-6a7608e40354	INFO	AmazonRootCA1.pemdevice.pem.crtprivate.pem.keypublic.pem.key
2023-01-09T15:58:30.601Z	b34092c7-e644-419b-b5a3-6a7608e40354	INFO	Creating client for REDACTED
2023-01-09T15:58:30.641Z	b34092c7-e644-419b-b5a3-6a7608e40354	INFO	Attempting Connect event
2023-01-09T15:58:30.765Z	b34092c7-e644-419b-b5a3-6a7608e40354	INFO	Connection failure event: Error: libaws-c-mqtt: AWS_ERROR_MQTT5_CONNACK_CONNECTION_REFUSED, Remote endpoint rejected the CONNECT attempt by returning an unsuccessful CONNACK
2023-01-09T15:58:30.766Z	b34092c7-e644-419b-b5a3-6a7608e40354	INFO	Connack: {     "type": 2,     "sessionPresent": false,     "reasonCode": 135,     "reasonString": "CONNACK:Client is not authenticated/authorized to send the message:aedac68e-53f1-6b62-7c0c-4cdc6f305980",     "userProperties": [],     "sharedSubscriptionsAvailable": false }
2023-01-09T15:58:30.906Z	b34092c7-e644-419b-b5a3-6a7608e40354	INFO	Attempting Connect event
2023-01-09T15:58:31.003Z	b34092c7-e644-419b-b5a3-6a7608e40354	INFO	Connection failure event: Error: libaws-c-mqtt: AWS_ERROR_MQTT5_CONNACK_CONNECTION_REFUSED, Remote endpoint rejected the CONNECT attempt by returning an unsuccessful CONNACK
2023-01-09T15:58:31.003Z	b34092c7-e644-419b-b5a3-6a7608e40354	INFO	Connack: {     "type": 2,     "sessionPresent": false,     "reasonCode": 135,     "reasonString": "CONNACK:Client is not authenticated/authorized to send the message:6b65b2f5-bc3c-e541-5534-c748b134f1fd",     "userProperties": [],     "sharedSubscriptionsAvailable": false }
2023-01-09T15:58:32.830Z	b34092c7-e644-419b-b5a3-6a7608e40354	INFO	Attempting Connect event
2023-01-09T15:58:32.914Z	b34092c7-e644-419b-b5a3-6a7608e40354	INFO	Connection failure event: Error: libaws-c-mqtt: AWS_ERROR_MQTT5_CONNACK_CONNECTION_REFUSED, Remote endpoint rejected the CONNECT attempt by returning an unsuccessful CONNACK
2023-01-09T15:58:32.914Z	b34092c7-e644-419b-b5a3-6a7608e40354	INFO	Connack: {     "type": 2,     "sessionPresent": false,     "reasonCode": 135,     "reasonString": "CONNACK:Client is not authenticated/authorized to send the message:247689be-d6a3-f5fa-6497-24aef2b2a55c",     "userProperties": [],     "sharedSubscriptionsAvailable": false }
2023-01-09T15:58:36.206Z	b34092c7-e644-419b-b5a3-6a7608e40354	INFO	Attempting Connect event
2023-01-09T15:58:36.298Z	b34092c7-e644-419b-b5a3-6a7608e40354	INFO	Connection failure event: Error: libaws-c-mqtt: AWS_ERROR_MQTT5_CONNACK_CONNECTION_REFUSED, Remote endpoint rejected the CONNECT attempt by returning an unsuccessful CONNACK
2023-01-09T15:58:36.298Z	b34092c7-e644-419b-b5a3-6a7608e40354	INFO	Connack: {     "type": 2,     "sessionPresent": false,     "reasonCode": 135,     "reasonString": "CONNACK:Client is not authenticated/authorized to send the message:5f2a0092-22ab-f77d-8201-e6ff69ab0718",     "userProperties": [],     "sharedSubscriptionsAvailable": false }
2023-01-09T15:58:44.158Z	b34092c7-e644-419b-b5a3-6a7608e40354	INFO	Attempting Connect event
2023-01-09T15:58:44.244Z	b34092c7-e644-419b-b5a3-6a7608e40354	INFO	Connection failure event: Error: libaws-c-mqtt: AWS_ERROR_MQTT5_CONNACK_CONNECTION_REFUSED, Remote endpoint rejected the CONNECT attempt by returning an unsuccessful CONNACK
2023-01-09T15:58:44.245Z	b34092c7-e644-419b-b5a3-6a7608e40354	INFO	Connack: {     "type": 2,     "sessionPresent": false,     "reasonCode": 135,     "reasonString": "CONNACK:Client is not authenticated/authorized to send the message:a209f37f-2231-9355-c2f3-4d9849ca19ab",     "userProperties": [],     "sharedSubscriptionsAvailable": false }
2023-01-09T15:58:45.097Z b34092c7-e644-419b-b5a3-6a7608e40354 Task timed out after 15.02 seconds
END RequestId: b34092c7-e644-419b-b5a3-6a7608e40354
REPORT RequestId: b34092c7-e644-419b-b5a3-6a7608e40354	Duration: 15017.61 ms	Billed Duration: 15000 ms	Memory Size: 128 MB	Max Memory Used: 72 MB	Init Duration: 265.30 ms

Reproduction Steps

Create a SAM api lambda function with no auth to hit the API. In the lambda try to connect to MQTT in greengrass and publish a topic.

SAM

AWSTemplateFormatVersion: "2010-09-09"
Transform: AWS::Serverless-2016-10-31
Description: >
  Issue Function

Resources:
  ApiRelayGW:
    Type: AWS::Serverless::Api
    Properties:
      StageName: prod
      # Auth:
      #   ApiKeyRequired: true
      CacheClusterEnabled: true
      CacheClusterSize: '0.5'
      MethodSettings:
        - ResourcePath: '/'
          HttpMethod: 'POST'
          CachingEnabled: true
          CacheTtlInSeconds: 300
  ApiRelayFunction:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: pkg/
      Handler: index.handler
      Runtime: nodejs12.x
      Timeout: 15
      Events:
        RelayRequest:
          Type: Api
          Properties:
            Path: /
            Method: POST
            RestApiId:
              Ref: ApiRelayGW

Outputs:
  Function:
    Description: "Lambda Function ARN"
    Value: !GetAtt ApiRelayFunction.Arn

MQTT client

var { mqtt5, iot } = require("aws-iot-device-sdk-v2");
var path = require("path");

function createClient() {
  let config = creatClientConfig();

  console.log("Creating client for " + config.hostName);
  let client = new mqtt5.Mqtt5Client(config);

  client.on('error', (error) => {
    console.log("Error event: " + error.toString());
  });

  client.on("messageReceived", (eventData) => {
    console.log("Message Received event: " + JSON.stringify(eventData.message));
  });

  client.on('attemptingConnect', (eventData) => {
    console.log("Attempting Connect event");
  });

  client.on('connectionSuccess', (eventData) => {
    console.log("Connection Success event");
    console.log("Connack: " + JSON.stringify(eventData.connack));
    console.log("Settings: " + JSON.stringify(eventData.settings));
  });

  client.on('connectionFailure', (eventData) => {
    console.log("Connection failure event: " + eventData.error.toString());
    if (eventData.connack) {
      console.log("Connack: " + JSON.stringify(eventData.connack));
    }
  });

  client.on('disconnection', (eventData) => {
    console.log("Disconnection event: " + eventData.error.toString());
    if (eventData.disconnect !== undefined) {
      console.log('Disconnect packet: ' + JSON.stringify(eventData.disconnect));
    }
  });

  client.on('stopped', (eventData) => {
    console.log("Stopped event");
  });

  return client;
}

function creatClientConfig() {
  let builder = iot.AwsIotMqtt5ClientConfigBuilder.newDirectMqttBuilderWithMtlsFromPath(
    "endpoint-goes-here",
    path.resolve(`${__dirname}/ssl/device.pem.crt`),
    path.resolve(`${__dirname}/ssl/private.pem.key`)
  );

  builder = iot.AwsIotMqtt5ClientConfigBuilder.newWebsocketMqttBuilderWithSigv4Auth(
    "endpoint-goes-here",
    { region: "us-east-1" }
  );

  builder.withConnectProperties({
    keepAliveIntervalSeconds: 1200
  });

  return builder.build();
}

module.exports = createClient

lambda handler

var mqtt = require("./mqtt")
var { once } = require("events")
var { mqtt5 } = require("aws-iot-device-sdk-v2")
var { v4: uuidv4 } = require('uuid');
var { toUtf8 } = require("@aws-sdk/util-utf8-browser")

module.exports = async ({ target, method, client_prefix, event }) => {
  // ! drop the request if any of the params are missing
  if (!target || !method || !client_prefix || !event) {
    reject({
      error: true,
      message: "Missing parameters for API Relay execution"
    })
  }

  var message_id = uuidv4()

  var TOPIC = `${target}/${method}/${client_prefix}/${message_id}`
  var TOPIC_ACK = `${client_prefix}/${message_id}/${target}/${method}/ack`

  var client = mqtt()
  const connectionSuccess = once(client, "connectionSuccess");
  client.start();
  await connectionSuccess;

  await client.subscribe({
    subscriptions: [
      { qos: mqtt5.QoS.AtLeastOnce, topicFilter: TOPIC_ACK }
    ]
  });

  console.log("Publishing message")
  console.log(TOPIC)

  var messagePromise = new Promise((resolve, reject) => {
    client.on("messageReceived", (eventData) => {
      var payload = toUtf8(new Uint8Array(eventData.message.payload))
      resolve(payload)
    })
  })

  await client.publish({
    qos: mqtt5.QoS.AtMostOnce,
    topicName: TOPIC,
    payload: JSON.stringify({ ...event, message_id })
  });

  var message = await messagePromise

  if (message.length > 0) {
    return {
      statusCode: 200,
      body: message
    }
  }

  return {
    statusCode: 500
  }
}

Possible Solution

Not sure, I think there's networking issues in the SDK

Additional Information/Context

No response

SDK version used

"aws-iot-device-sdk-v2": "^1.9.0"

Environment details (OS name and version, etc.)

Linux Ubuntu 22.04

[MaxOrelus]

To clarify when I say the lambda works locally I mean:

sam build --use-container && sam local start-api --debug

The function runs as intended with no network connectivity issues locally.

If I deploy:

sam build --use-container && sam deploy

and then swap the endpoint from my localhost to the prod endpoint provided by AWS and send the request. I get network issues when I look at the cloudwatch logs for the lambda invocation through the API GW logs.

[bretambrose]

2023-01-09T15:58:30.765Z	b34092c7-e644-419b-b5a3-6a7608e40354	INFO	Connection failure event: Error: libaws-c-mqtt: AWS_ERROR_MQTT5_CONNACK_CONNECTION_REFUSED, Remote endpoint rejected the CONNECT attempt by returning an unsuccessful CONNACK
2023-01-09T15:58:30.766Z	b34092c7-e644-419b-b5a3-6a7608e40354	INFO	Connack: {     "type": 2,     "sessionPresent": false,     "reasonCode": 135,     "reasonString": "CONNACK:Client is not authenticated/authorized to send the message:aedac68e-53f1-6b62-7c0c-4cdc6f305980",     "userProperties": [],     "sharedSubscriptionsAvailable": false }

Would indicate that it's an IAM permissions issue.

[MaxOrelus]

I'm not following?

The connection is established with these:

• AmazonRootCA1.pem
• device.pem.crt
• private.pem.key

If it was a IAM permission issue wouldn't it not work when I do an invocation locally when I run the sam build --use-container && sam local start-api --debug. I don't get a connection failure.

Same credentials across the board.

[bretambrose]

I don't have any experience with sam and/or the relationship (if any) between local deployments/services and associated real AWS services.

You're getting a CONNACK packet from IoT Core that is explicitly stating that there's no permission to send a CONNECT packet in the policy associated with the authentication context (cert/key pair).

[MaxOrelus]

Circled in the SAM folks.

[jmklix]

Can you make sure that you have set up your IAM policy correctly with the correct permissions? Your IAM policy can be set up to only allow certain things. You can test with a fully permissive IoT thing policy and then restrict it to only things that you need later:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "*"
      ],
      "Resource": [
        "*"
      ]
    }
  ]
}

You can read more about IoT Core policies here

[github-actions]

Greetings! It looks like this issue hasn’t been active in longer than a week. We encourage you to check if this is still an issue in the latest release. Because it has been longer than a week since the last update on this, and in the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please feel free to provide a comment or add an upvote to prevent automatic closure, or if the issue is already closed, please feel free to open a new one.