Get SQS Queue URL from ARN

[dinkzilla]

It appears that an SQS ARN has all of the information needed to construct a queue URL. A function to do this mapping without having to make an API call would be useful.

Usecase Example:

• A Lambda is triggered off of an SQS.
• The Lambda takes in batches, so it wants to delete the messages individually as they are completed so that if one fails the others in the batch do not need to go back on the queue (as they would if you used the callback).
• The messages sent into the Lambda already contain the SQS ARN, though not the URL, which is needed to delete. If we could delete using the ARN or translate the ARN easily into the queue URL, we can avoid having to add a queue environment variable to the lambda.

I can create create a pull request if this idea is deemed valuable.

[srchase]

@dinkzilla

We've initiated a request with the Lambda team to have the SQS Queue URL included with that event source (rather than needing it to be constructed from the ARN).

[AllanZhengYP]

Just add a little more information to @srchase reply. The SDKs currently don't have enough information on how the URL is constructed or whether they are generated in the same way across all the regions. It would be more reliable to have the URL in the Lambda event.

[daliadaud]

That would be great! I'm currently managing multiple queues, this would be very helpful in deleting the messages

[lucaspalencia]

Same situation here, multiple queues with Lambda trigger. Would be great to have the queue url on lambda event

[tgxn]

Hey, I was able to overcome the missing Queue URL with the following code in my NodeJS Function:

var accountId = incomingRecord.eventSourceARN.split(":")[4];
var queueName = incomingRecord.eventSourceARN.split(":")[5];
var queueUrl = sqs.endpoint.href + accountId + '/' + queueName;

console.log( 'deleting from ' + queueUrl );

var params = {
    QueueUrl: queueUrl, /* required */
    ReceiptHandle: incomingRecord.receiptHandle /* required */
};

It's still a little "hacky" but it works, and means I don't need to specify the Queue URL in the function or any variables.

[sturman]

Here is an example how you can get SQS queue URL by name using AWS CLI.

aws sqs list-queues | jq -r '.QueueUrls[] | select(. | contains("<queue_name_here>"))'

If you need to use environment variable then please use this sample

export NODE_ENV=production
aws sqs list-queues | jq -r '.QueueUrls[] | select(. | contains("<queue_prefix>" + env.NODE_ENV))'

[josh18]

@srchase @AllanFly120 Is the request to the lambda team publicly visible? Do you know if it has been actioned yet?

[mkulak]

The viable alternative would be to allow users to specify SQS ARN instead of queue url in the SDK API

// current code
sqsClient.deleteMessage { it.queueUrl(queueUrl).receiptHandle(receipt) }

// proposal
sqsClient.deleteMessage { it.queueArn(queueArn).receiptHandle(receipt) } 

In fact SNS client already works this way:

snsClient.publish { it.topicArn(topicArn).message(message) }

[williamjulianvicary]

Any update on this?

[ajredniwja]

Needs to be done by the service team, I can reach out to them for their progress.

[badfun]

@mkulak is correct. Other aws services seem to offer this flexibility. The ARN is all that is included in a message coming from a queue, and it seems pretty hacky to have to fiddle with the strings in order to poll the queue or send a message.

[mgmolisani]

Would also like to see this become more flexible. I have a simple API runner lambda that pulls request configs off multiple queues but I may need to update the message on failure so hardcoding the URL is not an option. Currently using the above "hack" from @tgxn to compose the URL.

[ghost]

@ajredniwja, @srchase any chance there's an update on this?

[NinovanderMark]

@ajredniwja @srchase It's been quite a while since the last update, do you know if and when we can expect this new field to be available?

[Leigh-M]

If you're provisioning via SAM template (or CloudFormation) you can just reference it in your template.yaml and pass in to lambda as an environment variable like this:

QueuedInvocationHandler:
  Type: AWS::Serverless::Function
  Properties:
    Handler: ....
    CodeUri: .......
    Environment:
      Variables:
        queueUrl: !Ref someQueueInTemplate

It is then available in process.env as:

const { queueUrl } = process.env;

[mike-driver]

Get SQS URL from eventSourceARN using Python function:
def getSQSUrlFromEventSourceArn(eventSourceARN):
# from this:
# arn:aws:sqs:eu-west-2:XXXXXXXXXXXX:sqs003
# to this:
# https://sqs.eu-west-2.amazonaws.com/XXXXXXXXXXXX/TestInputQueue

StrSplit = eventSourceARN.split(":")
service = StrSplit[2]
region = StrSplit[3]
accountId = StrSplit[4]
queueName = StrSplit[5]
queueUrl = "https://" + service + "." + region + ".amazonaws.com/" + accountId + '/' + queueName
return{
    queueUrl    
}

[delsinz]

What's the reason behind the design decision of using queue url instead of queue ARN for a lot of the SQS commands anyway?

[leifbennett]

Running into the converse while trying to subscribe an SQS queue to an SNS topic. I'm using the 2.x SDK (Java). I've got the queue URL from creating it with a CreateQueueRequest. However, the SNS SubscribeRequest builder needs the queue ARN for its endpoint.

Is there example code of using SNS over SQS, preferably in Java and using the 2.x SDK (or some equally good library)? That might show a workaround for this problem.

[mike-driver]

GetQueueAttributes.QueueArn

…

On Wed, 20 Apr 2022, 00:23 Leif Bennett, ***@***.***> wrote: Running into the converse while trying to subscribe an SQS queue to an SNS topic. I'm using the 2.x SDK (Java). I've got the queue URL from creating it with a CreateQueueRequest. However, the SNS SubscribeRequest builder needs the queue ARN for its endpoint. Is there example code of using SNS over SQS, preferably in Java and using the 2.x SDK (or some equally good library)? That might show a workaround for this problem. — Reply to this email directly, view it on GitHub <#15 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEGPCWQVI66TGX3SCAKORB3VF457LANCNFSM4TNILE2Q> . You are receiving this because you commented.Message ID: ***@***.***>

[Leigh-M]

@leifbennett for any standard serverless patterns check: https://serverlessland.com/patterns (the patterns & videos are awesome)

That sounds like a basic SNS -> SQS fan-out pattern you are describing there will probably be an official example as IaC (Infrastructure as Code) that you can simply deploy from the above website, or here is one I wrote recently in AWS SAM (as you can see it expects your index.js file to be in /dist folder with an exports.handler function - written for nodeJs - there will be Java examples in the patterns repo)

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: Example SAM SNS -> SQS fan-out pattern 

Resources:
  # SQS queues
  MyEventsQueue:
    Type: AWS::SQS::Queue
    Properties:
      QueueName: !Sub ${AWS::StackName}-MyEventsQueue
      MessageRetentionPeriod: 1209600 # 14 DAYS
      ReceiveMessageWaitTimeSeconds: 10 # mid-level polling. 0-20 secs short-long polling
      Tags:
        - Key: PROJECT
          Value: 'YOUR-PROJECT-NAME'
        - Key: PROVISIONEDBY
          Value: 'SAM/ CLOUDFORMATION'

  # SNS topics
  MyEventsSNSTopic:
    Type: AWS::SNS::Topic
    Properties:
      TopicName : MyEventsSNSTopic

  # lambdas
  MyEventsHandler:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: dist/MyEventsHandler
      Handler: index.handler
      FunctionName: !Sub ${AWS::StackName}-MyEventsHandler
      Runtime: nodejs14.x
      MemorySize: 128
      Timeout: 10
      ReservedConcurrentExecutions: 2 # protect our 1000 concurrent executions account limit from runaway lambdas
      Policies:
        - SQSPollerPolicy:
            QueueName: !GetAtt MyEventsQueue.QueueName
      Events:
        SQSTrigger:
          Type: SQS
          Properties:
            Queue: !GetAtt MyEventsQueue.Arn
      Tags:
        PROJECT: 'YOUR-PROJECT-NAME'
        PROVISIONEDBY: 'SAM/ CLOUDFORMATION'

  # allows SNS to publish to this SQS queue
  SnsToSqsPolicy:
    Type: AWS::SQS::QueuePolicy
    Properties:
      PolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Sid: "Allow SNS publish to SQS"
            Effect: Allow
            Principal: 
              Service: "sns.amazonaws.com"
            Resource: !GetAtt MyEventsQueue.Arn
            Action: SQS:SendMessage
            Condition:
              ArnEquals:
                aws:SourceArn: !Ref MyEventsSNSTopic
      Queues:
        - Ref: MyEventsQueue

Outputs:
  MySqsQueueArn:
    Description: SQS queue ARN
    Value: !GetAtt MyEventsQueue.Arn
  MyEventsSNSTopicArn:
    Description: SNS topic ARN
    Value: !Ref MyEventsSNSTopic

[MrSinisterX]

https://gist.github.com/MrSinisterX/5ed2ef3089e3be25430b932d88774f5f

I faced the same issue and implemented this ugly but working solution.

[LinusU]

JavaScript version for anyone that might need it:

// FIXME: https://github.com/aws/aws-sdk/issues/15
function sqsQueueUrlFromArn (arn) {
  const parts = arn.split(':')

  const service = parts[2]
  const region = parts[3]
  const accountId = parts[4]
  const queueName = parts[5]

  return `https://${service}.${region}.amazonaws.com/${accountId}/${queueName}`
}

---

@srchase have you heard anything back from the Lambda team?

[jcollum-nutrien]

@srchase it's been almost 5 years! People have been asking for updates for 2 years. Any news?

[varqasim]

To whoever is insterested. I created this helper function that takes two parameters, the account ID & the queue ARN if you have them in hand & this generates the queue URL. I use this for integration testing purposes

export function getQueueUrl(accountId: string, queueArn: string) {
  const regex = new RegExp(`${accountId}:`);
  const queueName = queueArn.split(regex)[1];
  return `https://sqs.${process.env.AWS_REGION}.amazonaws.com/${accountId}/${queueName}`
} 

Here is an example of how I use it in my tests

let accountId: string;
beforeAll(async () => {
  const sts = new STS({ region: process.env.AWS_REGION });
  const { Account: account } =  await sts.getCallerIdentity({}).promise();
  accountId = account!;
});

test('myTest', async () => {
    const lambdaFuncESM = await lambdaClient
      .listEventSourceMappings({ FunctionName: 'myFuncName' }).promise();
    const queueArn = lambdaFuncESM.EventSourceMappings![0].EventSourceArn!;
    const queueUrl = getQueueUrl(accountId, queueArn);

    console.log({ queueArn, queueUrl });
});

[ashishdhingra]

P87037653

[ashishdhingra]

Thank you for your feedback. We've let the responsible internal team know and they've placed this on their backlog. We can't share specific timelines on when this might be implemented in GitHub; you should monitor the AWS News Blog for updates.

[github-actions]

This issue is now closed.

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.