AMQP 1.0 Shovels: expose additional capabilities needed for successful connection to some AMQP 1.0 brokers

[madmalkav]

When connecting to some AMQP 1.0 brokers, we cannot set capabilities on the shovel, so in some scenarios we are unable to give the shovel the proper configuration. In example, for IBM MQ , if you want to use a queue but don't set capabilities to queue, it will default to believe the object name provided is for a topic:

This is documented on IBM website:

I was able to replicate this same behavior with QPid Proton and this code sample, this code doesn't set capabilities so the same behavior happens:

from __future__ import print_function, unicode_literals
from proton import Message
from proton.handlers import MessagingHandler
from proton.reactor import Container

class HelloWorld(MessagingHandler):
    def __init__(self, server, address):
        super(HelloWorld, self).__init__()
        self.server = server
        self.address = address

    def on_start(self, event):
        print("connecting ...")
        conn = event.container.connect(self.server, sasl_enabled=True, allowed_mechs="PLAIN", allow_insecure_mechs=True, user="app", password="passw0rd")
        event.container.create_sender(conn, self.address)

    def on_sendable(self, event):
        print("sending command")
        event.sender.send(Message(body="Hello World!", address="DEV.QUEUE.2", content_type="text/plain"))
        event.sender.close()
        event.connection.close()

Container(HelloWorld("queue://172.17.0.2:5672", "DEV.QUEUE.2")).run()

This code set capabilities so it works OK:

from __future__ import print_function, unicode_literals
from proton import Message, symbol
from proton.handlers import MessagingHandler
from proton.reactor import Container, SenderOption

class HelloWorld(MessagingHandler):
    def __init__(self, server, address):
        super(HelloWorld, self).__init__()
        self.server = server
        self.address = address

    def on_start(self, event):
        print("connecting ...")
        conn = event.container.connect(self.server, sasl_enabled=True, allowed_mechs="PLAIN", allow_insecure_mechs=True, user="app", password="passw0rd")
        event.container.create_sender(conn, self.address, options=CapabilityOptions())

    def on_sendable(self, event):
        print("sending command")
        event.sender.send(Message(body="Hello World!", address="DEV.QUEUE.2", content_type="text/plain"))
        event.sender.close()
        event.connection.close()

class CapabilityOptions(SenderOption):
    def apply(self, sender):
        sender.target.capabilities.put_object(symbol("queue"))

Container(HelloWorld("amqp://172.17.0.2:5672", "DEV.QUEUE.2")).run()

Describe the solution you'd like

Some way to be able to set those capabilities on the shovel configuration.

Describe alternatives you've considered

No response

Additional context

No response

[ansd]

Hey @madmalkav, if that's what IBM MQ requires, feel free to create PR to add a capabilities field to the shovel configuration and pass that value through to the AMQP 1.0 Erlang client when it sends the ATTACH frame.

[madmalkav]

I have no knowledge of coding in general or Erlang in particular (I’m still surprised I got those Python samples to work…), so sending a PR is way above my skillset 😅

[michaelklishin]

Thank you for providing enough details and executable versions. It's a fair feature to request, we certainly would like to have a more complete Shovel support for IBM MQ targets. However, we cannot promise any particular ETA date.

[MarcialRosales]

Hi @madmalkav , we already have a PR ready which addresses this issue. However, we are having some problems with IBM MQ server we are using for testing it. We are building a docker image (ibm-mqadvanced-server-dev:9.3.5.1-amd64) as per this instructions from IBM. This is necessary because the official docker image does not come with AMQP capability built-in or enabled.

The problem I am having is that sometimes the ibm mq container starts up healthy, without any errors, but after our amqp10_client sends the session establishment frame, it returns with an error amqp:unauthorized-access AMQXR0100E: A connection from 192.168.65.1 was not authorized..
In the mq log /mnt/mqm/data/qmgrs/QM1/errors/amqp_0.log , I have this log statement A connection from 192.168.65.1 for client identifier 'ibmmqtest' was rejected for channel SYSTEM.DEF.AMQP because of CONNAUTH or CHLAUTH rule configuration. MQCC 2, MQRC 2330 MQRC_CODED_CHAR_SET_ID_ERROR . This indicates that the charset conversion from CCSID to Java charset (IBM MQ AMQP service is a java service) is failing. Which is odd, because our amqp10_client is always sending the same userid , password and client_id. And in case, everything is UTF-8 encoded which is compatible with the CCSI of the QMGR which is CCSID(819).
If I restart the amqp service via STOP SERVICE(SYSTEM.AMQP.SERVICE) and START SERVICE(SYSTEM.AMQP.SERVICE) followed by START CHANNEL(SYSTEM.DEF.AMQP), the mq server seems to work and accept and authenticate the connections.

Which MQ server version are you running with?

Would you mind sharing your mq commands that set up your AMQP service? I am using exactly these ones provided by IBM.

By the way, I can share with you an internal docker image of RabbitMQ with this issue fixed so that you can test it as well. Let me know if you want it.

[madmalkav]

First of all, thanks for looking into this.

Last time I tested this I tried with IBM MQ 9.3.5.0 . I attach to this comment a .txt file where I made my notes back in the day, hope it helps. You will see for my tests I disabled security on the CHLAUTH as it wasn't relevant for my tests back in the day and IBM MQ auth/security can be a PITA. In any case looking also on AMQERR01.LOG (and 02 and 03), and if necessary enabling tracing (link available in the txt) can help debug the issue.

Lastly, don't hesitate to contact me on Discord or whatever other means if you want to have a quick chat about the topic, I'll love to help as much as possible to debug this issue.

amqp10 test.txt

PS: CCSID of the queue manager shouldn't have any influence in this stuff if the apps are not explicitly using CONVERT option, they should work ok sending/receiving messages on any charset. But nothing I have too much experience with, those kind of CONVERT options are rarely used and in all my years working with IBM MQ no problem that initially looked like a CCSID problem was really a CCSID problem in the end. "It's never Lupus", like on that TV show :)

[MarcialRosales]

Thank you very much @madmalkav for sharing your notes !
First of all, I have noticed we are both building the same docker image except that you are not running the set up commands provided by the IBM guide. Instead you only run these commands in order to configure AMQP:

ALTER CHANNEL(SYSTEM.DEF.AMQP) CHLTYPE(AMQP) MCAUSER('app')
DEFINE CHANNEL(AMQP_TEST) CHLTYPE(AMQP) LIKE(SYSTEM.DEF.AMQP)
START CHANNEL(AMQP_TEST)
SET CHLAUTH('AMQP_TEST') TYPE(ADDRESSMAP) ADDRESS('*') USERSRC(MAP) MCAUSER('app') ACTION(ADD)
END

You mention that you disable CHLAUTH for the AMQP channel. I am using a very similar command (SET CHLAUTH(SYSTEM.DEF.AMQP) TYPE(ADDRESSMAP) ADDRESS(*) MCAUSER('app') ACTION(REPLACE)) except that I do not have USERSRC(MAP) and I am setting the default AMQP cnannel. I am not declaring a separate AMQP channel.

I have also noticed that you are not running the command START SERVICE(SYSTEM.AMQP.SERVICE). I thought this one was necessary.

I will try to follow your notes to see if I have more luck with them :) than with the IBM ones.

Definitely, I will contact you on Discord. Thanks a lot !!

[madmalkav]

By the way, I tested with IBM MQ because at the moment I was working on a project with a real interest on that particular case, IBM MQ -> RabbitMQ, but maybe testing with other AMQP 1.0 brokers may provide some useful info, i.e. some Azure Service Bus free tier stuff?

[MarcialRosales]

IBM MQ is quite common specially in large organizations therefore we need to ensure that our internal amqp10_client is able to talk to IBM MQ. Addressing this issue was not complicated but testing it against IBM MQ has been really complicated.

About testing against Azure Service Bus it is indeed necessary too. Thanks for. mentioning it. We do test against Azure in other areas of RabbitMQ such as OAuth 2.0 authentication.

[michaelklishin]

@madmalkav we do test our AMQP 1.0 implementation (plus features that use it, such as Shovel) against Azure Service Bus (although not on an ongoing basis, every so often) and we continuously test against ActiveMQ.

[madmalkav]

Ok, so it’s definitely something with IBM MQ. I’m starting some holidays but will try to compile RMQ with the last version of the PR and replicate on my side to see if something rings a bell.El 8 ago 2024, a las 17:03, Michael Klishin ***@***.***> escribió: @madmalkav we do test our AMQP 1.0 implementation (plus features that use it, such as Shovel) against Azure Service Bus (although not on an ongoing basis, every so often) and we continuously test against ActiveMQ. —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: ***@***.***>