Azure Service Bus Python message generator

[aafanasyev]

Hi *,

I would like to sent a as much as possible messages to service bus and then read by multiple instances from a queue.

Based on this example: http://pymotw.com/2/multiprocessing/communication.html#process-pools
I created this script:

#some SB connection code above

queue_name = ConfigSectionMap("Queue")["queue_name"]
messages_quantity = int(ConfigSectionMap("Messages")["message_quantity"])
message_size = urandom(int(ConfigSectionMap("Messages")["message_size"]))

def send_messages(msg_q):
    #for i in xrange(messages_quantity):
    msg = Message (message_size)
    msg.custom_properties={'id':msg_q}    
    bus_service.send_queue_message(queue_name, msg)
    return msg.custom_properties
    #print(msg.body)

def start_process():
    print 'Starting', multiprocessing.current_process().name

if __name__ == '__main__':
    inputs = list(range(messages_quantity))
    print 'Input   :', inputs

    builtin_outputs = map(send_messages, inputs)
    print 'Built-in:', builtin_outputs

    pool_size = multiprocessing.cpu_count() * 2
    pool = multiprocessing.Pool(processes=pool_size,
                                initializer=start_process,
                                )

    pool_outputs = pool.map(send_messages, inputs)
    pool.close() # no more tasks
    pool.join()  # wrap up current tasks

    print 'Pool    :', pool_outputs

However, I am not sure if it working right. Or is it possible make the multiprocessing part much more optimal?

Further, where I can find examples of information about sending message properties and headers with python?

Thank you!

[zooba]

This is probably best asked at https://github.com/Azure/azure-sdk-for-python, but @huguesv may be able to answer you here to save you moving the post again.

[huguesv]

For an example of sending message properties:
https://github.com/Azure/azure-sdk-for-python/blob/master/tests/test_servicebusservice.py#L464

For maximum performance, Send Message Batch would be better:
https://msdn.microsoft.com/en-us/library/azure/dn798894.aspx

It's a relatively recent addition that is not in the Python SDK yet (or any of the other non .NET languages as far as I can tell).
Azure/azure-sdk-for-python#261

If you decide to try to implement that batch function and have questions and/or want to contribute it, feel free to post in the above issue.

[aafanasyev]

@huguesv thank you! :)
About Batch I heard from this nice presentation: https://www.youtube.com/watch?v=kYPvFvR3NOw
But I did not know this wasn't implemented yet. At least I would like to use just http and messages, like from this slide(https://youtu.be/kYPvFvR3NOw?t=25m12s), maybe amqp is also nice, but I didn't found any information about using it with ASB and Python :(

Coming back to my example (just http connection if I am not wrong), I am not sure if it works now correct with sending a multiple messages at once from one machine. At least, I do not know how to check it in a wise manner. I do this mostly with wireshark (imagine a http benchmark software penetrating a webserver), and I could do check ampq (Just idea!)(https://wiki.wireshark.org/AMQP), by measurement of the increased amount of packages per minute.

Therefore, maybe some one have a suggestion how to implement a Python multiprocessing library in this example in much more visual and efficient way? Preferably that I could see it on wireshark level. Thank you.

P.S.: @zooba I think you are right about moving to https://github.com/Azure/azure-sdk-for-python. However, my question was basically more related to efficient, functional and visualised use of parallel processing in python and less to a azure message sending function. My apologises.

[zooba]

Ah, I see. You could also post a general question like this to python-list and get responses from the broader community.

For IO bound work, you can use concurrent.futures.ThreadPoolExecutor and keep everything in a single process. It's only really CPU-bound work that suffers from threading in Python - when you're waiting on network requests threads are fine.

Sending multiple messages simultaneously should be handled correctly by the HTTP protocol, so that each response is received by the correct requester. If this is broken, it's probably not something you can fix in your own code anyway - find a minimal repro that works against multiple different servers and report it as a Python bug.