Hangs at image_hub.recv_image() if the stream was started before the image_hub was initialized

[drhoffma]

Hi Jeff,

Dave from PyImageSearch here. I'm loving imagezmq...it will make appearances in Adrian's book.

I'm wondering how we can improve the ImageHub so that if a stream server is already running on a Pi, and yet the image_hub client (on a Mac or other box) is restarted, it will pick up seamlessly.

Currently, my client code on a Mac looks like your example:

image_hub = imagezmq.ImageHub()

while True:
	# read the next frame from the stream
	_, frame = image_hub.recv_image()
	image_hub.send_reply(b'OK')

	# Do something

It hangs at image_hub.recv_image() if I were to add/fix functionality to my program and restart it.

Do you have any ideas, or is it better for the Pi to timeout when it doesn't receive the ack "OK"?

All the best,
Dave Hoffman

[jeffbass]

Hi Dave,
Short answer is no, I don't know how to restart an imagezmq server without restarting its clients. I restart Pi clients by having the Pi timeout and restart the imagenode program. When testing, I set the Patience option to a low value (5 seconds) for quick timeouts. I use the systemd / systemctl service setup with restart set, so imagenode clients restart themselves if the server is down. There is an example imagenode.service file in the imagenode repository. In production, I have multiple imagehubs with 8-10 Pi's on each and this system is very reliable. Pi imagenodes restart quickly after power outages or other issues and imagehub servers restart very, very seldom (but the Pi's restart quickly when they do), such as when I update the server software.

This is actually a design choice made by the ZMQ team for the simplest REQ / REP pattern (which is the one imagezmq currently uses). Quoting from the ZMQ docs:

If you kill the server (Ctrl-C) and restart it, the client won't recover properly. Recovering from crashing processes isn't quite that easy. Making a reliable request-reply flow is complex enough that we won't cover it until Chapter 4 - Reliable Request-Reply Patterns.

In the ZMQ "simplest" REQ / REP pattern, clients can restart all they want to and things keep running. But, if the server restarts, clients need to restart. With my systemd service setup on imagenodes this works very reliably for me.

The imagehubs (imagezmq servers) in my farm system are laptops (which have built-in laptop battery backup), so they run for months without failing, even through brief power outages. So I have chosen not to use one of ZMQ's recommended "more reliable, but more complex" REQ / REP patterns.

I am going to leave this issue open for further discussion. Feel free to ask further questions. Or suggest an improvement that works for you.

Thanks for your question,
Jeff

[youngsoul]

Hey Dave,

I had a similar question so I forked Jeff's repo and added a send and receive timeout to the ZMQ socket. After the timeout expired, there is an exception that is thrown, then in my client I handle that exception by creating a new ImageSender.

It has worked pretty well for me. I have killed the server for hours, then restarted it and the clients automatically reconnected.

You can find my fork here: https://github.com/youngsoul/imagezmq

In the CHANGES.md file, I point you to a client that I used to handle the re-connect. You certainly do not have to handle the exception from 'send_image' like I do. The idea is to create a new ImageSender instance to reconnect - however you would like to do that.

I hope this helps - you can let me know if you have any questions.

Pat

[jeffbass]

Thanks for your comment and your fork, Pat. I am going to leave this issue open so others can easily see and benefit from what you've done. Thanks for providing the link to your solution.

[jeffbass]

@youngsoul Hey Pat, I would like to add a list of "Interesting and Helpful imagezmq Forks for Other Projects" section to the imagezmq documentation. It is for forks like yours that are great improvements, but not related to my yin-yang-ranch project (which is why I am building imagezmq). Would that be OK with you? I will put the new section into a pull request and tag you to it. And I would specifically mention your fork and tag that also. Thanks, Jeff.

[youngsoul]

Hi Jeff absolutely - happy to contribute in any way I can. Thanks

…

On Mon, Jun 3, 2019 at 11:10 AM Jeff Bass ***@***.***> wrote: @youngsoul <https://github.com/youngsoul> Hey Pat, I would like to add a list of "Interesting and Helpful imagezmq Forks for Other Projects" section to the imagezmq documentation. It is for forks like yours that are great improvements, but not related to my yin-yang-ranch project (which is why I am building imagezmq). Would that be OK with you? I will put the new section into a pull request and tag you to it. And I would specifically mention your fork and tag that also. Thanks, Jeff. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1?email_source=notifications&email_token=AADPH67NRO4XPR3PSNHR67TPYU655A5CNFSM4GXX3QR2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODWZ42FQ#issuecomment-498322710>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AADPH64JRKDR2OU6Q6L765DPYU655ANCNFSM4GXX3QRQ> .

[jeffbass]

@youngsoul Hi Pat, I have added a "Helpful Forks of imagezmq" section to the README.rst in a development branch. Yours is the first one (Thanks!). Could you take a look at the branch "add-helpful-forks" and check the README that shows up on that branch. The new section is toward the bottom of the README. If it is OK with you, I'll merge it into master.
Thanks,
Jeff

[youngsoul]

Hi Jeff It looks good on the branch to me. Thanks for including my contribution. Pat

…

On Fri, Mar 13, 2020 at 4:16 PM Jeff Bass ***@***.***> wrote: @youngsoul <https://github.com/youngsoul> Hi Pat, I have added a "Helpful Forks of imagezmq" section to the README.rst in a development branch. Yours is the first one (Thanks!). Could you take a look at the branch "add-helpful-forks" and check the README that shows up on that branch. The new section is toward the bottom of the README. If it is OK with you, I'll merge it into master. Thanks, Jeff — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AADPH65NYBWMC2KYFXFOJKDRHKPDHANCNFSM4GXX3QRQ> .

[jeffbass]

Thanks, Pat. Merged add-helpful-forks into Master.
Jeff

[jeffbass]

Hi @drhoffma & @youngsoul ! I've added a lot of this discussion to the FAQ in the docs and I've added the link to your great fork, Pat. So I'm closing this issue.