Network communication: Pyro4 or another protocol?

[cylc]

Currently cylc uses Pyro3 as a rather minimal object oriented RPC interface, for network communication between cylc clients (tasks, commands, GUIs) and cylc servers (running suites).

Pyro3 is now in maintenance; Pyro4, which is compatible with Python 3, is recommended for new projects. In the future we will clearly have to upgrade to Pyro4, or switch to something else - possibly a custom protocol because cylc's needs in this respect are quite simple.

Unfortunately Pyro3's built in "connection authentication" is currently critical to cylc's operation (i.e. suite passphrases), and as of mid 2012 Pyro4 does not have any built in connection authentication. However, a mid-2011 post to the Pyro mailing list by the Pyro maintainer Irmen de Jong suggests that connection authentication is on his To Do list, and it will likely take a similar form to that in Pyro3. Options:

• wait on Pyro4 connection authentication before switching to Pyro4
• convert to Pyro4 regardless and build our own authentication mechanism into cylc
• drop Pyro and build our own network communication and authentication protocols

[matthewrmshin]

If we go for option 3, then it would probably make sense to introduce something language neutral. An API served via HTTP with a RESTful interface would give client developers the freedom to implement functionalities in whatever language.

[cylc]

I've had a quick discussion with Irmen de Jong on the pyro-core mailing list, and he now says he will put Pyro3-style connection authentication in the next-plus-one Pyro4 release:

The next release (4.15) will contain the sorely needed fix for the threadpool lockup
problem (and some other minor changes). I will release this first.
The version after that, 4.16, can be the first to contain a connection authentication
feature similar to that what Pyro3 has.
I'm not sure yet what the fate of the HMAC authentication will be.

Irmen puts up new Pyro releases on a pretty regular basis; given that, the easiest way forward for us is clearly to wait a little longer for Pyro 4.16, then switch to it and continue to use the simple encrypted passphrase authentication.

Unfortunately I know little about authentication or communication protocols, but I fear it might not be so easy to replace Pyro ourselves. Even if cylc's use of Pyro is quite minimal, what about multi-threading and handling multiple synchronous requests, for example - sounds difficult. Then, if we keep Pyro it might not be so hard to bolt on our own authentication mechanism, but is there any point in doing that? I don't think cylc has very strong security requirements, and the way we currently use connection authentication it is almost transparent to the user (other than deploying the auto-generated passphrase to remote task hosts).

[cylc]

Matt, you make a good point, but if we don't have to come up with something new anymore, then maybe that's something to consider in the longer term.

[m214089]

So, for the time being the pyro4 solution sounds good. Getting our own stuff there could be a real long term target.

[hjoliver]

Pyro 4.15 has just been released (26 Sept 2012). We can expect connection authentication in the next release (as per comments from Irmen de Jong above).

[m214089]

Hi Hilary,
that sounds great!
Cheerio,
Luis

On 26/09/2012 13:26, Hilary James Oliver wrote:

Pyro 4.15 has just been released (26 Sept 2012). We can expect
connection authentication in the next release (as per comments from
Irmen de Jong above).

—
Reply to this email directly or view it on GitHub
#72 (comment).

                          \\\\\\
                          (-0^0-)

--------------------------oOO--(_)--OOo-----------------------------

Luis Kornblueh Tel. : +49-40-41173289
Max-Planck-Institute for Meteorology Fax. : +49-40-41173298
Bundesstr. 53
D-20146 Hamburg Email: luis.kornblueh@zmaw.de
Federal Republic of Germany

[hjoliver]

Update: as far as I'm aware (although I haven't checked properly) connection authentication has not been put into Pyro 4 yet. And we have no immediate plans to move cylc to Python 3 in any case, or at least we still have higher priorities than that.

[hjoliver]

Update: we have pretty much settled on moving away from Pyro, to an HTTP-based system.

[hjoliver]

Closing this, nothing has changed here - we'll soon be moving ahead with HTTP.