RFC: Erlang interaction issues and related. #114

Closed
nwalker opened this Issue Jan 14, 2013 · 10 comments

Comments

Projects
None yet
3 participants
@nwalker
Contributor

nwalker commented Jan 14, 2013

In my current project I've got a need of easy and lightweight communication between Erlang- and Python-based parts. Python part is Flask-based web interface to RDBMS, Erlang part is notification router and other transport facilities. So, I started with experiment.

Experiment

I made two Erlang nodes, ev@asgard for Erlang VM instance and uwsgi@asgard for uWSGI instance loaded. Python code in UWSGI was something like

import uwsgi

# outgoing RPC block
testnode = uwsgi.erlang_connect('ev@local')  
print('erlang: ', uwsgi.erlang_rpc(testnode, 'io', 'format', ['hello~n']))
# outgoing RPC block end

# incoming RPC block
def func(*args, **kwargs):
    print('rpc:', args, kwargs)
    return 'test'
uwsgi.register_rpc(func, 'hello')

Just a test code. And now comes something interesting:

  • starting ev@local - ok
  • starting uwsgi@local - ok
Erlang C-Node uwsgi@asgard registered on port 53763
uWSGI http bound on :8000 fd 8
uwsgi socket 0 bound to TCP address 127.0.0.1:55314 (port auto-assigned) fd 7
Python version: 2.7.3 (default, Sep 26 2012, 21:53:55)  [GCC 4.7.2]
*** Python threads support is disabled. You can enable it with --enable-threads ***
Python main interpreter initialized at 0x238d920
enabled Python<->Erlang bridge
your server socket listen backlog is limited to 100 connections
mapped 217176 bytes (212 KB) for 2 cores
*** Operational MODE: preforking ***
('RPC:', u'ok')
registered RPC function hello
  • switching to Erlang console, 'hello' message is there - ok
  • trying RPC - fail:
(ev@asgard)1> rpc:call('uwsgi@asgard', unused, hello, ["test"]).
error 2013-01-14 18:54:48 ** Node uwsgi@asgard not responding **
** Removing (timedout) connection **

info 2013-01-14 18:54:48 Starting inet_gethost_native_sup:undefined <0.439.0>
info 2013-01-14 18:54:48 Starting kernel_safe_sup:inet_gethost_native_sup <0.438.0>
{badrpc,nodedown}
  • trying again - looks fine:
(ev@asgard)2> rpc:call('uwsgi@asgard', unused, hello, ["test"]).
"test"
  • switching to uwsgi - RPC arguments parsed incorrectly:
[erlang] message From: ev@asgard To (process): rex
('call: ', (), {})
ei_send to 8 ev@asgard 48 0 1: 38 0
  • removing RPC outgoing block from Python code fixes that "nodedown" issue!
  • trying to use uwsgi.erlang_connect after successful receive of any message from Erlang raises ValueError.

Conclusions

There is two issues in Erlang interactions:

  • minor - incorrect RPC argument parsing.
  • major - currently there is no possibility to do full duplex RPC between uwsgi and Erlang VM.
  • major, not mentioned before - uwsgi handles only one external node connection. If we try to launch 2nd Erlang node and do RPC to uwsgi, it will fail, even(and especially) after successful established connection from first Erlang node to uwsgi.

RPC args issue

RPC args passed as list of Erlang terms but parsing code converts only single atom or string, so there is no way to call function of arity more than 1. Fix is trivial, IMHO, I'll do it fast enough.

2nd Erlang node issue

Cause is obvious enough. There is only one accepted connection handled at time. The only way to fix is same with next issue.

Full duplex RPC issue

It looks like Erlang VM reuses established connection for interaction and blocks any new connection attempts. Starting from there I have very few ideas. It looks like there should be connection pool stored somewhere, connections from pool should be used for duplex communications and all connections from pool should be polled for new messages. I can imagine, how this should work, but it only raises new questions:

  • obviously, uwsgi already have some socket polling facility. But, how it should be used for described case? Where should I look for example?
  • how should RPC handler be implemented - threaded? Thread per node connection or fixed size thread pool for all connections? Will threads cause problems with Python interpreter instances?
  • how sockets should be passed between processes? Example: node connection created from Python code in one process and socket should be passed to node process, where it will be polled.

Maybe I'm able to implement this facility, but I need some help with desing decisions and uwsgi internals.

@unbit

This comment has been minimized.

Show comment Hide comment
@unbit

unbit Jan 14, 2013

Owner

Just one thing, the uWSGI RPC subsystem is string based (it accepts only positional string arguments) so

def foobar(one, two, three):
with one two and threee as strings is ok,

but

def foobar(_args, *_kwargs):

is not ok

http://projects.unbit.it/uwsgi/wiki/RPC

Regarding full duplex, the libel/erl libraries are not very versatile. The long-term goal is avoiding them and directly using OTP transports.

As it looks like very few users are interested in the erlang integration (in-fact only a python-erlang bridge was implemented) development slowed down (the current plugin is the same of the 0.9 version).

If you are interested in making reports (or helping in the development) i (and at least 2 other people) will be very happy to start improving the plugin.

Owner

unbit commented Jan 14, 2013

Just one thing, the uWSGI RPC subsystem is string based (it accepts only positional string arguments) so

def foobar(one, two, three):
with one two and threee as strings is ok,

but

def foobar(_args, *_kwargs):

is not ok

http://projects.unbit.it/uwsgi/wiki/RPC

Regarding full duplex, the libel/erl libraries are not very versatile. The long-term goal is avoiding them and directly using OTP transports.

As it looks like very few users are interested in the erlang integration (in-fact only a python-erlang bridge was implemented) development slowed down (the current plugin is the same of the 0.9 version).

If you are interested in making reports (or helping in the development) i (and at least 2 other people) will be very happy to start improving the plugin.

@maxlapshin

This comment has been minimized.

Show comment Hide comment
@maxlapshin

maxlapshin Jan 15, 2013

There is no use in trying to wrap libei.

I've written this in several days: https://github.com/maxlapshin/erlang it is a ruby binding to erlang.

There is no use in trying to wrap libei.

I've written this in several days: https://github.com/maxlapshin/erlang it is a ruby binding to erlang.

@unbit

This comment has been minimized.

Show comment Hide comment
@unbit

unbit Jan 15, 2013

Owner

You mean that even if you have implemented otp parsing, you do not think it is a good idea to throw away libei or the opposite ?

Owner

unbit commented Jan 15, 2013

You mean that even if you have implemented otp parsing, you do not think it is a good idea to throw away libei or the opposite ?

@maxlapshin

This comment has been minimized.

Show comment Hide comment
@maxlapshin

maxlapshin Jan 15, 2013

I think you should drop away libei and do erlang syntax parsing by your own hands.
In fact there are implementations of BERT in python, so it would be easy. Managing connection to node is rather easy unless you want to switch from C-node to real node.

So, drop libei and write all network transport in Python.

On Jan 15, 2013, at 2:54 PM, unbit wrote:

You mean that even if you have implemented otp parsing, you do not think it is a good idea to throw away libei or the opposite ?


Reply to this email directly or view it on GitHub.

I think you should drop away libei and do erlang syntax parsing by your own hands.
In fact there are implementations of BERT in python, so it would be easy. Managing connection to node is rather easy unless you want to switch from C-node to real node.

So, drop libei and write all network transport in Python.

On Jan 15, 2013, at 2:54 PM, unbit wrote:

You mean that even if you have implemented otp parsing, you do not think it is a good idea to throw away libei or the opposite ?


Reply to this email directly or view it on GitHub.

@unbit

This comment has been minimized.

Show comment Hide comment
@unbit

unbit Jan 15, 2013

Owner

Ok, that was what i want to hear. I leave the issue open as a tracker of future development

Owner

unbit commented Jan 15, 2013

Ok, that was what i want to hear. I leave the issue open as a tracker of future development

@nwalker

This comment has been minimized.

Show comment Hide comment
@nwalker

nwalker Jan 15, 2013

Contributor

Being honest - I don't understand, why not to use libei.
uwsgi written in C and main part of Erlang interaction lies in C code. The only reason I can imagine, that ei_accept/ei_send/ei_receive are not compatible with polling API like epoll - and I'm not sure about such incompatibility.

BTW, I've updated Erlang plugin for correct handling of rpc:call() from outside. Gist with patch. Currently it's a bit not finished. Maybe I've done too much of implicit conversions(currently, it converts integers, floats, strings and atoms to strings), but IMO it's nice enough already.
But, I've got some two big questions:

  • incoming strings and binaries should be limited in size. What should I use as limit?
  • when RPC arguments should be deallocated? Currently, this plugin leaks memory like crazy. It can be tested with two other files in gist I linked. So, free() on argv contents should be called in erlang.c:uwsgi_erlang_rpc right after uwsgi_rpc() call? Or, maybe arguments should be destroyed in the end of core/rpc.c:rpc_call? Yes, this is just stupid question.
Contributor

nwalker commented Jan 15, 2013

Being honest - I don't understand, why not to use libei.
uwsgi written in C and main part of Erlang interaction lies in C code. The only reason I can imagine, that ei_accept/ei_send/ei_receive are not compatible with polling API like epoll - and I'm not sure about such incompatibility.

BTW, I've updated Erlang plugin for correct handling of rpc:call() from outside. Gist with patch. Currently it's a bit not finished. Maybe I've done too much of implicit conversions(currently, it converts integers, floats, strings and atoms to strings), but IMO it's nice enough already.
But, I've got some two big questions:

  • incoming strings and binaries should be limited in size. What should I use as limit?
  • when RPC arguments should be deallocated? Currently, this plugin leaks memory like crazy. It can be tested with two other files in gist I linked. So, free() on argv contents should be called in erlang.c:uwsgi_erlang_rpc right after uwsgi_rpc() call? Or, maybe arguments should be destroyed in the end of core/rpc.c:rpc_call? Yes, this is just stupid question.
@maxlapshin

This comment has been minimized.

Show comment Hide comment
@maxlapshin

maxlapshin Jan 15, 2013

I had to rewrite it in Ruby because I want event infrastructure (like twisted) and it was no achieved in libei exactly
because ei_accept/send/receive lives in their own world, not in epoll

Next reason is, of course, memory management =)

Speaking about limit: I haven't implemented it, but why not to make global module constant?

On Jan 15, 2013, at 9:30 PM, nwalker wrote:

Being honest - I don't understand, why not to use libei.
uwsgi written in C and main part of Erlang interaction lies in C code. The only reason I can imagine, that ei_accept/ei_send/ei_receive are not compatible with polling API like epoll - and I'm not sure about such incompatibility.

BTW, I've updated Erlang plugin for correct handling of rpc:call() from outside. Gist with patch. Currently it's a bit not finished. Maybe I've done too much of implicit conversions(currently, it converts integers, floats, strings and atoms to strings), but IMO it's nice enough already.
But, I've got some two big questions:

incoming strings and binaries should be limited in size. What should I use as limit?
when RPC arguments should be deallocated? Currently, this plugin leaks memory like crazy. It can be tested with two other files in gist I linked. So, free() on argv contents should be called in erlang.c:uwsgi_erlang_rpc right after uwsgi_rpc() call? Or, maybe arguments should be destroyed in the end of core/rpc.c:rpc_call? Yes, this is just stupid question.

Reply to this email directly or view it on GitHub.

I had to rewrite it in Ruby because I want event infrastructure (like twisted) and it was no achieved in libei exactly
because ei_accept/send/receive lives in their own world, not in epoll

Next reason is, of course, memory management =)

Speaking about limit: I haven't implemented it, but why not to make global module constant?

On Jan 15, 2013, at 9:30 PM, nwalker wrote:

Being honest - I don't understand, why not to use libei.
uwsgi written in C and main part of Erlang interaction lies in C code. The only reason I can imagine, that ei_accept/ei_send/ei_receive are not compatible with polling API like epoll - and I'm not sure about such incompatibility.

BTW, I've updated Erlang plugin for correct handling of rpc:call() from outside. Gist with patch. Currently it's a bit not finished. Maybe I've done too much of implicit conversions(currently, it converts integers, floats, strings and atoms to strings), but IMO it's nice enough already.
But, I've got some two big questions:

incoming strings and binaries should be limited in size. What should I use as limit?
when RPC arguments should be deallocated? Currently, this plugin leaks memory like crazy. It can be tested with two other files in gist I linked. So, free() on argv contents should be called in erlang.c:uwsgi_erlang_rpc right after uwsgi_rpc() call? Or, maybe arguments should be destroyed in the end of core/rpc.c:rpc_call? Yes, this is just stupid question.

Reply to this email directly or view it on GitHub.

@nwalker

This comment has been minimized.

Show comment Hide comment
@nwalker

nwalker Jan 15, 2013

Contributor

ei_accept/send/receive lives in their own world, not in epoll

Looks like it's time to look into libei sources.

why not to make global module constant?

I was just unable to find a value used for other RPC sources.
Yes, I can use maximal value of uint16_t which used for argument length, but prefered to ask first - IMO things like that should be unified.

Contributor

nwalker commented Jan 15, 2013

ei_accept/send/receive lives in their own world, not in epoll

Looks like it's time to look into libei sources.

why not to make global module constant?

I was just unable to find a value used for other RPC sources.
Yes, I can use maximal value of uint16_t which used for argument length, but prefered to ask first - IMO things like that should be unified.

@unbit

This comment has been minimized.

Show comment Hide comment
@unbit

unbit Jan 24, 2013

Owner

i have managed to implement a good part of otp terms in c (yes, that was pretty easy). I think i will release a general library instead of embedding it into uWSGI. Once it will be complete we can start working on a better uWSGI plugin.

I think i will be able to commit something useful in the next week (sorry, not much free time as it is mainly a personal effort, as in my company there are no more erlang nodes :( )

Owner

unbit commented Jan 24, 2013

i have managed to implement a good part of otp terms in c (yes, that was pretty easy). I think i will release a general library instead of embedding it into uWSGI. Once it will be complete we can start working on a better uWSGI plugin.

I think i will be able to commit something useful in the next week (sorry, not much free time as it is mainly a personal effort, as in my company there are no more erlang nodes :( )

@unbit

This comment has been minimized.

Show comment Hide comment
@unbit

unbit Dec 5, 2014

Owner

Closing it, rpc->otp will (eventually) be managed in a third party plugin if someone has interest to do it (or some unbit customer will ask for it)

Owner

unbit commented Dec 5, 2014

Closing it, rpc->otp will (eventually) be managed in a third party plugin if someone has interest to do it (or some unbit customer will ask for it)

@unbit unbit closed this Dec 5, 2014

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment