Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Does not work with OTP/23's updated node communication set up #67

Closed
torifaye opened this issue Jul 4, 2020 · 19 comments
Closed

Does not work with OTP/23's updated node communication set up #67

torifaye opened this issue Jul 4, 2020 · 19 comments

Comments

@torifaye
Copy link

torifaye commented Jul 4, 2020 •
edited

Hello, I just came across this project and am looking to use it for something I'm working on but I can't seem to get any of the examples working. When I try running example 10, through make example10a and then make example10b in another terminal tab, I get this error on the elixir side:

16:52:10.878 [error] ** :"elixir@127.0.0.1": Connection attempt to node :"py@127.0.0.1" aborted since it cannot handle ['BIG_CREATION'].**

I've tried looking around on the internet for this issue but wasn't able to find anything describing this issue. My installation of elixir came with Erlang/OTP 23, and I see in the README it only lists support up to 21, could that be an issue?
@s2hc-johan
Copy link
Collaborator

s2hc-johan commented Jul 4, 2020 via email

@torifaye
Copy link
Author

torifaye commented Jul 4, 2020

Hi, I’m not using elixir, so I don’t know if it’s any difference in how connections are made. What versions are you using? /J

Here's the output of elixir --version for me

Erlang/OTP 23 [erts-11.0.2] [source] [64-bit] [smp:16:16] [ds:16:16:10] [async-threads:1] [hipe] [dtrace]

Elixir 1.10.3 (compiled with Erlang/OTP 22)

So Elixir 1.10.3 and Erlang/OTP 23

@torifaye
Copy link
Author

torifaye commented Jul 5, 2020

Hello,

I did a little bit of experimenting and it looks like it is due to an issue with OTP/23. I tried installing OTP/21 and running through the same test (python example 01 and then net_adm:ping('py@127.0.0.1'). in eshell) and had no issues with OTP/21 but had an issue with OTP/23 (i.e. got a pong from OTP/21, a pang from OTP/23). So I'm guessing the node connection logic between the two versions has changed and that's why I'm running into the error I'm having. This is a little bit beyond the scope of this issue, but do you have any idea what this might be? And what it would involve to have this library support OTP/23? I am willing to help with such a task.

@s2hc-johan
Copy link
Collaborator

s2hc-johan commented Jul 5, 2020 via email

@torifaye torifaye changed the title Examples are not working Does not work with OTP/23 Jul 5, 2020
@torifaye
Copy link
Author

torifaye commented Jul 5, 2020

Sure thing, I'm going to try and take a look into that in my spare time this next week. I've changed the name of the issue to more accurately reflect the problem

@torifaye torifaye changed the title Does not work with OTP/23 Does not work with OTP/23's updated node communication set up Jul 5, 2020
@yosh yosh mentioned this issue Jul 11, 2020
Merged
@yosh
Copy link
Contributor

yosh commented Jul 11, 2020

I noticed this a couple months ago, with the R23 release candidates. There's a dependent PR for Term as well.

@s2hc-johan
Copy link
Collaborator

@RobertYost I just merged @yosh PR in pyrlang and term, could you update and test if it works as expected for you know?

@agundy
Copy link
Contributor

agundy commented Jul 19, 2020

I have Erlang/OTP 23 and just tested. With the latest Term build it works with the Rust native codec but without the Rust I was getting an error in the Python.

2020-07-19 11:43:43,647 [pyrlang.dist_proto.base_dist_protocol] base_dist_protocol:201: out 21: b'r\x08\x99\xbb\xda@\x9e\x87\x06\x910\x90\xf3\x8a}{\xabv\xd5:\xab'
2020-07-19 11:43:43,648 [pyrlang.dist_proto.base_dist_protocol] base_dist_protocol:173: unconsumed: b'', state=recv_challenge_ack
2020-07-19 11:43:43,648 [pyrlang.dist_proto.base_dist_protocol] base_dist_protocol:423: Connected to erl@127.0.0.1
2020-07-19 11:43:43,649 [pyrlang.dist_proto.client] client:141: Outgoing dist connection: established with erl@127.0.0.1
2020-07-19 11:43:43,649 [pyrlang.dist_proto.base_dist_protocol] base_dist_protocol:173: unconsumed: b'', state=conn
2020-07-19 11:43:43,651 [asyncio] base_events:1707: Fatal error: protocol.data_received() call failed.
protocol: <pyrlang.dist_proto.client.DistClientProtocol object at 0xffff83070220>
transport: <_SelectorSocketTransport fd=9 read=polling write=<idle, bufsize=0>>
Traceback (most recent call last):
  File "/usr/lib/python3.8/asyncio/selector_events.py", line 860, in _read_ready__data_received
    self._protocol.data_received(data)
  File "/home/aaron/workspace/pyrlang/Pyrlang/pyrlang/dist_proto/base_dist_protocol.py", line 144, in data_received
    while self._data_received_inner():
  File "/home/aaron/workspace/pyrlang/Pyrlang/pyrlang/dist_proto/base_dist_protocol.py", line 171, in _data_received_inner
    self.on_packet(packet)                                 
  File "/home/aaron/workspace/pyrlang/Pyrlang/pyrlang/dist_proto/client.py", line 48, in on_packet
    return self.on_packet_connected(data)
  File "/home/aaron/workspace/pyrlang/Pyrlang/pyrlang/dist_proto/base_dist_protocol.py", line 401, in on_packet_connected
    (control_term, tail) = codec.binary_to_term(data[1:])
  File "/home/aaron/.local/lib/python3.8/site-packages/term/codec.py", line 40, in binary_to_term
    return co_impl.binary_to_term(data, opt)
  File "/home/aaron/.local/lib/python3.8/site-packages/term/py_codec_impl.py", line 120, in binary_to_term
    val, rem = binary_to_term_2(decode_data, options)
  File "/home/aaron/.local/lib/python3.8/site-packages/term/py_codec_impl.py", line 296, in binary_to_term_2
    term1, tail = binary_to_term_2(tail)
  File "/home/aaron/.local/lib/python3.8/site-packages/term/py_codec_impl.py", line 466, in binary_to_term_2
    raise PyCodecError("Unknown tag %d" % data[0])
term.py_codec_impl.PyCodecError: Unknown tag 0
2020-07-19 11:43:43,747 [pyrlang.node] node:354: Connected to erl@127.0.0.1
2020-07-19 11:43:43,749 [pyrlang.dist_proto.base_dist_protocol] base_dist_protocol:208: out 87: b'p\x83h\x04a\x06Xw\rpy1@127.0.0.1\x00\x00\x00\x00\x00\x00\x00\x02\x00\x00\x00\x00w\x00w\x08example8\x83h\x02w\x08registerXw\rpy1@127.0.0.1\x
00\x00\x00\x00\x00\x00\x00\x02\x00\x00\x00\x00'            
^CTraceback (most recent call last):
  File "examples/e08_pynode.py", line 89, in <module>
    main()                                                 
  File "examples/e08_pynode.py", line 85, in main
    n.run()                                                
  File "/home/aaron/workspace/pyrlang/Pyrlang/pyrlang/node.py", line 620, in run
    self._event_loop.run_until_complete(self.__completed_future)
  File "/usr/lib/python3.8/asyncio/base_events.py", line 603, in run_until_complete
    self.run_forever()                                     
  File "/usr/lib/python3.8/asyncio/base_events.py", line 570, in run_forever
    self._run_once()                                       
  File "/usr/lib/python3.8/asyncio/base_events.py", line 1823, in _run_once
    event_list = self._selector.select(timeout)
  File "/usr/lib/python3.8/selectors.py", line 468, in select
    fd_event_list = self._selector.poll(timeout, max_ev)
KeyboardInterrupt                                          
make: *** [Makefile:60: example8b] Interrupt

@yosh
Copy link
Contributor

yosh commented Jul 19, 2020

This is an off-by-one in the Python NEW_PID decode implementation. PR up at Pyrlang/Term#17, which includes test suite updates to catch this sort of thing.

@s2hc-johan
Copy link
Collaborator

s2hc-johan commented Jul 19, 2020 via email

@george-hopkins
Copy link

I tried to ping a Python node with OTP/23. @RobertYost: Were you able to ping a Python node with the code on the master branch?

$ iex --sname shell
Erlang/OTP 23 [erts-11.0.2] [source] [64-bit] [smp:2:2] [ds:2:2:10] [async-threads:1] [hipe]

Interactive Elixir (1.10.3) - press Ctrl+C to exit (type h() ENTER for help)
iex(shell@myhost)1> :net_adm.ping(:py@myhost)
:pang
INFO:pyrlang.dist_proto.server:RECV_NAME: (0, 5) shell@myhost
DEBUG:pyrlang.dist_proto.base_dist_protocol:out 3: b'sok'
INFO:pyrlang.dist_proto.server:Sending challenge (our number is 547689603) py@127.0.0.1
DEBUG:pyrlang.dist_proto.base_dist_protocol:out 23: b'n\x00\x05\x00\x07O\xbd \xa5\x14\x83py@127.0.0.1'
DEBUG:pyrlang.dist_proto.base_dist_protocol:unconsumed: b'', state=wait_ch_reply

@s2hc-johan
Copy link
Collaborator

don't know how ping works, but we needed to fix since the rpc isn't the same any more. Looking into a solution so that we're backwards compatible

@yosh
Copy link
Contributor

yosh commented Aug 28, 2020

@george-hopkins You're pinging the name py@myhost from iex, but the logs from the python side show the node name as py@127.0.0.1 ... those are two different node names. :net_adm.ping(:"py@127.0.0.1") is how you'd do it from Elixir.

@george-hopkins
Copy link

@yosh Thank you! Now the communication works flawlessly.

@s2hc-johan
Copy link
Collaborator

This works for us now anyway. Anyone else having issues with OTP23 or can I close this one?

@philipcristiano
Copy link

I'm still running into these problems with master for pyrlang-term Python implementation giving

Native term ETF codec library import failed, falling back to slower Python impl
Error: Handshake bad status: b'snot_allowed' (state recv_status)
Protocol requested to disconnect the socket

With rust I wind up getting the same thing minus the fallback notice

Error: Handshake bad status: b'snot_allowed' (state recv_status)
Protocol requested to disconnect the socket

On the Erlang/OTP 23 I see

** 'erl@127.0.0.1': Connection attempt to node 'py@127.0.0.1' aborted since it cannot handle ["BIG_CREATION"].**

@s2hc-johan
Copy link
Collaborator

hmm, maybe I've missed it in master. Could you try the dev_1.0 branch?

@philipcristiano
Copy link

Oh hm, it looks like I had a tag that didn't get overwritten when I tried to used master, it's working in master for me now. I'm so sorry for the noise!

@s2hc-johan
Copy link
Collaborator

well, glad it's working. I'll close this one then 😄

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@yosh @philipcristiano @george-hopkins @s2hc-johan @agundy @torifaye