Unsupported framed Thrift transports results in connection resets #6

Closed
teepark opened this Issue Jun 29, 2012 · 12 comments

3 participants

@teepark

happybase 0.3 with hbase 0.94.0 is getting its connection closed consistently:

>>> import happybase
>>> conn = happybase.Connection("localhost")
>>> t = conn.table("test")
>>> list(t.scan())
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "happybase/api.py", line 535, in scan
    scan_id = client.scannerOpenWithScan(self.name, scan)
  File "happybase/hbase/Hbase.py", line 1716, in scannerOpenWithScan
    return self.recv_scannerOpenWithScan()
  File "happybase/hbase/Hbase.py", line 1728, in recv_scannerOpenWithScan
    (fname, mtype, rseqid) = self._iprot.readMessageBegin()
  File "thrift/protocol/TBinaryProtocol.py", line 126, in readMessageBegin
    sz = self.readI32()
  File "thrift/protocol/TBinaryProtocol.py", line 203, in readI32
    buff = self.trans.readAll(4)
  File "thrift/transport/TTransport.py", line 58, in readAll
    chunk = self.read(sz-have)
  File "thrift/transport/TTransport.py", line 160, in read
    self.__rbuf = StringIO(self.__trans.read(max(sz, self.__rbuf_size)))
  File "thrift/transport/TSocket.py", line 94, in read
    buff = self.handle.recv(sz)
socket.error: [Errno 104] Connection reset by peer

and in the thrift server log:

2012-06-29 13:06:01,330 ERROR org.apache.thrift.server.TNonblockingServer: Read an invalid frame size of -2147418111. Are you using TFramedTransport on the client side?
@teepark

not sure about discovering the thrift version, but in hbase/lib I have a libthrift-0.8.0.jar if that helps

@teepark

the same with happybase git master

@wbolster
Owner
@teepark

Do you have any particular reason to use that transport instead if the other ones?

No, I'm running exactly the python code I posted above, not doing anything special. If happybase isn't using the framed transport then I guess the thrift server's log message is mis-diagnosing the issue.

@wbolster
Owner
@teepark

starting the thrift server in the foreground with bin/hbase thrift start -c -threadpool I see this error on the console running the server:

ERROR thrift.TBoundedThreadPoolServer: Thrift error occurred during processing of message.
org.apache.thrift.protocol.TProtocolException: Expected protocol id ffffff82 but got ffffff80
    at org.apache.thrift.protocol.TCompactProtocol.readMessageBegin(TCompactProtocol.java:445)
    at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:22)
    at org.apache.hadoop.hbase.thrift.TBoundedThreadPoolServer$ClientConnnection.run(TBoundedThreadPoolServer.java:287)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
    at java.lang.Thread.run(Thread.java:722)

and get this in the python client:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "happybase/api.py", line 546, in scan
  File "happybase/hbase/Hbase.py", line 1716, in scannerOpenWithScan
  File "happybase/hbase/Hbase.py", line 1728, in recv_scannerOpenWithScan
  File "thrift/protocol/TBinaryProtocol.py", line 126, in readMessageBegin
    sz = self.readI32()
  File "thrift/protocol/TBinaryProtocol.py", line 203, in readI32
    buff = self.trans.readAll(4)
  File "thrift/transport/TTransport.py", line 58, in readAll
    chunk = self.read(sz-have)
  File "thrift/transport/TTransport.py", line 160, in read
    self.__rbuf = StringIO(self.__trans.read(max(sz, self.__rbuf_size)))
  File "thrift/transport/TSocket.py", line 108, in read
    raise TTransportException(type=TTransportException.END_OF_FILE, message='TSocket read 0 bytes')
thrift.transport.TTransport.TTransportException: TSocket read 0 bytes

for -hsha, -nonblocking, and -threadedselector the help output says "this implies the framed transport" and indeed using any of those produces the error in the OP.

@teepark

ah HAH. the -c was wrong (I had thought that was a negation of -f for framed transport).

the thrift server must be started with bin/hbase thrift start -threadpool. The help output says of -threadpool that "this is the default", but it requires that one of [-hsha, -nonblocking, -threadpool, -threadedselector] be explicitly used.

starting it with bin/hbase-daemon.sh start thrift was picking one of the framed transport modes.

this is worth documenting imho

@teepark teepark closed this Jun 29, 2012
@wbolster
Owner

Ah, this should definitely end up in the docs. Maybe HappyBase can work with other Thrift transports as well. Reopening so that I don't forget about this.

@wbolster wbolster reopened this Jun 30, 2012
@wbolster wbolster added a commit that referenced this issue Jul 9, 2012
@wbolster Initial support for framed Thrift transports
See issue #6.

Add initial support for TFramedTransport in addition to
TBufferedTransport. This means HappyBase can connect to the different
Thrift server implementations in HBase 0.94.x. This is required for the
nonblocking, threadedselector, and hsha implementation. The threadpool
implementation requires TBufferedTransport.

This change is not exposed in the API yet.
6dcd486
@wbolster wbolster was assigned Jul 10, 2012
@wbolster wbolster added a commit that referenced this issue Jul 11, 2012
@wbolster Support framed Thrift transports in Connection constructor
The Connection constructor now features 'transport' parameter that
specifies the Thrift transport to use. Fixes issue #6.
9528b67
@wbolster wbolster added a commit that referenced this issue Jul 11, 2012
@wbolster Mention 'transport' arg to Connection() in tutorial
See issue #6.
fea12e7
@wbolster
Owner

Okay, this issues seems fixed now. I've also updated the tutorial and the API docs. Could you please have a look at http://happybase.readthedocs.org/en/latest/tutorial.html and tell me what you think? Thanks!

@wbolster wbolster closed this Jul 11, 2012
@teepark

looks great, thanks!

@bilkulbekar

I still see these errors on using latest code.

@wbolster
Owner

@bilkulbekar Please provide more details. This usually only happens when the Thrift server speaks a different protocol than the Thrift client (HappyBase).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment