Google Protobuf Error in python connection

[srams2018]

Hi,
could someone help me on the below error. i have build the libhdfs3 both with protobuf 2.3 and 3.5 but getting same error.

Python 2.7.13 |Continuum Analytics, Inc.| (default, Dec 20 2016, 23:09:15)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
Anaconda is brought to you by Continuum Analytics.
Please check out: http://continuum.io/thanks and https://anaconda.org

import hdfs3
conf={"hadoop.security.authentication": "kerberos"}
hdfs = hdfs3.HDFileSystem("server1.xyz.com", port=8020, pars=conf)
Traceback (most recent call last):
File "", line 1, in
File "/data1/apps/anaconda/4.3.1/2/lib/python2.7/site-packages/hdfs3-0.3.0-py2.7.egg/hdfs3/core.py", line 76, in init
self.connect()
File "/data1/apps/anaconda/4.3.1/2/lib/python2.7/site-packages/hdfs3-0.3.0-py2.7.egg/hdfs3/core.py", line 94, in connect
get_lib()
File "/data1/apps/anaconda/4.3.1/2/lib/python2.7/site-packages/hdfs3-0.3.0-py2.7.egg/hdfs3/core.py", line 657, in get_lib
from .lib import _lib as l
File "/data1/apps/anaconda/4.3.1/2/lib/python2.7/site-packages/hdfs3-0.3.0-py2.7.egg/hdfs3/lib.py", line 15, in
_lib = ct.cdll.LoadLibrary(name)
File "/data1/apps/anaconda/4.3.1/2/lib/python2.7/ctypes/init.py", line 440, in LoadLibrary
return self._dlltype(name)
File "/data1/apps/anaconda/4.3.1/2/lib/python2.7/ctypes/init.py", line 362, in init
self._handle = _dlopen(self._name, mode)
OSError: /data1/apps/anaconda/4.3.1/libhdfs3-downstream-2.3.0/libhdfs3/dist/lib/libhdfs3.so: undefined symbol: _ZN6google8protobuf2io17CodedOutputStream36default_serialization_deterministic_E

[martindurant]

You would be best off getting your required versions from here; although when you build your own, obviously you are out in the wild!
The specific error you are getting is not one that I have seen.

[srams2018]

Thanks for the inputs, i will check on the versions now.

[srams2018]

After rebuilding with all versions in meta.yml file , iam getting below error. also getting weird characters after user: connection.
#python
Python 2.7.13 |Continuum Analytics, Inc.| (default, Dec 20 2016, 23:09:15)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
Anaconda is brought to you by Continuum Analytics.
Please check out: http://continuum.io/thanks and https://anaconda.org

import hdfs3
conf={"hadoop.security.authentication": "kerberos"}
hdfs = hdfs3.HDFileSystem("server1.xyz.net", port=8020, pars=conf)
2018-05-03 13:59:39.024993, p29865, th140179328337664, ERROR Failed to invoke RPC call "getFsStats" on server "server1.xyz.net:8020":
RpcChannel.cpp: 483: HdfsRpcException: Failed to invoke RPC call "getFsStats" on server "server1.xyz.net:8020"
@ Hdfs::Internal::RpcChannelImpl::invokeInternal(std::shared_ptrHdfs::Internal::RpcRemoteCall)
@ Hdfs::Internal::RpcChannelImpl::invoke(Hdfs::Internal::RpcCall const&)
@ Hdfs::Internal::NamenodeImpl::invoke(Hdfs::Internal::RpcCall const&)
@ Hdfs::Internal::NamenodeImpl::getFsStats()
@ Hdfs::Internal::NamenodeProxy::getFsStats()
@ Hdfs::Internal::FileSystemImpl::getFsStats()
@ Hdfs::Internal::FileSystemImpl::connect()
@ Hdfs::FileSystem::connect(char const*, char const*, char const*)
@ hdfsBuilderConnect
@ ffi_call_unix64
@ ffi_call
@ _ctypes_callproc
@ PyCFuncPtr_call
@ PyObject_Call
@ PyEval_EvalFrameEx
@ PyEval_EvalCodeEx
@ PyEval_EvalFrameEx
@ PyEval_EvalCodeEx
@ function_call
@ PyObject_Call
@ instancemethod_call
@ PyObject_Call
@ slot_tp_init
@ type_call
@ PyObject_Call
@ PyEval_EvalFrameEx
@ PyEval_EvalCodeEx
@ PyEval_EvalCode
@ PyRun_InteractiveOneFlags
@ PyRun_InteractiveLoopFlags
@ PyRun_AnyFileExFlags
@ Py_Main
@ __libc_start_main
@ Unknown
Caused by
RpcChannel.cpp: 931: HdfsRpcServerException: org.apache.hadoop.security.authorize.AuthorizationException: User ��H�$t�H�D�]L�%Dr2 (auth:PROXY) via a_ndccdh@xyz.COM (auth:KERBEROS) is not authorized for protocol interface org.apache.hadoop.hdfs.protocol.ClientProtocol: denied by configured ACL
@ Hdfs::Internal::RpcChannelImpl::readOneResponse(bool)
@ Hdfs::Internal::RpcChannelImpl::invokeInternal(std::shared_ptrHdfs::Internal::RpcRemoteCall)
@ Hdfs::Internal::RpcChannelImpl::invoke(Hdfs::Internal::RpcCall const&)
@ Hdfs::Internal::NamenodeImpl::invoke(Hdfs::Internal::RpcCall const&)
@ Hdfs::Internal::NamenodeImpl::getFsStats()
@ Hdfs::Internal::NamenodeProxy::getFsStats()
@ Hdfs::Internal::FileSystemImpl::getFsStats()
@ Hdfs::Internal::FileSystemImpl::connect()
@ Hdfs::FileSystem::connect(char const*, char const*, char const*)
@ hdfsBuilderConnect
@ ffi_call_unix64
@ ffi_call
@ _ctypes_callproc
@ PyCFuncPtr_call
@ PyObject_Call
@ PyEval_EvalFrameEx
@ PyEval_EvalCodeEx
@ PyEval_EvalFrameEx
@ PyEval_EvalCodeEx
@ function_call
@ PyObject_Call
@ instancemethod_call
@ PyObject_Call
@ slot_tp_init
@ type_call
@ PyObject_Call
@ PyEval_EvalFrameEx
@ PyEval_EvalCodeEx
@ PyEval_EvalCode
@ PyRun_InteractiveOneFlags
@ PyRun_InteractiveLoopFlags
@ PyRun_AnyFileExFlags
@ Py_Main
@ __libc_start_main
@ Unknown

2018-05-03 13:59:39.025253, p29865, th140179328337664, INFO Retry idempotent RPC call "getFsStats" on server "server1.xyz.net:8020"
Traceback (most recent call last):
File "", line 1, in
File "/data1/apps/anaconda/4.3.1/2/lib/python2.7/site-packages/hdfs3/core.py", line 76, in init
self.connect()
File "/data1/apps/anaconda/4.3.1/2/lib/python2.7/site-packages/hdfs3/core.py", line 140, in connect
msg = ensure_string(_lib.hdfsGetLastError()).split('\n')[0]
File "/data1/apps/anaconda/4.3.1/2/lib/python2.7/site-packages/hdfs3/utils.py", line 129, in ensure_string
return s.decode()
UnicodeDecodeError: 'ascii' codec can't decode byte 0xef in position 211: ordinal not in range(128)

[krishdey]

I tested and got the same error.
Irrespective of it, this should not use impersonation or proxy user. Not sure why its trying to do.

[martindurant]

I agree, but I don't know why you are in a different security setting than you should be. There may well be a bug in the picking of the user in the C++ base, these things are hard to track down. You may also consider using arrow's hdfs module, if you are accessing the data from a machine that is within the cluster, it tends to have fewer security issues.