Unable to connect application in local mode

[xiaozhongtian]

Hello, I have met an errer ConnectionError: Unable to connect to application when i use app(ApplicationClient) to look at the Key/Value Store.
The environment is in local mode that means the Yarn is deployed in localhost.
For debug, I found that the errer is always happened when the ApplicationClient ask the request to the Container, for exemple: app.shutdown(), len(app.kv), app.get_specification()
For the detail of the errer, i just write a little test code for it.

skein.__version__ : '0.7.3'

spec = skein.ApplicationSpec.from_yaml("""
     name: example
     services:
       sleeper:
         resources:
           memory: 128
           vcores: 1
         script: |
            sleep infinity
     """)

client = skein.Client(log_level="Debug")
app = client.submit_and_connect(spec)
print(len(app.kv))

and the error:

---------------------------------------------------------------------------
ConnectionError                           Traceback (most recent call last)
<ipython-input-68-e18a869fbfdd> in <module>()
----> 1 len(app.kv)

~/anaconda3/lib/python3.7/site-packages/skein/kv.py in __len__(self)
    608 
    609     def __len__(self):
--> 610         return self.count()
    611 
    612     def __setitem__(self, key, value):

~/anaconda3/lib/python3.7/site-packages/skein/kv.py in method(self, *args, **kwargs)
    806         @_wraps(cls.__init__)
    807         def method(self, *args, **kwargs):
--> 808             return self._apply_op(cls(*args, **kwargs))
    809 
    810         if cls.__doc__ is not None:

~/anaconda3/lib/python3.7/site-packages/skein/kv.py in _apply_op(self, op, timeout)
    601     def _apply_op(self, op, timeout=None):
    602         req = op._build_operation()
--> 603         resp = self._client._call(op._rpc, req, timeout=timeout)
    604         return op._build_result(resp)
    605 

~/anaconda3/lib/python3.7/site-packages/skein/core.py in _call(self, method, req, timeout)
    287         code = exc.code()
    288         if code == grpc.StatusCode.UNAVAILABLE:
--> 289             raise ConnectionError("Unable to connect to %s" % self._server_name)
    290         if code == grpc.StatusCode.DEADLINE_EXCEEDED:
    291             raise TimeoutError("Unable to connect to %s" % self._server_name)

ConnectionError: Unable to connect to application

And the logs on Yarn:

Logs for container_1557414577115_0001_01_000001 

19/05/09 15:22:49 INFO skein.ApplicationMaster: Starting Skein version 0.7.3
19/05/09 15:22:49 INFO skein.ApplicationMaster: Running as user vagrant
19/05/09 15:22:49 INFO skein.ApplicationMaster: Application specification successfully loaded
19/05/09 15:22:50 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8030
19/05/09 15:22:51 INFO skein.ApplicationMaster: gRPC server started at ubuntu-bionic:46873
19/05/09 15:22:52 INFO skein.ApplicationMaster: WebUI server started at ubuntu-bionic:43823
19/05/09 15:22:52 INFO skein.ApplicationMaster: Registering application with resource manager
19/05/09 15:22:52 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
19/05/09 15:22:52 INFO skein.ApplicationMaster: Initializing service 'sleeper'.
19/05/09 15:22:52 INFO skein.ApplicationMaster: REQUESTED: sleeper_0
19/05/09 15:22:53 INFO skein.ApplicationMaster: Starting container_1557414577115_0001_01_000002...
19/05/09 15:22:53 INFO skein.ApplicationMaster: RUNNING: sleeper_0 on container_1557414577115_0001_01_000002

For me, normally len(app.kv) will return 0 seen that there is no Key/Value for this demo(sleep infinity) but there is a connection problem clearly.
I don't know how to debug for that. Is there maybe any configuration missed?

Thank you advancely for your reply.

[jcrist]

The environment is in local mode that means the Yarn is deployed in localhost.

By this do you mean that you're running YARN in pseudo-distributed mode (locally on your machine only)? I wonder if this is similar to the issue resolved by #165 (comment), but for the application master instead of the driver. We rely on the host specified by the nodemanager address, but if you're running things locally this may not resolve cleanly (especially if you have it set to localhost).

To check, what does the following report? (untested, there may be a few typos, but hopefully this is clear).

import skein

spec = skein.ApplicationSpec.from_yaml("""
    name: example
    master:
        log_level: debug
    services:
      sleeper:
        resources:
          memory: 128
          vcores: 1
        script: |
           sleep infinity
 """)

client = skein.Client(log_level="debug")
app = client.submit_and_connect(spec)

print("Application running at %s" % app.address)

if app.address.startswith('localhost'):
    print("Host is localhost, try manually specifying `127.0.0.1`")
    address = app.address.replace('localhost', '127.0.0.1')
    # Create a new client with the updated address
    app = skein.ApplicationClient(address, app.id, security=app.security)
    print(len(app.kv))

---

Also, I edited your original comment to enclose the code segments in backticks. This formats them nicely as code, making them much easier to read. I recommend reading https://guides.github.com/features/mastering-markdown/ for more information.

[xiaozhongtian]

Hello,
Glad to receive your reply so fast.
I have to say that I'm new in using github to publish the new issue. So I will try to make the formats more nicely.
The environment is in local mode that means the Yarn is deployed in localhost.

Yes, you are right. i'm running YARN in pseudo-distributed mode that means the RessourceManager, NameNode, DataNode are all in local.

app.address.startswith('localhost')
Ah OK. You help me to fix the big big big problem but so stupid.
I'm using "0.0.0.0" for localhost.
That's why it can't know the app.address.
So it likes better to be available "0.0.0.0" to denote the IP address not only the "localhost" and "127.0.0.1".

Thank you a lot and I think this issue can be closed so quickly.

[jcrist]

Ah, yeah, you need to specify a resolvable address when setting up yarn. You'll likely want 127.0.0.1 or localhost.

Closing.