Failed to get data in Kubernetes

[hongbo-miao]

Hi OPAL team, I have succeed Get started with OPAL playground (docker compose) tutorial for docker-compose file, now I am trying to make it work on Kubernetes.

Here is my pull request.

The server and client all can succeed running.
And when I update my opal-example-policy-repo policies, in the log, I can see the server can detect the new commit, and fetch data. But I am not sure if it really succeed fetching the data. Here is the difference between logs:

docker-composer log (succeed)

opal_server_1        | 2021-06-23T17:23:29.132068+0000 |opal_common.git.repo_watcher            | INFO  | Found new commits: old HEAD was '328e476fbbe46714a788f8176018415ca9944265', new HEAD is '2c6758f57cea55bd04617b28ecd63f464358dfec'
opal_server_1        | 2021-06-23T17:23:29.144560+0000 |opal_server.policy.watcher.callbacks    | INFO  | Publishing policy update, directories: ['.']
opal_client_1        | 2021-06-23T17:23:29.192895+0000 |opal_client.policy.updater              | INFO  | Received policy update: affected directories=['.'], new commit hash='2c6758f57cea55bd04617b28ecd63f464358dfec'
opal_client_1        | 2021-06-23T17:23:29.193100+0000 |opal_client.policy.updater              | INFO  | Refetching policy code (delta bundle), base hash: '328e476fbbe46714a788f8176018415ca9944265'
opal_server_1        | 2021-06-23T17:23:29.219159+0000 |uvicorn.protocols.http.httptools_impl   | INFO  | 172.20.0.4:38970 - "GET /policy?path=.&base_hash=328e476fbbe46714a788f8176018415ca9944265 HTTP/1.1" 200
opal_client_1        | 2021-06-23T17:23:29.220729+0000 |opal_client.policy.updater              | INFO  | got policy bundle (delta): '328e476fbbe46714a788f8176018415ca9944265' -> '2c6758f57cea55bd04617b28ecd63f464358dfec'
opal_server_1        | 2021-06-23T17:23:59.162523+0000 |opal_common.git.repo_watcher            | INFO  | Pulling changes from remote: 'origin'
opal_server_1        | 2021-06-23T17:24:00.253751+0000 |opal_common.git.repo_watcher            | INFO  | No new commits: HEAD is at '2c6758f57cea55bd04617b28ecd63f464358dfec'

Kubernetes log (not sure if it really succeed)

2021-06-23T17:23:36.405133+0000 |opal_common.git.repo_watcher            | INFO  | Found new commits: old HEAD was '328e476fbbe46714a788f8176018415ca9944265', new HEAD is '2c6758f57cea55bd04617b28ecd63f464358dfec'
2021-06-23T17:23:36.418702+0000 |opal_server.policy.watcher.callbacks    | INFO  | Publishing policy update, directories: ['.']
2021-06-23T17:24:06.433024+0000 |opal_common.git.repo_watcher            | INFO  | Pulling changes from remote: 'origin'
2021-06-23T17:24:07.308950+0000 |opal_common.git.repo_watcher            | INFO  | No new commits: HEAD is at '2c6758f57cea55bd04617b28ecd63f464358dfec'

When I hit the endpoint /v1/data/users, it only returns {}. So it means my OPAL still has issue.

Would be appreciate for any help. Thanks!

[asafc]

Hi @hongbo-miao, from the log it looks like OPAL server and client on your kubernetes did not successfully connect.
Can you please provide the full log for both server and client?

Also - we do have a helm chart, maybe you can try to see if it works for you.

I see now that it uses a bit older version - 0.1.5 - i am updating the chart to latest OPAL version now, but between 0.1.5 and 0.1.8 there were no major changes, and current helm chart should work regardless, so maybe you should try it.

[asafc]

@hongbo-miao you might need to pass https:// or http:// to OPAL client as part of the server URI.
you might also need a reverse proxy.

i would try both and see.

[asafc]

Hi @asafc sorry I didn't say clear. I mean in future I still need add OPAL client as a sidecar container in my own app pod next step like

My Pod = Container (my own app) + Sidecar container (OPAL Client)

Do I have to continue using my solution with my own Kubernetes yaml files for all OPAL server and client, instead of OPAL Helm Chart solution?

But based on your answer, seems the OPAL Helm Chart solution also works. But I am not sure how.

I know service mesh Linkerd provides two ways to add a sidecar container to my pod:

1. Manually inject like this

   kubectl get deployments --namespace=my-namespace --output=yaml | linkerd inject - | kubectl apply -f -

2. Using Automatic Proxy Injection, first run Linkerd, then any deployment yaml file with

   annotations:
     linkerd.io/inject: enabled

   will automatically be injected.

Regarding your question here, frankly i don't know enough kubernetes to help. I'm still ramping up :)

As i understand, the OPAL helm chart has a configuration var that conditionally adds a server and a client.

like here: https://github.com/authorizon/opal-helm-chart/blob/master/templates/service-server.yaml
notice the {{- if .Values.server }} line wrapping the config.

so you can pass different values to two separate installs (deployments) of the helm chart - one for only the server, and one to inject the client as a sidecar (but i don't know how to do so).

if it's not clear maybe the helm chart is not yet suitable for your use case and i would ignore it and use your own kubernetes configuration.

[hongbo-miao]

Thanks! http://opal-server.hm:7002 works
(http://opal-server.hm seems not work, but I will verify again later in the end)

The client can connect server now. And first time sync from the policy repo. However, when I update the policy repo, the server shows this

error log

2021-06-24T14:04:27.388483+0000 |opal_common.git.repo_watcher            | INFO  | Pulling changes from remote: 'origin'
2021-06-24T14:04:28.370753+0000 |opal_common.git.repo_watcher            | INFO  | No new commits: HEAD is at '2c6758f57cea55bd04617b28ecd63f464358dfec'
2021-06-24T14:04:58.350604+0000 |opal_common.git.repo_watcher            | INFO  | Pulling changes from remote: 'origin'
2021-06-24T14:04:59.655601+0000 |opal_common.git.repo_watcher            | INFO  | Found new commits: old HEAD was '2c6758f57cea55bd04617b28ecd63f464358dfec', new HEAD is '5a53abe63cb55c3c0b892916e077f41faabf6939'
2021-06-24T14:04:59.669715+0000 |opal_server.policy.watcher.callbacks    | INFO  | Publishing policy update, directories: ['.']
2021-06-24T14:04:59.694140+0000 |fastapi_websocket_pubsub.event_broadc...| INFO  | Subscribing to ALL TOPICS, and sharing messages with broadcast channel
2021-06-24T14:04:59.696553+0000 |fastapi_websocket_pubsub.event_notifier | INFO  | New subscription {'id': 'fd67a8056a144b8f822b439c2ea3aa3b', 'subscriber_id': '36fbec284d0543eb9bf4068ceebfa5b3', 'topic': '__EventNotifier_ALL_TOPICS__', 'callback': <bound method EventBroadcaster.__broadcast_notifications__ of <fastapi_websocket_pubsub.event_broadcaster.EventBroadcaster object at 0x7ff8234c1160>>, 'notifier_id': None}
2021-06-24T14:04:59.697144+0000 |fastapi_websocket_pubsub.event_notifier | INFO  | calling subscription callbacks for sub_id=36fbec284d0543eb9bf4068ceebfa5b3 with topic=policy:.
2021-06-24T14:04:59.697400+0000 |fastapi_websocket_pubsub.event_broadc...| INFO  | Broadcasting incoming event
2021-06-24T14:04:59.739913+0000 |fastapi_websocket_pubsub.event_notifier |ERROR  | Failed to notify subscriber sub_id=36fbec284d0543eb9bf4068ceebfa5b3 with topic=policy:.
Traceback (most recent call last):

  File "/root/.local/bin/gunicorn", line 8, in <module>
    sys.exit(run())
    │   │    └ <function run at 0x7ff8257be3a0>
    │   └ <built-in function exit>
    └ <module 'sys' (built-in)>
  File "/root/.local/lib/python3.8/site-packages/gunicorn/app/wsgiapp.py", line 67, in run
    WSGIApplication("%(prog)s [OPTIONS] [APP_MODULE]").run()
    └ <class 'gunicorn.app.wsgiapp.WSGIApplication'>
  File "/root/.local/lib/python3.8/site-packages/gunicorn/app/base.py", line 231, in run
    super().run()
  File "/root/.local/lib/python3.8/site-packages/gunicorn/app/base.py", line 72, in run
    Arbiter(self).run()
    │       └ <gunicorn.app.wsgiapp.WSGIApplication object at 0x7ff8258195b0>
    └ <class 'gunicorn.arbiter.Arbiter'>
  File "/root/.local/lib/python3.8/site-packages/gunicorn/arbiter.py", line 202, in run
    self.manage_workers()
    │    └ <function Arbiter.manage_workers at 0x7ff825109ee0>
    └ <gunicorn.arbiter.Arbiter object at 0x7ff825819520>
  File "/root/.local/lib/python3.8/site-packages/gunicorn/arbiter.py", line 551, in manage_workers
    self.spawn_workers()
    │    └ <function Arbiter.spawn_workers at 0x7ff82510a040>
    └ <gunicorn.arbiter.Arbiter object at 0x7ff825819520>
  File "/root/.local/lib/python3.8/site-packages/gunicorn/arbiter.py", line 622, in spawn_workers
    self.spawn_worker()
    │    └ <function Arbiter.spawn_worker at 0x7ff825109f70>
    └ <gunicorn.arbiter.Arbiter object at 0x7ff825819520>
  File "/root/.local/lib/python3.8/site-packages/gunicorn/arbiter.py", line 589, in spawn_worker
    worker.init_process()
    │      └ <function UvicornWorker.init_process at 0x7ff824a249d0>
    └ <uvicorn.workers.UvicornWorker object at 0x7ff824a67cd0>
  File "/root/.local/lib/python3.8/site-packages/uvicorn/workers.py", line 64, in init_process
    super(UvicornWorker, self).init_process()
          │              └ <uvicorn.workers.UvicornWorker object at 0x7ff824a67cd0>
          └ <class 'uvicorn.workers.UvicornWorker'>
  File "/root/.local/lib/python3.8/site-packages/gunicorn/workers/base.py", line 142, in init_process
    self.run()
    │    └ <function UvicornWorker.run at 0x7ff824a24af0>
    └ <uvicorn.workers.UvicornWorker object at 0x7ff824a67cd0>
  File "/root/.local/lib/python3.8/site-packages/uvicorn/workers.py", line 77, in run
    loop.run_until_complete(server.serve(sockets=self.sockets))
    │    │                  │      │             │    └ [<gunicorn.sock.TCPSocket object at 0x7ff8248e0880>]
    │    │                  │      │             └ <uvicorn.workers.UvicornWorker object at 0x7ff824a67cd0>
    │    │                  │      └ <function Server.serve at 0x7ff8249ea9d0>
    │    │                  └ <uvicorn.server.Server object at 0x7ff8247ce910>
    │    └ <method 'run_until_complete' of 'uvloop.loop.Loop' objects>
    └ <uvloop.Loop running=True closed=False debug=False>
> File "/root/.local/lib/python3.8/site-packages/fastapi_websocket_pubsub/event_notifier.py", line 156, in callback_subscribers
    await self.trigger_callback(data, topic, subscriber_id, event)
          │    │                │     │      │              └ Subscription(id='fd67a8056a144b8f822b439c2ea3aa3b', subscriber_id='36fbec284d0543eb9bf4068ceebfa5b3', topic='policy:.', callb...
          │    │                │     │      └ '36fbec284d0543eb9bf4068ceebfa5b3'
          │    │                │     └ 'policy:.'
          │    │                └ '5a53abe63cb55c3c0b892916e077f41faabf6939'
          │    └ <function EventNotifier.trigger_callback at 0x7ff823ce2b80>
          └ <fastapi_websocket_pubsub.websocket_rpc_event_notifier.WebSocketRpcEventNotifier object at 0x7ff8234c10a0>
  File "/root/.local/lib/python3.8/site-packages/fastapi_websocket_pubsub/event_notifier.py", line 129, in trigger_callback
    await subscription.callback(subscription, data)
          │            │        │             └ '5a53abe63cb55c3c0b892916e077f41faabf6939'
          │            │        └ Subscription(id='fd67a8056a144b8f822b439c2ea3aa3b', subscriber_id='36fbec284d0543eb9bf4068ceebfa5b3', topic='policy:.', callb...
          │            └ <bound method EventBroadcaster.__broadcast_notifications__ of <fastapi_websocket_pubsub.event_broadcaster.EventBroadcaster ob...
          └ Subscription(id='fd67a8056a144b8f822b439c2ea3aa3b', subscriber_id='36fbec284d0543eb9bf4068ceebfa5b3', topic='policy:.', callb...
  File "/root/.local/lib/python3.8/site-packages/fastapi_websocket_pubsub/event_broadcaster.py", line 158, in __broadcast_notifications__
    async with self._sharing_broadcast_channel:
               │    └ <broadcaster._base.Broadcast object at 0x7ff823317a60>
               └ <fastapi_websocket_pubsub.event_broadcaster.EventBroadcaster object at 0x7ff8234c1160>
  File "/root/.local/lib/python3.8/site-packages/broadcaster/_base.py", line 48, in __aenter__
    await self.connect()
          │    └ <function Broadcast.connect at 0x7ff823ced4c0>
          └ <broadcaster._base.Broadcast object at 0x7ff823317a60>
  File "/root/.local/lib/python3.8/site-packages/broadcaster/_base.py", line 55, in connect
    await self._backend.connect()
          │    │        └ <function PostgresBackend.connect at 0x7ff8230e1d30>
          │    └ <broadcaster._backends.postgres.PostgresBackend object at 0x7ff82335cc70>
          └ <broadcaster._base.Broadcast object at 0x7ff823317a60>
  File "/root/.local/lib/python3.8/site-packages/broadcaster/_backends/postgres.py", line 13, in connect
    self._conn = await asyncpg.connect(self._url)
    │                  │       │       │    └ 'postgres://postgres:postgres@broadcast-channel.hm/postgres'
    │                  │       │       └ <broadcaster._backends.postgres.PostgresBackend object at 0x7ff82335cc70>
    │                  │       └ <function connect at 0x7ff823360a60>
    │                  └ <module 'asyncpg' from '/root/.local/lib/python3.8/site-packages/asyncpg/__init__.py'>
    └ <broadcaster._backends.postgres.PostgresBackend object at 0x7ff82335cc70>
  File "/root/.local/lib/python3.8/site-packages/asyncpg/connection.py", line 1981, in connect
    return await connect_utils._connect(
                 │             └ <function _connect at 0x7ff8232b3c10>
                 └ <module 'asyncpg.connect_utils' from '/root/.local/lib/python3.8/site-packages/asyncpg/connect_utils.py'>
  File "/root/.local/lib/python3.8/site-packages/asyncpg/connect_utils.py", line 748, in _connect
    raise last_error
          └ gaierror(-2, 'Name does not resolve')
  File "/root/.local/lib/python3.8/site-packages/asyncpg/connect_utils.py", line 732, in _connect
    con = await _connect_addr(
                └ <function _connect_addr at 0x7ff8232b3af0>
  File "/root/.local/lib/python3.8/site-packages/asyncpg/connect_utils.py", line 632, in _connect_addr
    return await __connect_addr(params, timeout, True, *args)
                 │              │       │               └ (('broadcast-channel.hm', 5432), <uvloop.Loop running=True closed=False debug=False>, ConnectionConfiguration(command_timeout...
                 │              │       └ 60
                 │              └ ConnectionParameters(user='postgres', password='postgres', database='postgres', ssl=<ssl.SSLContext object at 0x7ff823338ec0>...
                 └ <function __connect_addr at 0x7ff8232b3b80>
  File "/root/.local/lib/python3.8/site-packages/asyncpg/connect_utils.py", line 676, in __connect_addr
    tr, pr = await compat.wait_for(connector, timeout=timeout)
                   │      │        │                  └ 60
                   │      │        └ <Task finished name='Task-10' coro=<_create_ssl_connection() done, defined at /root/.local/lib/python3.8/site-packages/asyncp...
                   │      └ <function wait_for at 0x7ff82333b700>
                   └ <module 'asyncpg.compat' from '/root/.local/lib/python3.8/site-packages/asyncpg/compat.py'>
  File "/root/.local/lib/python3.8/site-packages/asyncpg/compat.py", line 103, in wait_for
    return await asyncio.wait_for(fut, timeout)
                 │       │        │    └ 60
                 │       │        └ <Task finished name='Task-10' coro=<_create_ssl_connection() done, defined at /root/.local/lib/python3.8/site-packages/asyncp...
                 │       └ <function wait_for at 0x7ff824b1f820>
                 └ <module 'asyncio' from '/usr/local/lib/python3.8/asyncio/__init__.py'>
  File "/usr/local/lib/python3.8/asyncio/tasks.py", line 494, in wait_for
    return fut.result()
           │   └ <method 'result' of '_asyncio.Task' objects>
           └ <Task finished name='Task-10' coro=<_create_ssl_connection() done, defined at /root/.local/lib/python3.8/site-packages/asyncp...
  File "/root/.local/lib/python3.8/site-packages/asyncpg/connect_utils.py", line 542, in _create_ssl_connection
    tr, pr = await loop.create_connection(
                   │    └ <method 'create_connection' of 'uvloop.loop.Loop' objects>
                   └ <uvloop.Loop running=True closed=False debug=False>
  File "uvloop/loop.pyx", line 1956, in create_connection
    ai_remote = f1.result()

socket.gaierror: [Errno -2] Name does not resolve
2021-06-24T14:04:59.754890+0000 |fastapi_websocket_pubsub.event_broadc...| INFO  | Unsubscribing from ALL TOPICS
2021-06-24T14:04:59.755021+0000 |fastapi_websocket_pubsub.event_notifier | INFO  | Removing Subscription of topic='__EventNotifier_ALL_TOPICS__' for subscriber=36fbec284d0543eb9bf4068ceebfa5b3
2021-06-24T14:05:29.694733+0000 |opal_common.git.repo_watcher            | INFO  | Pulling changes from remote: 'origin'
2021-06-24T14:05:30.827400+0000 |opal_common.git.repo_watcher            | INFO  | No new commits: HEAD is at '5a53abe63cb55c3c0b892916e077f41faabf6939'

Mine current OPAL_BROADCAST_URI is postgres://postgres:postgres@broadcast-channel.hm:5432/postgres which uses Kubernetes DNS.

I will spend some time trying to locate the issue, if I cannot, I will ping you. Thanks!

[hongbo-miao]

Wow it works now, the new issue is because I missed that broadcast service file! Thanks @asafc !
The final working pull request is at hongbo-miao/hongbomiao.com#2679

[asafc]

Cool @hongbo-miao! 🥳 if you have any more questions please don't hesitate to ask.
And if you have more suggestions to missing features you are welcome to open more issues :)