access token was not provided when running opal-client #119

farhah · 2021-08-02T07:46:58Z

farhah
Aug 2, 2021

I run opal-server and opal-client.

This is the error I got on opal-client

2021-08-02T15:37:52.223415+0800 |opal_client.data.fetcher                | INFO  | [<ClientResponse(http://localhost:7002/policy-data) [401 Unauthorized]>
<CIMultiDictProxy('Date': 'Mon, 02 Aug 2021 07:37:52 GMT', 'Server': 'uvicorn', 'WWW-Authenticate': 'Bearer', 'Content-Length': '65', 'Content-Type': 'application/json')>
]
2021-08-02T15:37:52.223593+0800 |opal_client.data.updater                |ERROR  | Failed to fetch url http://localhost:7002/policy-data, got response code 401 with error: {'detail': {'error': 'access token was not provided', 'token': None}}
2021-08-02T15:37:52.953338+0800 |opal_client.client                      | INFO  | stopping background tasks...

I did some checking on the code and edit it, the error went away.

opal_client/data/fetcher.py

async def handle_urls(self, urls: List[Tuple[str, FetcherConfig]] = None) -> List[Tuple[str, FetcherConfig, Any]]:
        """
        Fetch data for each given url with the (optional) fetching configuration; return the resulting data mapped to each URL

        Args:
            urls (List[Tuple[str, FetcherConfig]], optional): Urls (and fetching configuration) to fetch from.
            Defaults to None - init data_url with HttpFetcherConfig (loaded with the provided auth token).

        Returns:
            List[Tuple[str,FetcherConfig, Any]]: urls mapped to their resulting fetched data
        """

        # tasks
        tasks = []
        # if no url provided - default to the builtin route
        if urls is None:
            urls = [(self._data_url, self._default_fetcher_config)]
        # create a task for each url
        for url, config in urls:
            if not config:
                config = self._default_fetcher_config
            tasks.append(self.handle_url(url, config))

        # wait for all data fetches to complete
        results = await asyncio.gather(*tasks, return_exceptions=True)
        logging.info(results)

Please let me know if this is the intended behavior of the function in order for it to work?

Answered by asafc

Aug 2, 2021

Ok, i got it:

Your client token is what identifies you against OPAL server, not against the data-source server (the server that is serving the data source entry).

In your case, you are using the example data sources (which are served by the OPAL server directly), thus the data source server is also the OPAL server in your case.

Assuming your OPAL_CLIENT_TOKEN is eyJ0...8wsk, you need to pass the following OPAL_DATA_CONFIG_SOURCES:

instead of:

export OPAL_DATA_CONFIG_SOURCES='{"config":{"entries":[{"url":"http://localhost:7002/policy-data","topics":["policy_data"]}]}}'

you should pass:

export OPAL_DATA_CONFIG_SOURCES='{"config":{"entries":[{"url":"http://localhost:7002/policy-data","confi…

View full answer

asafc · 2021-08-02T07:52:58Z

asafc
Aug 2, 2021
Maintainer

Hi @farhah, welcome to the OPAL community! :)

I am missing some information in order to help you:

did you run OPAL directly in python or using the docker images?
can you write the commands (or docker configuration) you used to run OPAL?

There's likely no bug here, from the log i am gathering that you ran the client without a valid access token, and so the OPAL server is rejecting the connection with HTTP 401 Unauthorized. OPAL server will only require a valid token when you are running it in secure mode. did you?

The example configuration shows you how to run both OPAL client and OPAL server in non-secure mode (without token verification), which is great for dev environments.

Let me know exactly what you did and we'll continue from there :)

0 replies

farhah · 2021-08-02T08:12:49Z

farhah
Aug 2, 2021
Author

Hi there!

I pip installed opal-server and opal-client
Set up environment variables as needed.
No, I don't run it on docker.

OPAL_AUTH_PRIVATE_KEY=~/.ssh/opal/id_rsa
OPAL_AUTH_PUBLIC_KEY=~/.ssh/opal/id_rsa.pub
OPAL_AUTH_MASTER_TOKEN=sometoken
OPAL_CLIENT_TOKEN=eyJ0eXAiOiJ-somejwt-token

Run opal-server with no problem/error:
(test) ➜ ~ opal-server run
Run opal-client with this errors:
(test) ➜ ~ opal-client run


    └ <uvloop.Loop running=True closed=False debug=False>
> File "/Users/me/.virtualenvs/test/lib/python3.9/site-packages/opal_client/data/fetcher.py", line 65, in handle_url
    response = await self._engine.handle_url(url, config=config)
                     │    │       │          │           └ None
                     │    │       │          └ 'http://localhost:7002/policy-data'
                     │    │       └ <function FetchingEngine.handle_url at 0x1126fce50>
                     │    └ <opal_common.fetcher.engine.fetching_engine.FetchingEngine object at 0x112ceea30>
                     └ <opal_client.data.fetcher.DataFetcher object at 0x1129d8dc0>
  File "/Users/me/.virtualenvs/test/lib/python3.9/site-packages/opal_common/fetcher/engine/fetching_engine.py", line 104, in handle_url
    await asyncio.wait_for(wait_event.wait(), timeout)
          │       │        │          │       └ 10
          │       │        │          └ <function Event.wait at 0x110910790>
          │       │        └ <asyncio.locks.Event object at 0x112dd1610 [unset]>
          │       └ <function wait_for at 0x110913f70>
          └ <module 'asyncio' from '/usr/local/Cellar/python@3.9/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/asyncio/__i...

  File "/usr/local/Cellar/python@3.9/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/asyncio/tasks.py", line 494, in wait_for
    raise exceptions.TimeoutError() from exc
          │          └ <class 'asyncio.exceptions.TimeoutError'>
          └ <module 'asyncio.exceptions' from '/usr/local/Cellar/python@3.9/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/...

asyncio.exceptions.TimeoutError
2021-08-02T16:06:14.515909+0800 |opal_client.data.updater                |ERROR  | Failed to fetch url http://localhost:7002/policy-data with config None, got exception:

0 replies

asafc · 2021-08-02T08:24:24Z

asafc
Aug 2, 2021
Maintainer

Hi @farhah, you are indeed running the server in secure mode, and so you should also run the client with a valid access token.

You can obtain an access token with this command:

opal-client obtain-token $OPAL_AUTH_MASTER_TOKEN --server-url=$YOUR_SERVERS_ADDRESS

If you are running locally you can omit --server-url, it defaults to http://localhost:7002.

example output:

{
    "token": "eyJ0...8wsk",
    "type": "bearer",
    "details": { ... }
}

You can then start the client with said access token by setting the environment variable OPAL_CLIENT_TOKEN with the value of the token key:

export OPAL_CLIENT_TOKEN=eyJ0...8wsk

See more details here.

0 replies

farhah · 2021-08-02T08:25:18Z

farhah
Aug 2, 2021
Author

I did have the token OPAL_CLIENT_TOKEN=eyJ0eXAiOiJ-somejwt-token
I edited my previous message to include it.

0 replies

asafc · 2021-08-02T08:27:08Z

asafc
Aug 2, 2021
Maintainer

Oh i am sorry i did not see you pass the token.
Let me take a closer look at your log.

0 replies

asafc · 2021-08-02T08:49:16Z

asafc
Aug 2, 2021
Maintainer

Ok, i got it:

Your client token is what identifies you against OPAL server, not against the data-source server (the server that is serving the data source entry).

In your case, you are using the example data sources (which are served by the OPAL server directly), thus the data source server is also the OPAL server in your case.

Assuming your OPAL_CLIENT_TOKEN is eyJ0...8wsk, you need to pass the following OPAL_DATA_CONFIG_SOURCES:

instead of:

export OPAL_DATA_CONFIG_SOURCES='{"config":{"entries":[{"url":"http://localhost:7002/policy-data","topics":["policy_data"]}]}}'

you should pass:

export OPAL_DATA_CONFIG_SOURCES='{"config":{"entries":[{"url":"http://localhost:7002/policy-data","config":{"headers":{"Authorization": "Bearer eyJ0...8wsk"}},"topics":["policy_data"]}]}}'

farhah
Aug 2, 2021
Author

Thanks for the help Asaf,
However, I encountered errors
It seems like the same error as before regarding the fetcher
opal_client/data/fetcher.py
If I edit the code like here it'll pass https://github.com/authorizon/opal/issues/118#issue-957861508

2021-08-02T17:42:22.209371+0800 |fastapi_websocket_pubsub.pub_sub_client | INFO  | Connected to PubSub server ws://localhost:7002/ws
2021-08-02T17:42:32.152276+0800 |opal_client.data.fetcher                |ERROR  | Timeout while fetching url: http://localhost:7002/policy-data
Traceback (most recent call last):

  File "/usr/local/Cellar/python@3.9/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/asyncio/locks.py", line 226, in wait
    await fut
          └ <Future cancelled>

asyncio.exceptions.CancelledError


During handling of the above exception, another exception occurred:


Traceback (most recent call last):

  File "/usr/local/Cellar/python@3.9/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/asyncio/tasks.py", line 492, in wait_for
    fut.result()
    │   └ <method 'result' of '_asyncio.Task' objects>
    └ <Task cancelled name='Task-54' coro=<Event.wait() done, defined at /usr/local/Cellar/python@3.9/3.9.6/Frameworks/Python.frame...

asyncio.exceptions.CancelledError


The above exception was the direct cause of the following exception:


Traceback (most recent call last):

  File "/Users/me/.virtualenvs/test/bin/opal-client", line 8, in <module>
    sys.exit(cli())
    │   │    └ <function cli at 0x10f641280>
    │   └ <built-in function exit>
    └ <module 'sys' (built-in)>
  File "/Users/me/.virtualenvs/test/lib/python3.9/site-packages/opal_client/cli.py", line 63, in cli
    opal_client_config.cli([opal_common_config], typer_app=app, help=main_texts.docs, on_start=on_start)
    │                  │    │                              │         │          │              └ <function cli.<locals>.on_start at 0x10f480160>
    │                  │    │                              │         │          └ '\x08\n    Config top level options:\n        - Use env-vars (same as cmd options) but uppercase \n            and with "_" i...
    │                  │    │                              │         └ <opal_common.cli.docs.MainTexts object at 0x111b79eb0>
    │                  │    │                              └ <typer.main.Typer object at 0x10f6cf370>
    │                  │    └ {
    │                  │        "ALLOWED_ORIGINS": "['*']",
    │                  │        "CLIENT_SELF_SIGNED_CERTIFICATES_ALLOWED": "False",
    │                  │        "CLIENT_SSL_CONTEXT_TRUSTED_CA_FILE":...
    │                  └ <function Confi.cli at 0x1119c2820>
    └ {
        "CLIENT_API_SERVER_HOST": "127.0.0.1",
        "CLIENT_API_SERVER_PORT": "7000",
        "CLIENT_API_SERVER_WORKER_COUNT": "1",
        "CL...
  File "/Users/me/.virtualenvs/test/lib/python3.9/site-packages/opal_common/confi/confi.py", line 207, in cli
    self.get_cli_object(config_objects, typer_app=typer_app, help=help, on_start=on_start)()
    │    │              │                         │               │              └ <function cli.<locals>.on_start at 0x10f480160>
    │    │              │                         │               └ '\x08\n    Config top level options:\n        - Use env-vars (same as cmd options) but uppercase \n            and with "_" i...
    │    │              │                         └ <typer.main.Typer object at 0x10f6cf370>
    │    │              └ [{
    │    │                  "ALLOWED_ORIGINS": "['*']",
    │    │                  "CLIENT_SELF_SIGNED_CERTIFICATES_ALLOWED": "False",
    │    │                  "CLIENT_SSL_CONTEXT_TRUSTED_CA_FILE"...
    │    └ <function Confi.get_cli_object at 0x1119c2790>
    └ {
        "CLIENT_API_SERVER_HOST": "127.0.0.1",
        "CLIENT_API_SERVER_PORT": "7000",
        "CLIENT_API_SERVER_WORKER_COUNT": "1",
        "CL...
  File "/Users/me/.virtualenvs/test/lib/python3.9/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
           │    │     │       └ {}
           │    │     └ ()
           │    └ <function BaseCommand.main at 0x1118b9b80>
           └ <Group callback>
  File "/Users/me/.virtualenvs/test/lib/python3.9/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
         │    │      └ <click.core.Context object at 0x111bdbf40>
         │    └ <function MultiCommand.invoke at 0x1118ba940>
         └ <Group callback>
  File "/Users/me/.virtualenvs/test/lib/python3.9/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
           │               │       │       │      └ <click.core.Context object at 0x111bef4f0>
           │               │       │       └ <function Command.invoke at 0x1118ba550>
           │               │       └ <TyperCommand run>
           │               └ <click.core.Context object at 0x111bef4f0>
           └ <function MultiCommand.invoke.<locals>._process_result at 0x1118cae50>
  File "/Users/me/.virtualenvs/test/lib/python3.9/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           │   │      │    │           │   └ {'engine_type': 'uvicron'}
           │   │      │    │           └ <click.core.Context object at 0x111bef4f0>
           │   │      │    └ <function run at 0x111be85e0>
           │   │      └ <TyperCommand run>
           │   └ <function Context.invoke at 0x1118b9670>
           └ <click.core.Context object at 0x111bef4f0>
  File "/Users/me/.virtualenvs/test/lib/python3.9/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
           │         │       └ {'engine_type': 'uvicron'}
           │         └ ()
           └ <function run at 0x111be85e0>
  File "/Users/me/.virtualenvs/test/lib/python3.9/site-packages/typer/main.py", line 497, in wrapper
    return callback(**use_params)  # type: ignore
           │          └ {'engine_type': 'uvicron'}
           └ <function run at 0x111bc9ca0>
  File "/Users/me/.virtualenvs/test/lib/python3.9/site-packages/opal_client/cli.py", line 36, in run
    run_uvicorn("opal_client.main:app",
    └ <function run_uvicorn at 0x111bc7ee0>
  File "/Users/me/.virtualenvs/test/lib/python3.9/site-packages/opal_common/corn_utils.py", line 47, in run_uvicorn
    uvicorn.run(app_path, **options)
    │       │   │           └ {'host': '127.0.0.1', 'port': 7000, 'reload': False, 'workers': 1}
    │       │   └ 'opal_client.main:app'
    │       └ <function run at 0x111d193a0>
    └ <module 'uvicorn' from '/Users/me/.virtualenvs/test/lib/python3.9/site-packages/uvicorn/__init__.py'>
  File "/Users/me/.virtualenvs/test/lib/python3.9/site-packages/uvicorn/main.py", line 393, in run
    server.run()
    │      └ <function Server.run at 0x111d0ee50>
    └ <uvicorn.server.Server object at 0x111d62460>
  File "/Users/me/.virtualenvs/test/lib/python3.9/site-packages/uvicorn/server.py", line 50, in run
    loop.run_until_complete(self.serve(sockets=sockets))
    │    │                  │    │             └ None
    │    │                  │    └ <function Server.serve at 0x111d0eee0>
    │    │                  └ <uvicorn.server.Server object at 0x111d62460>
    │    └ <method 'run_until_complete' of 'uvloop.loop.Loop' objects>
    └ <uvloop.Loop running=True closed=False debug=False>
> File "/Users/me/.virtualenvs/test/lib/python3.9/site-packages/opal_client/data/fetcher.py", line 65, in handle_url
    response = await self._engine.handle_url(url, config=config)
                     │    │       │          │           └ {'headers': {'Authorization': 'Bearer eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiJ9.eyJpYXQiOjE2Mjc1Nzc5OTMsImV4cCI6MTY1OTExMzk5MywiYX...
                     │    │       │          └ 'http://localhost:7002/policy-data'
                     │    │       └ <function FetchingEngine.handle_url at 0x111a50e50>
                     │    └ <opal_common.fetcher.engine.fetching_engine.FetchingEngine object at 0x112042ac0>
                     └ <opal_client.data.fetcher.DataFetcher object at 0x111d2be50>
  File "/Users/me/.virtualenvs/test/lib/python3.9/site-packages/opal_common/fetcher/engine/fetching_engine.py", line 104, in handle_url
    await asyncio.wait_for(wait_event.wait(), timeout)
          │       │        │          │       └ 10
          │       │        │          └ <function Event.wait at 0x10fc5a790>
          │       │        └ <asyncio.locks.Event object at 0x11211b370 [unset]>
          │       └ <function wait_for at 0x10fc5df70>
          └ <module 'asyncio' from '/usr/local/Cellar/python@3.9/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/asyncio/__i...

  File "/usr/local/Cellar/python@3.9/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/asyncio/tasks.py", line 494, in wait_for
    raise exceptions.TimeoutError() from exc
          │          └ <class 'asyncio.exceptions.TimeoutError'>
          └ <module 'asyncio.exceptions' from '/usr/local/Cellar/python@3.9/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/...

asyncio.exceptions.TimeoutError
2021-08-02T17:42:32.178276+0800 |opal_client.data.updater                |ERROR  | Failed to fetch url http://localhost:7002/policy-data with config {'headers': {'Authorization': 'Bearer ey3-some-jwt-token'}}, got exception:

0 replies

asafc · 2021-08-02T10:10:04Z

asafc
Aug 2, 2021
Maintainer

Probably still a config issue, let's jump on zoom and debug together:
https://calendly.com/asaf-cohen

0 replies

farhah · 2021-08-03T03:41:54Z

farhah
Aug 3, 2021
Author

Ok Asaf, I got it working now. There was a typo at the OPAL_DATA_CONFIG_SOURCES.
Before that I didn't set this env variable.

These 2 docs are helpful in understanding the flow of the data source server. Thanks.

0 replies

asafc · 2021-08-03T08:04:56Z

asafc
Aug 3, 2021
Maintainer

Cool, i am glad you got it working! :)

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

access token was not provided when running opal-client #119

{{title}}

Replies: 10 comments

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

Select a reply

access token was not provided when running opal-client #119

farhah Aug 2, 2021

Replies: 10 comments

asafc Aug 2, 2021 Maintainer

farhah Aug 2, 2021 Author

asafc Aug 2, 2021 Maintainer

farhah Aug 2, 2021 Author

asafc Aug 2, 2021 Maintainer

asafc Aug 2, 2021 Maintainer

farhah Aug 2, 2021 Author

asafc Aug 2, 2021 Maintainer

farhah Aug 3, 2021 Author

asafc Aug 3, 2021 Maintainer

farhah
Aug 2, 2021

asafc
Aug 2, 2021
Maintainer

farhah
Aug 2, 2021
Author

asafc
Aug 2, 2021
Maintainer

farhah
Aug 2, 2021
Author

asafc
Aug 2, 2021
Maintainer

asafc
Aug 2, 2021
Maintainer

farhah
Aug 2, 2021
Author

asafc
Aug 2, 2021
Maintainer

farhah
Aug 3, 2021
Author

asafc
Aug 3, 2021
Maintainer