Got error "Unable to parse JWT for call ..." on core-command when get command by kuiper rule with security enabled

[cherrycl]

🐞 Bug Report

Affected Services [REQUIRED]

The issue is located in: core-command, rules-engine

Is this a regression?

Yes

Description and Minimal Reproduction [REQUIRED]

1. Deploy EdgeX with compose file
2. Create stream by rules-engine

POST https://locahost:8443/rules-engine/streams
{
    "sql": "create stream edgexsource() WITH (FORMAT=\"JSON\", TYPE=\"edgex\")"
}

3. Create rules by rules-engine

POST https://locahost:8443/rules-engine/rules
{
   "id":"rest-rule-0",
   "sql":"SELECT * FROM edgexsource WHERE Uint8>=0",
   "actions":[
      {
         "rest":{
            "url":"http://edgex-core-command:59882/api/v2/device/name/Random-Integer-Device/Int16",
            "method":"PUT",
            "dataTemplate":"{\"Int16\":\"123\"}",
            "sendSingle":false
         }
      }
   ]
}

🔥 Exception or Error

Found the following error on core-command and rules-engine services.

core-command
level=ERROR ts=2023-04-11T09:14:11.431321449Z app=core-command source=auth_middleware.go:68 msg="Unable to parse JWT for call to '/api/v2/device/name/Random-Integer-Device/Int16'; unauthorized"

rules-engine
time="2023-04-11 09:14:31" level=warning msg="sink collect error: io error: http error Unauthorized\n" file="node/sink_node.go:183" rule=rest-rule-0

🌍 Your Environment

Deployment Environment: Ubuntu22 / x86

EdgeX Version [REQUIRED]: Main

Anything else relevant?

[lenny-goodell]

This is always going to be an issue as eKuiper will never have the JWT. We could try going through the API Gateway, but again, eKuiper will not have the gateway token.

@bnevis-i , thoughts?

[bnevis-i]

This is always going to be an issue as eKuiper will never have the JWT. We could try going through the API Gateway, but again, eKuiper will not have the gateway token.

@bnevis-i , thoughts?

1. If Kuiper was invoked via a REST API call, that API might have a JWT on its request that could be forwarded back to EdgeX (requires changes in Kuiper)
2. Could modify Kuiper like we did for support-scheduler and make it JWT-aware somehow. (requires changes to Kuiper)
3. If invoking an EdgeX service, e.g. core-command, disable JWT validation for that service. (workaround; env var override in the docker-compose)

[lenny-goodell]

@bnevis-i

1. If Kuiper was invoked via a REST API call, that API might have a JWT on its request that could be forwarded back to EdgeX (requires changes in Kuiper)

Kuiper is invoked via MessageBus.

2. Could modify Kuiper like we did for support-scheduler and make it JWT-aware somehow. (requires changes to Kuiper)

Yea, I wonder if their REST action spec in the rule can take a JWT. But it would be in plain text since eKuiper doesn't have access to our SecretStore. :-(

3. If invoking an EdgeX service, e.g. core-command, disable JWT validation for that service. (workaround; env var override in the docker-compose)

Is this built in already? I wouldn't think this is a good thing as it does end around the secure endpoints

[bnevis-i]

@bnevis-i

1. If Kuiper was invoked via a REST API call, that API might have a JWT on its request that could be forwarded back to EdgeX (requires changes in Kuiper)

Kuiper is invoked via MessageBus.

2. Could modify Kuiper like we did for support-scheduler and make it JWT-aware somehow. (requires changes to Kuiper)

Yea, I wonder if their REST action spec in the rule can take a JWT. But it would be in plain text since eKuiper doesn't have access to our SecretStore. :-(

Would need the ability to invoke an external module to provide it. JWT's are bearer tokens and they need to be fresh.

3. If invoking an EdgeX service, e.g. core-command, disable JWT validation for that service. (workaround; env var override in the docker-compose)

Is this built in already? I wouldn't think this is a good thing as it does end around the secure endpoints

https://github.com/edgexfoundry/go-mod-bootstrap/blob/main/bootstrap/handlers/auth_middleware.go#L94

EDGEX_DISABLE_JWT_VALIDATION=true|y|Y|1

4. Send the message to core-command on the message bus instead of via the REST API.

See https://ekuiper.org/docs/en/latest/edgex/edgex_rule_engine_command.html#option-2-use-messaging

[lenny-goodell]

EDGEX_DISABLE_JWT_VALIDATION=true|y|Y|1

@bnevis-i, This needs to be added to the docs here: https://docs.edgexfoundry.org/3.0/microservices/configuration/CommonEnvironmentVariables/#standard-environment-variables

I think the other security Envs are also missing from this section.

[bnevis-i]

EDGEX_DISABLE_JWT_VALIDATION=true|y|Y|1

@bnevis-i, This needs to be added to the docs here: https://docs.edgexfoundry.org/3.0/microservices/configuration/CommonEnvironmentVariables/#standard-environment-variables

Docs update here:
edgexfoundry/edgex-docs#1021

[bnevis-i]

5. Add an OAuth2 hook to the eKuiper rules (perhaps inject a private server into the eKuiper container that is accessible over localhost)

See https://ekuiper.org/docs/en/latest/guide/sources/builtin/http_pull.html#global-http-pull-configurations

[cloudxxx8]

According to the discussion in Core WG meeting on April 11 2023, we decided to skip Kuiper tests until we fix this issue.

[bnevis-i]

6. Modify the eKuiper EdgeX plugin

[bnevis-i]

So I've been thinking about this issue. Here is where I currently sit:

EDGEX_DISABLE_JWT_VALIDATION

Still seems to be the most expedient way.

Message bus option

As in https://ekuiper.org/docs/en/latest/edgex/edgex_rule_engine_command.html#option-2-use-messaging. Looking at that it doesn't appear as if there is a way to authenticate to the external message bus. The example seems to imply that the message bus will just take any client that connects to it, which isn't right. If you can just send stuff to core-command via an unauthenticated message bus, it is exactly equivalent to setting EDGEX_DISABLE_JWT_VALIDATION on core-command and using an http rule action.

OAuth method

As in https://ekuiper.org/docs/en/latest/guide/sources/builtin/http_pull.html#global-http-pull-configurations. I performed a small prototype where I created an HTTP server that returned a JWT, and fed it into the request as prescribed, and it worked just fine. However, in the example, the authentication credential is right there in the configuration.

In the simplest case, I could inject a small token service inside the eKuiper container. Because it was "inside" it would be trusted. However, because we also need to support snap-based deployments, creating such a service would be a back door when it listened on the host's localhost interface. Thus, that solution is docker specific. Moreover, unless the eKuiper API is secured, eKuiper could be used to inject ANY rest API call into EdgeX, whether eKuiper is dockerized or not.

For something that would work with snaps, we would have to make it an internal (authenticated) API. However, since eKuiper itself isn't authenticated, any client could just connect to eKuiper, read the rules config, and get whatever secret was there. Thus, regardless of the authentication method (API key, client_id/client_secret, vault username/password, etc), because eKuiper itself is not currently deployed in secure mode, we're just creating a backdoor and thus authenticating the ekuiper client would be security theater.

To have this approach be meaningful, we also need to ensure that eKuiper API security is also enabled. See https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/api/restapi/authentication.md (Filed: #4538) Still, manually syncing a credential to be inlined into a rule would seem difficult to use.

Custom EdgeX plugin

Having a way to for eKuiper to securely drop something into the EdgeX message bus seems to best way. After all, eKuiper can already authenticate in secure mode. But none of the http nor mqtt rule examples really show how this integration as it exists today would work for a rule action.

[bnevis-i]

This behavior, and recommended workarounds, has been documented in the EdgeX 3.0 migration guide:

https://docs.edgexfoundry.org/3.0/V3TopLevelMigration/#secure-mode

Additionally, the OpenZiti zero-trust prototype has shown that it can resolve this issue fully by zitifying eKuiper.