Istio Origin Authentication doesn't work with some EC Keys

[jammerful]

Bug description
Istio is unable to parse JWKS' from PingFederate that contain EC keys. This is blocking me from upgrading istio versions, which is security issue with istio.

Issuer 'https://pingfederate.mycorp.io:5555' in jwt_authn config has invalid local jwks: Jwks EC [x] or [y] field is missing or has a parse error.

Expected behavior
Istio should parse the JWKS below as it is perfectly valid, the error that is being printed is incorrect--all EC keys have their required components.

Steps to reproduce the bug

• Copy this JWKS to a location that istio can access:
  {"keys":[{"kty":"EC","kid":"Ozxgl1WsWyQBF1lcbYyjYl","use":"sig","x":"Vq3vKJCeJyihZIauouQp3eFmuYLCsEcID_sGigAx2gs","y":"3UaPoda-HCrJeU4i960qFHMschTdmqjkitiIyYc5svA","crv":"P-256"},{"kty":"EC","kid":"RBsu7DAEUcrgx4X91TVrsY","use":"sig","x":"dQ_SlIctw4nWZEJ2rasiyrps7jXuxr1E81zw4X-aaY1LLRhcaUxpInytXeZK5mOm","y":"xrhQlZQZeDWm7VRHThevLTKqOkqfNwTmHL7P_f23BPn8SLlXd9p1jS4LzL0KK0rL","crv":"P-384"},{"kty":"RSA","kid":"X3oArm2sGh5pO0jWl5O41C","use":"sig","n":"lOTXC2Pfi0dajzDAOERQLTtT0_GbjAFaNflzV-0tWxban1CPEN0n5UG5z-c0KxKY6fhZshQ1Opr3VbQmE1MGSeYf3qEUD4Th3ZubVV_2Yhuio-UZXllz1EGgAh7sC9TzQi84jIYj_mhdno4l6Y3FVlvM6VtNYRGPDNgjRoVfd63vocXAqgUok6LpEcL9MbrvmK-hFPVNX7euGN_xm_qZM5-JrJMnKz6shnjrul7yZ-ZExzfFx_LSBqum-fkKv2FfEoJqyzyVlbabmDUZ81B9ZP0nfaP3e-IRSQECuXf52PfHqEgZbrax8hKAfpZKJX643gxSCnBWdmNO-BFSlho4pw","e":"AQAB"},{"kty":"EC","kid":"sxG_WeuLxIKXoVit-8vyQf","use":"sig","x":"AG3w2vYgVbn4E27rkxZPUVrzLWhMctY5GOP6xygLLFwNRaoOx2gnlQPwAsEXHxz80u5lfmOms0pJSjuDrNqs5pB4","y":"Ad0K-hbFmTVj3nMOw7jAdl21dlU35pG1g7h_Tswr0VYfxqg4ubIPyXrrtmlKH8q3c2Gqgq77Uq12qfcDE8zF2a4v","crv":"P-521"},{"kty":"EC","kid":"7uLnfLOhXPmOZ2BUaeUue-","use":"sig","x":"fR71QOze0q-0uDBuOyTdJANOA0Kz_WT0ykuuuLH94uCJzD13B5OBB1y4jtOvIYvX","y":"WwjH9z9kzSsGZmkl5BvXhyCO8udP9nNFuZM4WugT7C-kD0UNqojpCEkigWorRlin","crv":"P-384"},{"kty":"EC","kid":"FWwgHZozeidVHog8YEjhT-","use":"sig","x":"C2U_Vuv_t3VEOv-UaprJTir3L_SWq_gjwVSUcLTq5Bc","y":"U9F6c5qlnRLSR_KRiCyAzRk7YUpXrXbqsUJYYlTPw7A","crv":"P-256"},{"kty":"RSA","kid":"aNspNiifg5EiN-R1R-RO4O","use":"sig","n":"klTzMXRM83AWKoKRZQwI5XOdJ0tB0jr1ieo_uLT_W6w_hNY8jbFzN1xwxjyUZFBHQVDU2PlyjRZDKQ5tIRs05__DwCvYdDiN0i5f5ChpB66UkAUn1IuuGUQYP7OHVSdeDCFCq3q8UGrl2FCFRxyL9k9hu0VX7Pj06SNGr74_hbCXF6VJmcgDmoXrTXEC5hSBciauGmAxlIbhDbDx-3QHrPJ5P4OLCwH0kfW2RB50O6DMZmKhXeSqE0WiI9KbMz_Mq-TIMiPNkRC1Wsdv4sKoYHf1t6sdp6n9qconQRGwGZNaWcKk6nno-26a_8CxGAuHh8BIFIvFj5AohFFXu2-slw","e":"AQAB"},{"kty":"EC","kid":"ikc6qm-M_topQoGzydo8zI","use":"sig","x":"AE8C5SEixbzl9ez1NPsYOehpnPohIAMxucjZjw2E9aVCBdGo7t2hKv2Aa7ql4zpeTucrDP3ZyUKK6-D3m3C10ojx","y":"AXNjApqwEKFjFmR4bbxAf_8nS74zDuOtXsq1PnoU58ZcHON1ZBUOEXY0Y6IPw1Q8ngaLHg30I300L1ZL0aIX4ygv","crv":"P-521"},{"kty":"EC","kid":"403dlMPeSjSNp4cQ0GBSLL","use":"sig","x":"AI4FHJCcr7fMCb2BfCj9l6bptD513AVVSbOLP2wPM0Aq6CKOEWlN5Yc8qW1Z8NFOPkjeBsCagV02qNJFxH0oi9em","y":"AK3nxsAk3aMOwqIGU-9xXW2I_wwkRxbTR5cPPceWEzpvUHXqyfOXihVYPTJkbaYggyOWjlpw5RYJZGBTLkFUzdal","crv":"P-521"},{"kty":"EC","kid":"IcGz3uUD_EfbMQWQ-6SrmT","use":"sig","x":"10bDbsUH92XHCizkNtBzumyzro8aZypDTqG6ob1fMXk","y":"zfradNJdJ739STTO9vQmOtMl3r7XlnX2SNEusGsFtz0","crv":"P-256"},{"kty":"EC","kid":"oTrCYqttZBLTIv4R5lrfiZ","use":"sig","x":"kOxZSdBd2DPGjuf0lrV30Bc8LCj2EBMfmluPc3sV44fnsZWtTnQz4pCcDjz2hzOY","y":"587zK_ggKmzpYaKz8AaBKERnAsF0AfLjIo2dAu0BdCO0FzSJGyV_cDpNmAdx8ah4","crv":"P-384"},{"kty":"RSA","kid":"vty_MATEPV9warjejj_hef","use":"sig","n":"6TDzjPXHfjDygJ3wa-0BB4m028hatl-PnfT5BEFcIkkWhGRswQSpiGCGjb3DXQ4LxAwZ_XM5RuBYMOMZw9qEU01lhQycqYVOub7R0lli2oDETW4pOATa6JW7QyyXcbbcnYxqj6qfwKb_XfXBDfLpwT8K1_ylJxAymV1ZgfeXDexGBr7d4fLNzgGV7CjZcYmMftn3CktKrA2vy8fLQ2wfVQgfo1J9UqCmLoo6sorW2Sn23Vsx4sTN8OkcrYzpd8_0Gj0X4jXsnk05rfvAWfn8iBHxe6ERBuVh48SWkGjpRnLLoyFmZFg0d1aDlskWp2rYu0VeeUAfKDAeGAU3_1ihuQ","e":"AQAB"}]}
• Make any deployment/pod with a sidecar injected
• Add this istio authentication policy pointed the JWKS location aforementioned

apiVersion: authentication.istio.io/v1alpha1
kind: Policy
metadata:
  name: test
  namespace: default
spec:
  targets:
    - name: test
      ports:
        - number: 8443
  peers:
    - mtls: {}
  origins:
    - jwt:
        issuer: pinfederate.test.example.com
        jwksUri: your_jwks_location
  principalBinding: USE_ORIGIN

• Looking at the logs for the istio-proxy sidecar and you'll see:

[2019-10-18 22:15:55.856][33][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:87] gRPC config stream closed: 14, no healthy upstream
[2019-10-18 22:15:55.856][33][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:50] Unable to establish new stream
[2019-10-18 22:15:56.947][33][warning][config] [external/envoy/source/common/config/grpc_mux_subscription_impl.cc:81] gRPC config for type.googleapis.com/envoy.api.v2.Listener rejected: Error adding/updating listener(s) 10.110.123.108_8443: Issuer 'https://pingfederate.mycorp.io:5555' in jwt_authn config has invalid local jwks: Jwks EC [x] or [y] field is missing or has a parse error., virtualInbound: Issuer 'https://pingfederate.mycorp.io:5555' in jwt_authn config has invalid local jwks: Jwks EC [x] or [y] field is missing or has a parse error.

Version (include the output of istioctl version --remote and kubectl version and helm version if you used Helm)
Istio version:

client version: 1.3.3
citadel version: 1.3.3
galley version: 1.3.3
ingressgateway version: 1.3.3
pilot version: 1.3.3
policy version: 1.3.3
sidecar-injector version: 1.3.3
telemetry version: 1.3.3

Kubernetes version:

Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.4", GitCommit:"67d2fcf276fcd9cf743ad4be9a9ef5828adc082f", GitTreeState:"archive", BuildDate:"1970-01-01T00:00:01Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"14+", GitVersion:"v1.14.8-eks-b7174d", GitCommit:"b7174db5ee0e30c94a0b9899c20ac980c0850fc8", GitTreeState:"clean", BuildDate:"2019-10-18T17:56:01Z", GoVersion:"go1.12.10", Compiler:"gc", Platform:"linux/amd64"}

Helm version:

Client: &version.Version{SemVer:"v2.14.3", GitCommit:"", GitTreeState:"clean"}
Error: could not find tiller

How was Istio installed?
Through the helm chart, via templating with no tiller.

Environment where bug was observed (cloud vendor, OS, etc)
AWS EKS version 1.14.8

[JimmyCYJ]

https://discuss.istio.io/t/jwks-parse-error/4142/10 has some context on this issue.

[yangminzhu]

As mentioned in https://discuss.istio.io/t/jwks-parse-error/4142/4?u=yangminzhu, Istio switched to use the Envoy JWT filter in 1.3 and the the upstream JWT verify library used in Envoy doesn't support the EC key at that time.

The Envoy and the upstream JWT library added the support in envoyproxy/envoy#8807 but this fix is not available in 1.3 (but should be in 1.4).

Some options for now:

• Upgrade to 1.3 with option --set pilot.env.USE_ISTIO_JWT_FILTER=true when generating the yaml for installation. This flag allows to fallback to use the Istio JWT filter. Note, the original user report says Istio 1.2 is working.
• Cherrypick the upstream Envoy fix to the Envoy used in Istio 1.3 and release a new Istio 1.3.x. I'm afraid it's hard to do this.
• Upgrade directly to 1.4. I think it's not recommended to upgrade from 1.2 directly to 1.4 so just listed as an option for completeness.

So probably the best option for the short term is to use the flag USE_ISTIO_JWT_FILTER in 1.3 and upgrade to 1.4. @jammerful Let me know your thoughts. Thank you again for reporting the issue.

[jammerful]

@yangminzhu I will try upgrading to 1.3 first using that flag, however I did also run into this issue in 1.2.7, but not 1.2.2. So I'm not sure it will work.
Thank you for your help.

[jammerful]

@yangminzhu @JimmyCYJ I tested moving to all of the latest versions of 1.2, 1.3, and 1.4 and I'm unable to change my versions. If not resolved by next week, I'll have to abandon using istio.

Version 1.2.10

Both with the USE_ISTIO_JWT_FILTER environment variable and without it, I'm seeing an error on the sidecar parsing the JWKS.

[warning][filter] [./src/envoy/http/jwt_auth/pubkey_cache.h:85] Invalid inline jwks for issuer: https://pingfed.mycorp.io, jwks: REDACTED , error: JWK_EC_PUBKEY_PARSE_ERROR

Version 1.3.6

For version 1.3.6 using the USE_ISTIO_JWT_FILTER environment variable, and ran into the same issue:

[warning][filter] [./src/envoy/http/jwt_auth/pubkey_cache.h:85] Invalid inline jwks for issuer: https://pingfed.mycorp.io, jwks: REDACTED , error: JWK_EC_PUBKEY_PARSE_ERROR

Version 1.4.2

Even more concerning is moving to version 1.4.2 (not upgrading, but starting from a new cluster) is giving me a new error without the USE_ISTIO_JWT_FILTER environment variable:

[Envoy (Epoch 0)] ...[warning][config] [external/envoy/source/common/config/grpc_mux_subscription_impl.cc:82] gRPC config for type.googleapis.com/envoy.api.v2.Listener rejected: Error adding/updating listener(s) 10.110.64.204_8443: Issuer 'https://pingfed.mycorp.io' in jwt_authn config has invalid local jwks: [crv] field specified is not compatible with [alg] for an EC key, virtualInbound: Issuer 'https://pingfed.mycorp.io:' in jwt_authn config has invalid local jwks: [crv] field specified is not compatible with [alg] for an EC key

Note this is a really big deal for anyone using PingFederate as an IdP as we can't upgrade and current versions of envoy have security flaws.

[yangminzhu]

The USE_ISTIO_JWT_FILTER environment variable is introduced in istio 1.3 to fall back to use the Istio JWT filter, it doesn't exist in istio 1.2.

It seems the JWT from pingfed is not recognised by either Istio JWT filter nor Envoy JWT filter, this really has to be fixed in Encoy JWT filter.

By taking a quick look at the code here https://github.com/google/jwt_verify_lib/blob/9f10e2d60d42edeb6662e185707a7d6a4ebc5604/src/jwks.cc#L168, it looks like the error is caused because the JWT from pingfed doesn't set the alg field.

The code uses ES256 if alg is not set and check if the crv is set to P-256. However the JWT from pingfed sometimes set the crv to P-521 or P-384 which in turn caused the error.

@jammerful I'm OOO today and Thursday, I will do some more research of the standard to see what's the right thing to do here, I'm not sure should the JWT set alg field explicitly or should the code be more tolorent here.

Also @qiwzhang could you take a look and let me know what do you think? Thanks

[jammerful]

@yangminzhu Thanks for the information. Regarding the change in the parsing library it looks like it was changed in the 1.2 branch, as I'm unable to move from 1.2.x to 1.2.10.

I will also open an issue with the envoyproxy today.

[jammerful]

@yangminzhu @qiwzhang I've created issues in both envoy and google/jwt_verify_lib, you can see them linked above. Any further help would be appreciated.

[yangminzhu]

This should be fixed by google/jwt_verify_lib#32, I think @qiwzhang will have another PR to update the Envoy to pick it up. Once that is done, I will update Istio to pick up the latest Envoy with the fix.

@jammerful, I think we may have this in Istio 1.4.3 or 1.5 depending on if we have another patch release for 1.4, either case, it will probably take a while for the release.

[jammerful]

@yangminzhu Thank you so much for resolving this, I look forward to upgrading istio.😀
I'll note it may make sense to look at the workflow for envoy upgrades as this "breaking change" made it into older versions.

[jammerful]

@yangminzhu Should the latest envoy fix be pulled in now? Also is it looking likely that we'll have a 1.4.3 release?

[yangminzhu]

@jammerful I'm not sure about the 1.4.3 release, @howardjohn may know about this.

I'm trying to pull in the upstream Envoy in envoyproxy/envoy-wasm#355, but there are some mysterious error related to the wasm, @kyessenov @PiotrSikora could you help here? Thanks.

[howardjohn]

1.4.3 will be released next week, so this needs to be done asap if you want to get it in that release

[jammerful]

@howardjohn @yangminzhu It would be amazing if this could be make it into the 1.4.3 release.

[yangminzhu]

@jammerful I did some more investigation and it turns out we don't need to update envoy-wasm to cherrypick this fix to 1.4, we should actually do this in istio/envoy repo: istio/envoy#131, @howardjohn could you review the cherrypick for 1.4? thank you

[jammerful]

Thank you, hope this makes into 1.4.3 as it will be much easier to upgrade to.

[yangminzhu]

@jammerful This should be fixed in 1.4.3 with the merge of #19912. Is it possible for you to verify the fix? The 1.4.3 is not released yet so you have to build it manually from the release-1.4 branch. If you happen to have some JWT token that could be shared with me, I can also verify it locally. Thank you.

[jammerful]

@yangminzhu Awesome, thank you so much. Unfortunately, I can't easily build istio from a branch. I can however give you a JWKS to test:

{"keys":[{"kty":"EC","kid":"5-xNFS6MjiPK8WUTWEco_-","use":"sig","x":"AXmanLMT6pGyGX4fgMiQcQPzNTRKlmmeafMtwcktC-xng1lqW2FMmodwtujC93AwxzPfEH1vNVtBwQlsyd1vOxgd","y":"APW5Q8LhxTfjIjm1cNfW1IAFldD0UqXIa2-tS1hEstw_v9JvVg7usnku_bCkj2BQ9npmfrhm480-Vn7cpREf36PI","crv":"P-521"},{"kty":"EC","kid":"Ml25yjvoLsaX0zlAAeYk4q","use":"sig","x":"Hp1ek8vLYXxekz4aw5LvxiMRq1pmAQjw7hl4xQA-wTw","y":"cI0pGYCmtWRwaujBGSj3yWBNLP9j1x7sD3VlnX_yFhs","crv":"P-256"},{"kty":"EC","kid":"XHlH8ube3OhDaVtZZ6uZRk","use":"sig","x":"_JfYQgClxHClsjav07HhppCezmmL0dl_MJbt4WiwUeB-NdImDJ1fMyMwkkmXwsIl","y":"rLAG1_Dai0GhCVPRCtYaA_AC7SkgOksAKA4oPLSuLc_1HIwUHKJLYe4PPTPstb3I","crv":"P-384"},{"kty":"RSA","kid":"oHoGWeIMWkTgIY7ldE2lrR","use":"sig","n":"m3KsUkoFjXpcFKS0PJuPVYVPJiLzHLRzEcP9c-U6E4QQxNVZXHLFg-mzjrMxOFCoG2Mv9ZR4mRCiijnvl8CnD4CzRil_XDF8PmMiWmV1qHSNRzrRZKBaHUlRLXH7DKku0-we4z---5nd84uD0Vpt7Jp_vhj5PPDq0yFpkUoouoxMRnuWJLUUlsouY73AdSgGZrtn4LvtM0Qyc2wf5j8JwxVb47xcf_-_dkMJVoPh9aeezqRPXMnT7g9j5XlXk3JQ2o3UGhlZ-KtumKor_RpdTdGiH6SSDrXK8K49fz124myYLsRA9kvKVythdxKielrhr5m0hXvafwVTCHialnl7ew","e":"AQAB"},{"kty":"EC","kid":"40S0J7TaSKB3vm9OCX34HP","use":"sig","x":"Hoj6_aiQI3fVRYb-IQoQoxh2UuklW-ZnvbDpmqLMTNB_kGx-b2mOk5XZmL2TBunW","y":"zSb6G7Br-KHtETvcfATjcI3wqyHxBLwgOMWNDSdFYypCkUWv4h28iqT0v-mwO0Wr","crv":"P-384"},{"kty":"EC","kid":"JWvidMAMUVUSy4I3R58eM7","use":"sig","x":"AUiaeqqUVZFPKTFoq_zgSfeVl0ToN2KOZO_dlQo7898xavgLeoMJZBfWWuZWrNClgApmJf9OdzVG50TALQtfH6Hh","y":"AJw8jhaayAYPLnvynVljeN6WOFFro-jmtoLt4xJBHTajyaqm7bpt4Jf6Qwxvb8OnH4Nr2gh1LYN2-XdTq_6yosjo","crv":"P-521"},{"kty":"RSA","kid":"Sq-P4RzfoRCfnBH4Dwa3_L","use":"sig","n":"hdwZKBTNmUudeLnou6T1k-E9sNzQ5RenQk7Ktul_UEWLKL3uSYQOdNAAmsxvUscmY22sua1aaeOtNyOC8VHEdL2KhSCdSBLXMbmBITnvHxn9oPHBixipT1w3pjK6TbOwfwkGnnG1aKol8L7srwqsRC5WDiJJ0eTj4RS10IgdpqH5zCQ6deCAHPNuOiY2Xc71ySxxaapvqwREkMycOAuDae5lHJjiCNVFCVYARAFygpTia1l5Ef8yYsIxg_qZx6BVkYjfBhXRV2T0jVmR58Oi66ATtiWqUseUfQ8pGAMJj9alHbXNg1AUQwNT6lKWn-UDzOfkJW_2EHojKmdBNN1dAw","e":"AQAB"},{"kty":"EC","kid":"gHd3Clt8OW8KV7LmfctObH","use":"sig","x":"wrgPey2EFAzEhsBcqdt1-MPoM-Ym9gNA6_5dfPAKfl4","y":"5xWSI0P5PddZQEejEZqSawygdBEWTorQjWgXgH9LtwY","crv":"P-256"},{"kty":"EC","kid":"9IpylhBzTqeNAQoJ9R8Yhr","use":"sig","x":"ASVKpSmuoG2VirOoBU1zVVLyyEisn-dCcNOrsNih8ARIxu29qc2qGVcX1LQQ8IZM2AzL2jUCr9kqY2WFcZ6cDRJX","y":"AK4GDNlKWicfKqE-5ykIob3lAwTcQyS1coVzXFIaH1jHO8cZNPbWBc4je5GTSYAW--y0Rqf2rvJMTzGz8QSf9hpO","crv":"P-521"},{"kty":"RSA","kid":"RAQMXMLgBjKA-k1ew7Du0c","use":"sig","n":"6JM1aIS7x-FsXgVK8bej-VwH4XmrcOKXLq7dfmwA3mCsNFfI1stvAZqmvMM_iC7DMzVFco3xg1oiVDRoIFj37WSBYYim76W90rT0O0RcQm4X7MerFYoSnILJ2YHVLbIG3dHd98ImgnDD09h1r3weLhAMxjv2r4MUtkM1f06xoqXqlifiQFvXo2zBohIBbenEC6YH-oSOEA30kwZH4yTeaj1l_Kki6OdVEpibYCttLv16wKifkhCvXLSv3Wpo83mzXFPV6JF8Zl-xhCu41VWxhXNTOcFps7HjM7sM7jR3x4EMl6rHMqFAoGxuBWHaQkkhbJITLjlpgRHKl1IBgAYvMQ","e":"AQAB"},{"kty":"EC","kid":"S3HdZZi9zSQm3GsxBE_iIr","use":"sig","x":"VWHrURLcURHdkJYjUZivew53A_9SAu7W66S2aOt7v6CcAA2--BFEYLzOC__zk3fp","y":"uusZTNUL4-SbgFHXex2etJsjWvaYq9c7pkwWbBZ95whWusaJtS0Cy4RivjLKzoYj","crv":"P-384"},{"kty":"EC","kid":"xcFB0mJhCSuCqk_HlMIsfV","use":"sig","x":"qWMhCUuo7TIh5X7joFe5RgNz7rdNYYOgRCtP75__9Ow","y":"_34fniS3TUy8t0eyDKq9ivv_-nxTYBs_YGkgJgjt664","crv":"P-256"}]}

Let me know if can help you in any other way, or if there is anything else you need.

[yangminzhu]

@jammerful I just tested the JWKS in my local build and I didn't see any errors in the envoy log. I do not have the JWT for testing with a real request but I think the log is enough to tell that the bug had been fixed.

I'm closing this now and feel free to re-open if there is still any issues, thank you.