jwt verify error from library hmac.lua

[siaomingjeng]

Summary

I write a plugin to verify jwt token using open library openidc.lua which depends on jwt.lua and hmac.lua.
Error happens when verifying jwt token in Kong2.0 however it is good in Kong 1.5.
/usr/local/share/lua/5.1/resty/hmac.lua:81: size of C type is unknown or too large at line 44

The same issue is also mentioned here by SerdarN92 on Jan28 and edmargandra-hotmart on Feb6 2020.

Steps To Reproduce

1. My plugin source code: https://github.com/siaomingjeng/telstra-oidc-ccgf
2. Error happening version Kong 2.0 only.
3. Configure a API and configure the plugin on the API
4. Call the API with jwt token, which triggers the jwt token verification process.

Actually this is not the only process of reproducing the error. Any jwt token verification can reproduce the error as long as the jwt related functions are called from jwt.lua.

Additional Details & Logs

• Kong version ($ 2.0.0)

• Kong error logs (<KONG_PREFIX>/logs/error.log)
  2020/02/10 13:04:10 [error] 7292#0: *120 lua coroutine: runtime error: /usr/local/share/lua/5.1/resty/hmac.lua:81: size of C type is unknown or too large at line 44
  stack traceback:
  coroutine 0:
  [C]: in function 'require'
  /usr/local/share/lua/5.1/resty/openidc.lua:927: in function 'openidc_load_jwt_and_verify_crypto'
  /usr/local/share/lua/5.1/resty/openidc.lua:1667: in function 'jwt_verify'
  /usr/local/share/lua/5.1/resty/openidc.lua:1704: in function 'bearer_jwt_verify'
  ...share/lua/5.1/kong/plugins/telstra-oidc-ccgf/handler.lua:78: in function <...share/lua/5.1/kong/plugins/telstra-oidc-ccgf/handler.lua:70>
  coroutine 1:
  [C]: in function 'resume'
  coroutine.wrap:21: in function coroutine.wrap:21
  /usr/local/share/lua/5.1/kong/init.lua:683: in function 'access'
  access_by_lua(nginx-kong.conf:87):2: in main chunk, client: 10.211.55.2, server: kong, request: "GET /test2 HTTP/1.1", host: "10.211.55.7"
  2020/02/10 13:04:10 [error] 7292#0: *120 [kong] init.lua:690 /usr/local/share/lua/5.1/resty/hmac.lua:81: size of C type is unknown or too large at line 44, client: 10.211.55.2, server: kong, request: "GET /test2 HTTP/1.1", host: "10.211.55.7"

• Kong configuration (the output of a GET request to Kong's Admin port - see
  https://docs.konghq.com/latest/admin-api/#retrieve-node-information)

• Operating system
  Linux version 3.10.0-1062.9.1.el7.x86_64 (mockbuild@kbuilder.bsys.centos.org) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-39) (GCC) ) Finish Cassandra DAO #1 SMP Fri Dec 6 15:49:49 UTC 2019

• Kong debug-level startup logs ($ kong start --vv)[root@localhost ~]# kong start --vv
  2020/02/10 13:32:25 [verbose] Kong: 2.0.0
  2020/02/10 13:32:25 [debug] ngx_lua: 10015
  2020/02/10 13:32:25 [debug] nginx: 1015008
  2020/02/10 13:32:25 [debug] Lua: LuaJIT 2.1.0-beta3
  2020/02/10 13:32:25 [verbose] reading config file at /etc/kong/kong.conf
  2020/02/10 13:32:25 [debug] reading environment variables
  2020/02/10 13:32:25 [debug] admin_access_log = "logs/admin_access.log"
  2020/02/10 13:32:25 [debug] admin_error_log = "logs/error.log"
  2020/02/10 13:32:25 [debug] admin_listen = {"0.0.0.0:8001","127.0.0.1:8444 ssl"}
  2020/02/10 13:32:25 [debug] anonymous_reports = true
  2020/02/10 13:32:25 [debug] cassandra_consistency = "ONE"
  2020/02/10 13:32:25 [debug] cassandra_contact_points = {"127.0.0.1"}
  2020/02/10 13:32:25 [debug] cassandra_data_centers = {"dc1:2","dc2:3"}
  2020/02/10 13:32:25 [debug] cassandra_keyspace = "kong"
  2020/02/10 13:32:25 [debug] cassandra_lb_policy = "RequestRoundRobin"
  2020/02/10 13:32:25 [debug] cassandra_port = 9042
  2020/02/10 13:32:25 [debug] cassandra_refresh_frequency = 60
  2020/02/10 13:32:25 [debug] cassandra_repl_factor = 1
  2020/02/10 13:32:25 [debug] cassandra_repl_strategy = "SimpleStrategy"
  2020/02/10 13:32:25 [debug] cassandra_schema_consensus_timeout = 10000
  2020/02/10 13:32:25 [debug] cassandra_ssl = false
  2020/02/10 13:32:25 [debug] cassandra_ssl_verify = false
  2020/02/10 13:32:25 [debug] cassandra_timeout = 5000
  2020/02/10 13:32:25 [debug] cassandra_username = "kong"
  2020/02/10 13:32:25 [debug] client_body_buffer_size = "8k"
  2020/02/10 13:32:25 [debug] client_max_body_size = "0"
  2020/02/10 13:32:25 [debug] client_ssl = false
  2020/02/10 13:32:25 [debug] cluster_control_plane = "127.0.0.1:8005"
  2020/02/10 13:32:25 [debug] cluster_listen = {"0.0.0.0:8005"}
  2020/02/10 13:32:25 [debug] database = "postgres"
  2020/02/10 13:32:25 [debug] db_cache_ttl = 0
  2020/02/10 13:32:25 [debug] db_cache_warmup_entities = {"services","plugins"}
  2020/02/10 13:32:25 [debug] db_resurrect_ttl = 30
  2020/02/10 13:32:25 [debug] db_update_frequency = 5
  2020/02/10 13:32:25 [debug] db_update_propagation = 0
  2020/02/10 13:32:25 [debug] dns_error_ttl = 1
  2020/02/10 13:32:25 [debug] dns_hostsfile = "/etc/hosts"
  2020/02/10 13:32:25 [debug] dns_no_sync = false
  2020/02/10 13:32:25 [debug] dns_not_found_ttl = 30
  2020/02/10 13:32:25 [debug] dns_order = {"LAST","SRV","A","CNAME"}
  2020/02/10 13:32:25 [debug] dns_resolver = {}
  2020/02/10 13:32:25 [debug] dns_stale_ttl = 4
  2020/02/10 13:32:25 [debug] error_default_type = "text/plain"
  2020/02/10 13:32:25 [debug] go_plugins_dir = "off"
  2020/02/10 13:32:25 [debug] go_pluginserver_exe = "/usr/local/bin/go-pluginserver"
  2020/02/10 13:32:25 [debug] headers = {"server_tokens","latency_tokens"}
  2020/02/10 13:32:25 [debug] log_level = "notice"
  2020/02/10 13:32:25 [debug] lua_package_cpath = ""
  2020/02/10 13:32:25 [debug] lua_package_path = "./?.lua;./?/init.lua;"
  2020/02/10 13:32:25 [debug] lua_socket_pool_size = 30
  2020/02/10 13:32:25 [debug] lua_ssl_verify_depth = 1
  2020/02/10 13:32:25 [debug] mem_cache_size = "128m"
  2020/02/10 13:32:25 [debug] nginx_admin_directives = {}
  2020/02/10 13:32:25 [debug] nginx_daemon = "on"
  2020/02/10 13:32:25 [debug] nginx_events_directives = {{name="worker_connections",value="auto"},{name="multi_accept",value="on"}}
  2020/02/10 13:32:25 [debug] nginx_events_multi_accept = "on"
  2020/02/10 13:32:25 [debug] nginx_events_worker_connections = "auto"
  2020/02/10 13:32:25 [debug] nginx_http_client_body_buffer_size = "8k"
  2020/02/10 13:32:25 [debug] nginx_http_client_max_body_size = "0"
  2020/02/10 13:32:25 [debug] nginx_http_directives = {{name="client_max_body_size",value="0"},{name="ssl_prefer_server_ciphers",value="off"},{name="client_body_buffer_size",value="8k"},{name="ssl_protocols",value="TLSv1.2 TLSv1.3"},{name="ssl_session_tickets",value="on"},{name="ssl_session_timeout",value="1d"}}
  2020/02/10 13:32:25 [debug] nginx_http_ssl_prefer_server_ciphers = "off"
  2020/02/10 13:32:25 [debug] nginx_http_ssl_protocols = "TLSv1.2 TLSv1.3"
  2020/02/10 13:32:25 [debug] nginx_http_ssl_session_tickets = "on"
  2020/02/10 13:32:25 [debug] nginx_http_ssl_session_timeout = "1d"
  2020/02/10 13:32:25 [debug] nginx_http_status_directives = {}
  2020/02/10 13:32:25 [debug] nginx_http_upstream_directives = {{name="keepalive_requests",value="100"},{name="keepalive_timeout",value="60s"},{name="keepalive",value="60"}}
  2020/02/10 13:32:25 [debug] nginx_http_upstream_keepalive = "60"
  2020/02/10 13:32:25 [debug] nginx_http_upstream_keepalive_requests = "100"
  2020/02/10 13:32:25 [debug] nginx_http_upstream_keepalive_timeout = "60s"
  2020/02/10 13:32:25 [debug] nginx_main_daemon = "on"
  2020/02/10 13:32:25 [debug] nginx_main_directives = {{name="daemon",value="on"},{name="worker_rlimit_nofile",value="auto"},{name="worker_processes",value="auto"}}
  2020/02/10 13:32:25 [debug] nginx_main_worker_processes = "auto"
  2020/02/10 13:32:25 [debug] nginx_main_worker_rlimit_nofile = "auto"
  2020/02/10 13:32:25 [debug] nginx_optimizations = true
  2020/02/10 13:32:25 [debug] nginx_proxy_directives = {{name="real_ip_header",value="X-Real-IP"},{name="real_ip_recursive",value="off"}}
  2020/02/10 13:32:25 [debug] nginx_proxy_real_ip_header = "X-Real-IP"
  2020/02/10 13:32:25 [debug] nginx_proxy_real_ip_recursive = "off"
  2020/02/10 13:32:25 [debug] nginx_sproxy_directives = {}
  2020/02/10 13:32:25 [debug] nginx_status_directives = {}
  2020/02/10 13:32:25 [debug] nginx_stream_directives = {}
  2020/02/10 13:32:25 [debug] nginx_supstream_directives = {}
  2020/02/10 13:32:25 [debug] nginx_upstream_directives = {{name="keepalive_requests",value="100"},{name="keepalive_timeout",value="60s"},{name="keepalive",value="60"}}
  2020/02/10 13:32:25 [debug] nginx_upstream_keepalive = "60"
  2020/02/10 13:32:25 [debug] nginx_upstream_keepalive_requests = "100"
  2020/02/10 13:32:25 [debug] nginx_upstream_keepalive_timeout = "60s"
  2020/02/10 13:32:25 [debug] nginx_worker_processes = "auto"
  2020/02/10 13:32:25 [debug] pg_database = "kong15"
  2020/02/10 13:32:25 [debug] pg_host = "127.0.0.1"
  2020/02/10 13:32:25 [debug] pg_max_concurrent_queries = 0
  2020/02/10 13:32:25 [debug] pg_password = "******"
  2020/02/10 13:32:25 [debug] pg_port = 5432
  2020/02/10 13:32:25 [debug] pg_semaphore_timeout = 60000
  2020/02/10 13:32:25 [debug] pg_ssl = false
  2020/02/10 13:32:25 [debug] pg_ssl_verify = false
  2020/02/10 13:32:25 [debug] pg_timeout = 5000
  2020/02/10 13:32:25 [debug] pg_user = "kong"
  2020/02/10 13:32:25 [debug] plugins = {"bundled","telstra-oidc-acgf","telstra-oidc-ccgf"}
  2020/02/10 13:32:25 [debug] prefix = "/usr/local/kong/"
  2020/02/10 13:32:25 [debug] proxy_access_log = "logs/access.log"
  2020/02/10 13:32:25 [debug] proxy_error_log = "logs/error.log"
  2020/02/10 13:32:25 [debug] proxy_listen = {"0.0.0.0:80","0.0.0.0:443 ssl"}
  2020/02/10 13:32:25 [debug] real_ip_header = "X-Real-IP"
  2020/02/10 13:32:25 [debug] real_ip_recursive = "off"
  2020/02/10 13:32:25 [debug] role = "traditional"
  2020/02/10 13:32:25 [debug] router_consistency = "strict"
  2020/02/10 13:32:25 [debug] router_update_frequency = 1
  2020/02/10 13:32:25 [debug] ssl_cipher_suite = "intermediate"
  2020/02/10 13:32:25 [debug] ssl_ciphers = "ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384"
  2020/02/10 13:32:25 [debug] status_access_log = "off"
  2020/02/10 13:32:25 [debug] status_error_log = "logs/status_error.log"
  2020/02/10 13:32:25 [debug] status_listen = {"off"}
  2020/02/10 13:32:25 [debug] stream_listen = {"off"}
  2020/02/10 13:32:25 [debug] trusted_ips = {}
  2020/02/10 13:32:25 [debug] upstream_keepalive = 60
  2020/02/10 13:32:25 [verbose] prefix in use: /usr/local/kong
  2020/02/10 13:32:25 [debug] loading subsystems migrations...
  2020/02/10 13:32:25 [verbose] retrieving database schema state...
  2020/02/10 13:32:25 [verbose] schema state retrieved
  2020/02/10 13:32:25 [verbose] preparing nginx prefix directory at /usr/local/kong
  2020/02/10 13:32:25 [verbose] SSL enabled, no custom certificate set: using default certificate
  2020/02/10 13:32:25 [verbose] default SSL certificate found at /usr/local/kong/ssl/kong-default.crt
  2020/02/10 13:32:25 [verbose] Admin SSL enabled, no custom certificate set: using default certificate
  2020/02/10 13:32:25 [verbose] admin SSL certificate found at /usr/local/kong/ssl/admin-kong-default.crt
  2020/02/10 13:32:25 [warn] ulimit is currently set to "1024". For better performance set it to at least "4096" using "ulimit -n"
  2020/02/10 13:32:25 [debug] searching for OpenResty 'nginx' executable
  2020/02/10 13:32:25 [debug] /usr/local/openresty/nginx/sbin/nginx -v: 'nginx version: openresty/1.15.8.2'
  2020/02/10 13:32:25 [debug] found OpenResty 'nginx' executable at /usr/local/openresty/nginx/sbin/nginx
  2020/02/10 13:32:25 [debug] testing nginx configuration: KONG_NGINX_CONF_CHECK=true /usr/local/openresty/nginx/sbin/nginx -t -p /usr/local/kong -c nginx.conf
  2020/02/10 13:32:31 [debug] searching for OpenResty 'nginx' executable
  2020/02/10 13:32:31 [debug] /usr/local/openresty/nginx/sbin/nginx -v: 'nginx version: openresty/1.15.8.2'
  2020/02/10 13:32:31 [debug] found OpenResty 'nginx' executable at /usr/local/openresty/nginx/sbin/nginx
  2020/02/10 13:32:31 [debug] sending signal to pid at: /usr/local/kong/pids/nginx.pid
  2020/02/10 13:32:31 [debug] kill -0 cat /usr/local/kong/pids/nginx.pid >/dev/null 2>&1
  2020/02/10 13:32:31 [debug] starting nginx: /usr/local/openresty/nginx/sbin/nginx -p /usr/local/kong -c nginx.conf
  2020/02/10 13:32:36 [debug] nginx started
  2020/02/10 13:32:36 [info] Kong started

[bungle]

@fffonion, I suspect this might be because of lua-resty-openssl? Does it ring any bells?

[fffonion]

It's caused by resty.hmac (included from lua-resty-jwt which is a dependency of lua-resty-openidc) needs to know the size of HMAC_CTX (different line number on github but same origin).
When defining struct members, type EVP_MD_CTX is ultimately resolved to env_md_ctx_st which is the internal name used for openssl 1.0.x.

While in lua-resty-openssl EVP_MD_CTX is typedef'ed to evp_md_ctx_st which is used since openssl 1.1.x.

This feels a bit tricky to me. The quickest hack would be to insert a cdef before you require "resty.openidc"

ffi.cdef [[
struct evp_md_ctx_st
    {
    const EVP_MD *digest;
    ENGINE *engine;
    unsigned long flags;
    void *md_data;
    EVP_PKEY_CTX *pctx;
    int (*update)(EVP_MD_CTX *ctx,const void *data,size_t count);
    };
]]

[siaomingjeng]

Hi @fffonion,
Thanks very much for your reply! It does resolve the problem putting the patch into handler.lua.
However, I am still pretty confused that, this struct 'env_md_ctx_st' is already defined at the beginning of hmac.lua. Why do we still need to add it additionally?

-------- original code of hmac.lua---------
'''
25 struct env_md_ctx_st
26 {
27 const EVP_MD *digest;
28 ENGINE *engine;
29 unsigned long flags;
30 void *md_data;
31 EVP_PKEY_CTX *pctx;
32 int (*update)(EVP_MD_CTX *ctx,const void *data,size_t count);
33 };
'''

Thanks!

Cheers

Raymond

[fffonion]

@siaomingjeng The struct defined in hmac.lua is env_md_ctx_st. But Kong loads lua-resty-openssl first and EVP_MD_CTX is typedef to evp_md_ctx_st. And if you typedef multiple times in ffi, it will still use the first declared type silently.

All those result in while you load hmac.lua, ffi try to determine the size of EVP_MD_CTX but it thinks it's equal to evp_md_ctx_st and don't know what exatly it means. So this hack actually let ffi know what it is, then problem solved.

BTW I've added a new function in next release of lua-resty-openssl openssl.resty_hmac_compat() to do the same thing.

[hishamhm]

@fffonion is a change in Kong needed to use this new function?

[fffonion]

@hishamhm I haven't released the new version of library, not sure if it's urgent to bump version for this. Maybe it can wait into next release of Kong?

[ghost]

I have a great need for this fix to be made available in a new version of Kong.

In my case, the suggested correction works perfectly in the Vagrant environment but does not work in Kubernets. The same code does not work.

What is the forecast of this correction? We are migrating our api gateway to Kong and this correction is very important for the success of our migration.

[prompt-bot]

wait for fix ...
we want to use version 2.0+ ^_^

[fffonion]

@edmargandra-hotmart could you elaborate more on how it doesn't work in k8s? an error log and also linked openssl version of kong/nginx will be helpful. you could get the latter by running /path/to/nginx -V in the container.

I'm going to release a new version of lua-resty-openssl. Also going to look for more robust fix with upstream library as the current fix is still a hack.

[PSheshenya]

Unfortunately, @bungle commit https://github.com/Kong/kong/blob/14a5533a840c0eb2d4d16726779f0fba3143b325/kong-2.0.0-0.rockspec doesn’t exists in new https://github.com/Kong/kong/blob/master/kong-2.0.2-0.rockspec

[fffonion]

@PSheshenya In Kong 2.0.2 patch release there's 15d8cf0. Not sure why it's not shown as referenced here.

Looks like I misread : ) I'd encouraged to use the temporary hack as above for now. I'm also opening PR in upstream to push this forward.

[ctessier]

Thank you for the great work @fffonion, @bungle ! When can we expect this fix to be available in Kong? Would it be in 2.0.3? Thank you !

[NilsTellow]

When can we expect the new release that contains the fix?

The release made on https://github.com/Kong/kong/blob/2.0.3/kong-2.0.3-0.rockspec also does not contain @bungle's commit.

[fffonion]

Cross posting here as well: the related lua-resty-jwt library also got updated, if you are still experiencing error because of old version of kong, try update lua-resty-jwt to 0.2.2.

[kaitoy]

Kong 2.0.4 doesn't contain the fix.

https://github.com/Kong/kong/blob/master/kong-2.0.4-0.rockspec

Why is this issue closed?

[fffonion]

@kaitoy the fix we added for kong is a temporary fix. the correct fix is in the lua-resty-hmac library and also got propagated to lua-resty-jwt. i would encourage to use the updated lua-resty-jwt library as path forward.

[PSheshenya]

Thanks @fffonion for your suggestions.

Here https://github.com/PSheshenya/kong-oidc I just build a new dependency tree:

kong-oidc
|-lua-resty-openidc 1.7.2-1 depends on lua-resty-jwt
|--lua-resty-jwt/third-party/ 
|--- lua-resty-hmac

@kaitoy , @NilsTellow why you need hmac fix in a base kong image?

[ctessier]

@kaitoy the fix we added for kong is a temporary fix. the correct fix is in the lua-resty-hmac library and also got propagated to lua-resty-jwt. i would encourage to use the updated lua-resty-jwt library as path forward.

Thanks, I got rid of the workaround after updated to lua-resty-jwt to 2.2.0. I am running Kong 2.0.2.

[NilsTellow]

Thanks @fffonion for your suggestions.

Here https://github.com/PSheshenya/kong-oidc I just build a new dependency tree:

kong-oidc
|-lua-resty-openidc 1.7.2-1 depends on lua-resty-jwt
|--lua-resty-jwt/third-party/ 
|--- lua-resty-hmac

@kaitoy , @NilsTellow why you need hmac fix in a base kong image?

I pruned my cache and created a new image. The issue was resolved a while ago. See the issue that is also mentioned above: nokia/kong-oidc#157