bug: [error] 43#43: init_worker_by_lua error: /usr/local/apisix/apisix/plugin.lua:264: attempt to index upvalue 'local_conf' (a nil value)

[anjia0532]

Issue description

Environment

• apisix version (cmd: apisix version): 2.4 (apache/apisix:2.4-alpine)
• OS (cmd: uname -a): Linux apisix-8548b7764-m2gnf 5.5.8-1.el7.elrepo.x86_64 change: added doc of how to load plugin. #1 SMP Tue Mar 3 18:13:28 EST 2020 x86_64 Linux
• OpenResty / Nginx version (cmd: nginx -V or openresty -V): openresty/1.19.3.1
• etcd version, if have (cmd: run curl http://127.0.0.1:9090/v1/server_info to get the info from server-info API):
• apisix-dashboard version, if have: 2.4

Minimal test code / Steps to reproduce the issue

2021/03/15 09:54:07 [notice] 1#1: start worker processes
2021/03/15 09:54:07 [notice] 1#1: start worker process 43
2021/03/15 09:54:07 [notice] 1#1: start worker process 44
2021/03/15 09:54:07 [notice] 43#43: sched_setaffinity(): using cpu #0
2021/03/15 09:54:07 [notice] 1#1: start worker process 45
2021/03/15 09:54:07 [notice] 44#44: sched_setaffinity(): using cpu #1
2021/03/15 09:54:07 [notice] 1#1: start worker process 46
2021/03/15 09:54:07 [notice] 46#46: sched_setaffinity(): using cpu #3
2021/03/15 09:54:07 [notice] 1#1: start cache manager process 47
2021/03/15 09:54:07 [notice] 1#1: start cache loader process 48
2021/03/15 09:54:07 [notice] 1#1: start privileged agent process 49
2021/03/15 09:54:07 [notice] 45#45: sched_setaffinity(): using cpu #2
2021/03/15 09:54:07 [info] 49#49: *1 [lua] init.lua:93: http_init_worker(): random test in [1, 10000]: 5204, context: init_worker_by_lua*
2021/03/15 09:54:07 [info] 43#43: *2 [lua] init.lua:93: http_init_worker(): random test in [1, 10000]: 3576, context: init_worker_by_lua*
2021/03/15 09:54:07 [info] 43#43: *2 [lua] radixtree.lua:346: pre_insert_route(): path: /apisix/admin/ operator: <=, context: init_worker_by_lua*
2021/03/15 09:54:07 [info] 43#43: *2 [lua] radixtree.lua:234: insert_route(): insert route path: /apisix/admin/ dataprt: 1, context: init_worker_by_lua*
2021/03/15 09:54:07 [info] 43#43: *2 [lua] radixtree.lua:346: pre_insert_route(): path: /apisix/admin/stream_routes/ operator: <=, context: init_worker_by_lua*
2021/03/15 09:54:07 [info] 43#43: *2 [lua] radixtree.lua:234: insert_route(): insert route path: /apisix/admin/stream_routes/ dataprt: 2, context: init_worker_by_lua*
2021/03/15 09:54:07 [info] 43#43: *2 [lua] radixtree.lua:346: pre_insert_route(): path: /apisix/admin/plugins/list operator: =, context: init_worker_by_lua*
2021/03/15 09:54:07 [info] 43#43: *2 [lua] radixtree.lua:346: pre_insert_route(): path: /apisix/admin/plugins/reload operator: =, context: init_worker_by_lua*
2021/03/15 09:54:07 [notice] 43#43: *2 [lua] timers.lua:70: init_worker(): succeed to create background timer, context: init_worker_by_lua*
2021/03/15 09:54:07 [info] 46#46: *5 [lua] init.lua:93: http_init_worker(): random test in [1, 10000]: 2936, context: init_worker_by_lua*
2021/03/15 09:54:07 [info] 45#45: *4 [lua] init.lua:93: http_init_worker(): random test in [1, 10000]: 7273, context: init_worker_by_lua*
2021/03/15 09:54:07 [info] 44#44: *3 [lua] init.lua:93: http_init_worker(): random test in [1, 10000]: 6626, context: init_worker_by_lua*
2021/03/15 09:54:07 [info] 49#49: *1 [lua] radixtree.lua:346: pre_insert_route(): path: /apisix/admin/ operator: <=, context: init_worker_by_lua*
2021/03/15 09:54:07 [info] 49#49: *1 [lua] radixtree.lua:234: insert_route(): insert route path: /apisix/admin/ dataprt: 1, context: init_worker_by_lua*
2021/03/15 09:54:07 [info] 49#49: *1 [lua] radixtree.lua:346: pre_insert_route(): path: /apisix/admin/stream_routes/ operator: <=, context: init_worker_by_lua*
2021/03/15 09:54:07 [info] 49#49: *1 [lua] radixtree.lua:234: insert_route(): insert route path: /apisix/admin/stream_routes/ dataprt: 2, context: init_worker_by_lua*
2021/03/15 09:54:07 [info] 49#49: *1 [lua] radixtree.lua:346: pre_insert_route(): path: /apisix/admin/plugins/list operator: =, context: init_worker_by_lua*
2021/03/15 09:54:07 [info] 49#49: *1 [lua] radixtree.lua:346: pre_insert_route(): path: /apisix/admin/plugins/reload operator: =, context: init_worker_by_lua*
2021/03/15 09:54:07 [notice] 49#49: *1 [lua] timers.lua:70: init_worker(): succeed to create background timer, context: init_worker_by_lua*
2021/03/15 09:54:07 [info] 44#44: *3 [lua] radixtree.lua:346: pre_insert_route(): path: /apisix/admin/ operator: <=, context: init_worker_by_lua*
2021/03/15 09:54:07 [info] 44#44: *3 [lua] radixtree.lua:234: insert_route(): insert route path: /apisix/admin/ dataprt: 1, context: init_worker_by_lua*
2021/03/15 09:54:07 [info] 44#44: *3 [lua] radixtree.lua:346: pre_insert_route(): path: /apisix/admin/stream_routes/ operator: <=, context: init_worker_by_lua*
2021/03/15 09:54:07 [info] 44#44: *3 [lua] radixtree.lua:234: insert_route(): insert route path: /apisix/admin/stream_routes/ dataprt: 2, context: init_worker_by_lua*
2021/03/15 09:54:07 [info] 44#44: *3 [lua] radixtree.lua:346: pre_insert_route(): path: /apisix/admin/plugins/list operator: =, context: init_worker_by_lua*
2021/03/15 09:54:07 [info] 44#44: *3 [lua] radixtree.lua:346: pre_insert_route(): path: /apisix/admin/plugins/reload operator: =, context: init_worker_by_lua*
2021/03/15 09:54:07 [info] 46#46: *5 [lua] radixtree.lua:346: pre_insert_route(): path: /apisix/admin/ operator: <=, context: init_worker_by_lua*
2021/03/15 09:54:07 [info] 46#46: *5 [lua] radixtree.lua:234: insert_route(): insert route path: /apisix/admin/ dataprt: 1, context: init_worker_by_lua*
2021/03/15 09:54:07 [info] 46#46: *5 [lua] radixtree.lua:346: pre_insert_route(): path: /apisix/admin/stream_routes/ operator: <=, context: init_worker_by_lua*
2021/03/15 09:54:07 [info] 46#46: *5 [lua] radixtree.lua:234: insert_route(): insert route path: /apisix/admin/stream_routes/ dataprt: 2, context: init_worker_by_lua*
2021/03/15 09:54:07 [info] 46#46: *5 [lua] radixtree.lua:346: pre_insert_route(): path: /apisix/admin/plugins/list operator: =, context: init_worker_by_lua*
2021/03/15 09:54:07 [info] 46#46: *5 [lua] radixtree.lua:346: pre_insert_route(): path: /apisix/admin/plugins/reload operator: =, context: init_worker_by_lua*
2021/03/15 09:54:07 [notice] 44#44: *3 [lua] timers.lua:70: init_worker(): succeed to create background timer, context: init_worker_by_lua*
2021/03/15 09:54:07 [notice] 46#46: *5 [lua] timers.lua:70: init_worker(): succeed to create background timer, context: init_worker_by_lua*
2021/03/15 09:54:07 [error] 43#43: init_worker_by_lua error: /usr/local/apisix/apisix/plugin.lua:264: attempt to index upvalue 'local_conf' (a nil value)

What's the actual result? (including assertion message & call stack if applicable)

What's the expected result?

[anjia0532]

access any url return 503 with curl/browser

[anjia0532]

apache/apisix:2.4.0-centos

2021/03/15 10:19:02 [info] 45#45: [lua] core.lua:26: use config_center: etcd
2021/03/15 10:19:02 [info] 45#45: [lua] init.lua:56: parse_args(): dns resolver["10.43.0.10"]
2021/03/15 10:19:02 [notice] 45#45: [lua] id.lua:78: init(): not found apisix uid, generate a new one: 20d0d3f7-8cbd-4440-8213-ea8c1f11463c
2021/03/15 10:19:02 [notice] 45#45: using the "epoll" event method
2021/03/15 10:19:02 [notice] 45#45: openresty/1.19.3.1
2021/03/15 10:19:02 [notice] 45#45: built by gcc 8.3.1 20190311 (Red Hat 8.3.1-3) (GCC) 
2021/03/15 10:19:02 [notice] 45#45: OS: Linux 5.5.8-1.el7.elrepo.x86_64
2021/03/15 10:19:02 [notice] 45#45: getrlimit(RLIMIT_NOFILE): 1048576:1048576
2021/03/15 10:19:02 [notice] 45#45: start worker processes
2021/03/15 10:19:02 [notice] 45#45: start worker process 46
2021/03/15 10:19:02 [notice] 45#45: start cache manager process 47
2021/03/15 10:19:02 [notice] 46#46: sched_setaffinity(): using cpu #0
2021/03/15 10:19:02 [notice] 45#45: start cache loader process 48
2021/03/15 10:19:02 [notice] 45#45: start privileged agent process 49
2021/03/15 10:19:02 [info] 49#49: *1 [lua] init.lua:93: http_init_worker(): random test in [1, 10000]: 4632, context: init_worker_by_lua*
2021/03/15 10:19:02 [info] 46#46: *2 [lua] init.lua:93: http_init_worker(): random test in [1, 10000]: 6549, context: init_worker_by_lua*
2021/03/15 10:19:02 [info] 46#46: *2 [lua] radixtree.lua:346: pre_insert_route(): path: /apisix/admin/ operator: <=, context: init_worker_by_lua*
2021/03/15 10:19:02 [info] 46#46: *2 [lua] radixtree.lua:234: insert_route(): insert route path: /apisix/admin/ dataprt: 1, context: init_worker_by_lua*
2021/03/15 10:19:02 [info] 46#46: *2 [lua] radixtree.lua:346: pre_insert_route(): path: /apisix/admin/stream_routes/ operator: <=, context: init_worker_by_lua*
2021/03/15 10:19:02 [info] 46#46: *2 [lua] radixtree.lua:234: insert_route(): insert route path: /apisix/admin/stream_routes/ dataprt: 2, context: init_worker_by_lua*
2021/03/15 10:19:02 [info] 46#46: *2 [lua] radixtree.lua:346: pre_insert_route(): path: /apisix/admin/plugins/list operator: =, context: init_worker_by_lua*
2021/03/15 10:19:02 [info] 46#46: *2 [lua] radixtree.lua:346: pre_insert_route(): path: /apisix/admin/plugins/reload operator: =, context: init_worker_by_lua*
2021/03/15 10:19:02 [notice] 46#46: *2 [lua] timers.lua:70: init_worker(): succeed to create background timer, context: init_worker_by_lua*
2021/03/15 10:19:02 [info] 49#49: *1 [lua] radixtree.lua:346: pre_insert_route(): path: /apisix/admin/ operator: <=, context: init_worker_by_lua*
2021/03/15 10:19:02 [info] 49#49: *1 [lua] radixtree.lua:234: insert_route(): insert route path: /apisix/admin/ dataprt: 1, context: init_worker_by_lua*
2021/03/15 10:19:02 [info] 49#49: *1 [lua] radixtree.lua:346: pre_insert_route(): path: /apisix/admin/stream_routes/ operator: <=, context: init_worker_by_lua*
2021/03/15 10:19:02 [info] 49#49: *1 [lua] radixtree.lua:234: insert_route(): insert route path: /apisix/admin/stream_routes/ dataprt: 2, context: init_worker_by_lua*
2021/03/15 10:19:02 [info] 49#49: *1 [lua] radixtree.lua:346: pre_insert_route(): path: /apisix/admin/plugins/list operator: =, context: init_worker_by_lua*
2021/03/15 10:19:02 [info] 49#49: *1 [lua] radixtree.lua:346: pre_insert_route(): path: /apisix/admin/plugins/reload operator: =, context: init_worker_by_lua*
2021/03/15 10:19:02 [notice] 49#49: *1 [lua] timers.lua:70: init_worker(): succeed to create background timer, context: init_worker_by_lua*
2021/03/15 10:19:02 [error] 46#46: init_worker_by_lua error: /usr/local/apisix/apisix/plugin.lua:264: attempt to index upvalue 'local_conf' (a nil value)
stack traceback:
        /usr/local/apisix/apisix/plugin.lua:264: in function 'load'
        /usr/local/apisix/apisix/plugin.lua:502: in function 'init_worker'
        /usr/local/apisix/apisix/init.lua:112: in function 'http_init_worker'
        init_worker_by_lua:2: in main chunk
2021/03/15 10:19:02 [info] 49#49: *1 [lua] plugin.lua:143: plugins_eq(): plugin list changed, context: init_worker_by_lua*

[anjia0532]

怀疑是因为k8s的configMap导致的 plugin.lua core.config.local_conf(true) 和 config-etcd.lua 中的config_local.local_conf() 最终调用 file.read_yaml_conf() 时得到的 nil

因为我改成本地卷的配置文件，就可以了

[membphis]

can we close this issue now? @anjia0532

[anjia0532]

maybe not. I think this is a bug. local system file is ok ,but k8s configMap mapping as file not work. @membphis

我觉得这是一个bug，因为读取本地文件没问题，但是在k8s环境下，将configMap映射为文件，就读不到。

[tokers]

@anjia0532 It seems APISIX needs the write permission when opening config-default.yaml, What's the volumes and volumeMounts items in the APISIX Deployment?

[tokers]

@anjia0532 It seems APISIX needs the write permission when opening config-default.yaml, What's the volumes and volumeMounts items in the APISIX Deployment?

This is absolutely an uncalled for, we may investigate and fix it.

[anjia0532]

thanks and waiting for this. @tokers

[tokers]

thanks and waiting for this. @tokers

Could you please provide the file permission of apisix config.yaml in the container.

[anjia0532]

thanks and waiting for this. @tokers

Could you please provide the file permission of apisix config.yaml in the container.

yeah, default permission is 400 , 6xx(rw,xx,xx) is ok. thanks.

why apisix need to write permission abount config.yaml ?