help request: error loading custom plugin

[satishviswanathan]

Description

I followed this link to see if I can have a custom plugin loaded.

https://apisix.apache.org/docs/apisix/next/plugin-develop/

But getting this error

nginx: [alert] failed to load the 'resty.core' module (https://github.com/openresty/lua-resty-core); ensure you are using an OpenResty release from https://openresty.org/en/download.html (reason: /opt/apisix/plugins/3rd-party.lua:2: loop or previous error loading module 'apisix.core') in /usr/local/apisix/conf/nginx.conf:415

Not sure what I'm missing, any help is appreciated.

apisix config.yml

apisix:
  node_listen: 9080              # APISIX listening port
  enable_ipv6: false
  extra_lua_path: "/opt/apisix/plugins/3rd-party.lua"
  plugins:                           # plugin list (sorted by priority)
    - 3rd-party                      # priority: 2561
  enable_control: true
  control:
    ip: "0.0.0.0"
    port: 9092

deployment:
  admin:
    allow_admin:               # https://nginx.org/en/docs/http/ngx_http_access_module.html#allow
      - 0.0.0.0/0              # We need to restrict ip access rules for security. 0.0.0.0/0 is for test.

    admin_key:
      - name: "admin"
        key: edd1c9f034335f136f87ad84b625c8f1
        role: admin                 # admin: manage all configuration data

      - name: "viewer"
        key: 4054f7cf07e344346cd3f287985e76a2
        role: viewer

  etcd:
    host:                           # it's possible to define multiple etcd hosts addresses of the same etcd cluster.
      - "http://etcd:2379"          # multiple etcd address
    prefix: "/apisix"               # apisix configurations prefix
    timeout: 30                     # 30 seconds

plugin_attr:
  prometheus:
    export_addr:
      ip: "0.0.0.0"
      port: 9091

Apisix docker-compose.yml

version: "3"

services:
  apisix-dashboard:
    image: apache/apisix-dashboard:3.0.1-alpine
    restart: always
    volumes:
    - ./dashboard_conf/conf.yaml:/usr/local/apisix-dashboard/conf/conf.yaml
    ports:
    - "8080:9000"
    networks:
      apisix:

  apisix:
    image: apache/apisix:${APISIX_IMAGE_TAG:-3.6.0-debian}
    restart: always
    volumes:
      - ./apisix_conf/config.yaml:/usr/local/apisix/conf/config.yaml:ro
      - ./auth:/usr/local/apisix/auth
      - ./custom-plugins:/opt/apisix/plugins:ro
    depends_on:
      - etcd
    ##network_mode: host
    ports:
      - "9180:9180/tcp"
      - "9080:9080/tcp"
      - "9091:9091/tcp"
      - "9443:9443/tcp"
      - "9092:9092/tcp"
    networks:
      apisix:

  etcd:
    image: bitnami/etcd:3.4.15
    restart: always
    volumes:
      - etcd_data:/bitnami/etcd
    environment:
      ETCD_ENABLE_V2: "true"
      ALLOW_NONE_AUTHENTICATION: "yes"
      ETCD_ADVERTISE_CLIENT_URLS: "http://etcd:2379"
      ETCD_LISTEN_CLIENT_URLS: "http://0.0.0.0:2379"
    ports:
      - "2379:2379/tcp"
    networks:
      apisix:

  web1:
    image: nginx:1.19.0-alpine
    restart: always
    volumes:
      - ./upstream/web1.conf:/etc/nginx/nginx.conf
    ports:
      - "9081:80/tcp"
    environment:
      - NGINX_PORT=80
    networks:
      apisix:

  web2:
    image: nginx:1.19.0-alpine
    restart: always
    volumes:
      - ./upstream/web2.conf:/etc/nginx/nginx.conf
    ports:
      - "9082:80/tcp"
    environment:
      - NGINX_PORT=80
    networks:
      apisix:

  prometheus:
    image: prom/prometheus:v2.25.0
    restart: always
    volumes:
      - ./prometheus_conf/prometheus.yml:/etc/prometheus/prometheus.yml
    ports:
      - "9090:9090"
    networks:
      apisix:

  grafana:
    image: grafana/grafana:7.3.7
    restart: always
    ports:
      - "3000:3000"
    volumes:
      - "./grafana_conf/provisioning:/etc/grafana/provisioning"
      - "./grafana_conf/dashboards:/var/lib/grafana/dashboards"
      - "./grafana_conf/config/grafana.ini:/etc/grafana/grafana.ini"
    networks:
      apisix:
  
  postgres:
      image: postgres:12.2
      container_name: postgres
      environment:
        POSTGRES_DB: keycloak
        POSTGRES_USER: keycloak
        POSTGRES_PASSWORD: password
      networks:
        apisix:
      depends_on:
        - apisix
        
  keycloak:
      image: jboss/keycloak:9.0.2
      container_name: keycloak
      environment:
        DB_VENDOR: POSTGRES
        DB_ADDR: postgres
        DB_DATABASE: keycloak
        DB_USER: keycloak
        DB_PASSWORD: password
        KEYCLOAK_USER: admin
        KEYCLOAK_PASSWORD: password
        PROXY_ADDRESS_FORWARDING: "true"
      ports:
        - 3001:8080
      networks:
        apisix:
      depends_on:
        - postgres 
networks:
  apisix:
    driver: bridge

volumes:
  etcd_data:
    driver: local

3rd-party.lua

local require       = require
local core          = require("apisix.core")
local plugin_name   = "3rd-party"


local schema = {
    type = "object",
    properties = {
        body = {
            description = "body to replace response.",
            type = "string"
        },
    },
    required = {"body"},
}

local plugin_name = "3rd-party"

local _M = {
    version = 0.1,
    priority = 12,
    name = plugin_name,
    schema = schema,
}


function _M.check_schema(conf)
    return core.schema.check(schema, conf)
end


function _M.access(conf, ctx)
    return 200, conf.body
end

Environment

• APISIX version (run apisix version):
• Operating system (run uname -a):
• OpenResty / Nginx version (run openresty -V or nginx -V):
• etcd version, if relevant (run curl http://127.0.0.1:9090/v1/server_info):
• APISIX Dashboard version, if relevant:
• Plugin runner version, for issues related to plugin runners:
• LuaRocks version, for installation issues (run luarocks --version):

[shreemaan-abhishek]

Hi, the configs you have shared look scary as they have been misunderstood as markdown files by github. Please use "```" wherever necessary.

[satishviswanathan]

@shreemaan-abhishek Thank you for your response. Can you check if my config are good to load a custom plugin

Apologies should have uploaded the file for better readability. I have renamed the extension of the files so that it can be uploaded.

3rd-party.txt
docker-compose.txt
config.txt (apisix config)
conf.txt (dashboard config)

[kayx23]

Tru updating extra_lua-path to /opt/\?.lua

[satishviswanathan]

@kayx23 initially I tired that way as suggested in the documentation but the plugin didn't get picked up .. but when I gave the full path it's picking up but getting errors as well. I'm also using apisix dashboard. I have updated with it's config as well.

[kayx23]

I have seem that error when using the full path as well. Here is a newer piece of documentation on the topic for reference:

https://docs.api7.ai/apisix/how-to-guide/custom-plugins/create-plugin-in-lua#update-apisix-configuration-file

I don't have knowledge of the dashboard unfortunately.

[satishviswanathan]

@kayx23 - I tried that but still it didn't work, however I resolved that issue by copying the custom plugin under the plugins folder where all the plugins are residing (apisix/plugins/) and then updated the config-default.yaml with the new custom plugin. So I don't have the extra_lua_path any more. With these changes I was able to enable the custom plugin. However as soon as the plugin was enabled I can see the following logs from APISIX.

Do you know what this error is ? Any help is appreciated

2024/02/16 17:05:39 [error] 212#212: *148682 [lua] config_etcd.lua:883: failed to fetch data from etcd: /usr/local/openresty/lualib/resty/lrucache.lua:229: table index is nil
apisix_1 | stack traceback:
apisix_1 | /usr/local/openresty/lualib/resty/lrucache.lua:229: in function 'set'
apisix_1 | /usr/local/apisix/apisix/core/lrucache.lua:97: in function 'cached_validator'
apisix_1 | /usr/local/apisix/apisix/core/schema.lua:49: in function 'get_validator'
apisix_1 | /usr/local/apisix/apisix/core/schema.lua:60: in function 'check_schema'
apisix_1 | /usr/local/apisix/apisix/plugin.lua:887: in function 'check_single_plugin_schema'
apisix_1 | /usr/local/apisix/apisix/plugin.lua:1041: in function 'check_schema'
apisix_1 | /usr/local/apisix/apisix/plugin.lua:1090: in function 'plugin_checker'
apisix_1 | /usr/local/apisix/apisix/http/route.lua:122: in function 'checker'
apisix_1 | /usr/local/apisix/apisix/core/config_etcd.lua:677: in function 'sync_data'
apisix_1 | /usr/local/apisix/apisix/core/config_etcd.lua:827: in function </usr/local/apisix/apisix/core/config_etcd.lua:817>
apisix_1 | [C]: in function 'xpcall'
apisix_1 | /usr/local/apisix/apisix/core/config_etcd.lua:817: in function </usr/local/apisix/apisix/core/config_etcd.lua:795>, etcd key: /apisix/routes, context: ngx.timer

[kayx23]

Any help here @shreemaan-abhishek

[satishviswanathan]

quick update I got the plugins loaded by settings the extra_lua_path "/opt/apisix/plugins/?.lua" and in my apisix docker I gave it as follows.

apisix:
image: apache/apisix:${APISIX_IMAGE_TAG:-3.8.0-debian}
restart: always
volumes:
- ./apisix_conf/config.yaml:/usr/local/apisix/conf/config.yaml:ro
- ./auth:/usr/local/apisix/auth
- ./custom-plugins:/opt/apisix/plugins/apisix/plugins:ro

The change was I gave the path as "/opt/apisix/plugins/apisix/plugins". If I give anyother path it wouldn't work. For example initially I was giving it as "/opt/apisix/plugins/".

Now apis six is running with the custom plugin enabled and I'm bale to hit by endpoint and get the response back. I'm not sure if the custom plugin got executed. However I'm still seeing that error in the background

2024/02/16 21:54:26 [error] 52#52: *17900 [lua] config_etcd.lua:883: failed to fetch data from etcd: /usr/local/openresty/lualib/resty/lrucache.lua:229: table index is nil
apisix_1 | stack traceback:
apisix_1 | /usr/local/openresty/lualib/resty/lrucache.lua:229: in function 'set'
apisix_1 | /usr/local/apisix/apisix/core/lrucache.lua:97: in function 'cached_validator'
apisix_1 | /usr/local/apisix/apisix/core/schema.lua:49: in function 'get_validator'
apisix_1 | /usr/local/apisix/apisix/core/schema.lua:60: in function 'check_schema'
apisix_1 | /usr/local/apisix/apisix/plugin.lua:887: in function 'check_single_plugin_schema'
apisix_1 | /usr/local/apisix/apisix/plugin.lua:1041: in function 'check_schema'
apisix_1 | /usr/local/apisix/apisix/plugin.lua:1090: in function 'checker'
apisix_1 | /usr/local/apisix/apisix/core/config_etcd.lua:677: in function 'sync_data'
apisix_1 | /usr/local/apisix/apisix/core/config_etcd.lua:827: in function </usr/local/apisix/apisix/core/config_etcd.lua:817>
apisix_1 | [C]: in function 'xpcall'
apisix_1 | /usr/local/apisix/apisix/core/config_etcd.lua:817: in function </usr/local/apisix/apisix/core/config_etcd.lua:795>, etcd key: /apisix/global_rules, context: ngx.timer

Custom plugin. I don't have much knowledge on lua, just trying to see if it can be loaded and see it in action so that later we can built custom plugins according to our needs

local core = require("apisix.core")
local io = require("io")
local ngx = ngx
local plugin_name = "3rd-party"

local plugin_schema = {
type = "object",
properties = {
path = {
description = "path to replace response.",
type = "string"
},
},
required = {"path"},
}

local _M = {
version = 1.0,
priority = 12,
name = plugin_name,
schema = plugin_schema
}

function _M.check_schema(conf)
return core.schema.check(consumer_schema, conf)
end

function _M.access(conf, ctx)
return 200, conf.body
end

function _M.log(conf, ctx)
core.log.warn(core.json.encode(conf))

end

return _M

[satishviswanathan]

My custom plugin worked.

Loading issue resolved by giving the following volume path in the docker to load the custom plugin.

• ./custom-plugins:/opt/apisix/plugins/apisix/plugins:ro

The above error might be a syntax issue. I just got a copy of existing plugin and wrote a log message and it worked.

[kayx23]

Ok, any other question pending on this thread?

[satishviswanathan]

Thank you all for the help.

My objective of the custom plugin is to execute an existing plugin "plugins.authz-casbin". Can you show me how to invoke a plugin from a custom plugin.