Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Declarative config does not produce the same result for Consumer ACLS groups as admin api #4909

Closed
gchristidis opened this issue Aug 14, 2019 · 2 comments

Comments

@gchristidis
Copy link
Contributor

commented Aug 14, 2019

Summary

I am looking to move our Kong configuration from a scripted set of curl commands using the admin api to using the declarative config.
The issue is that I have 2 consumers with the same ACLS group names and the end result I end up with using declarative config is different, it is missing ACLS groups for a consumer. In the example below if i change the declarative config so that each consumer has different group names then it creates everything as expected, its just when the 2 consumers have the same groups.

In this cut down example, using the admin api I create 2 consumers with both consumers having the same 2 ACLS group names, this results in 2 consumer and 4 ACLS entries. Converting this to declarative config I end up with 2 Consumers and only 2 ACLS entries for 1 of the consumers, the 2 ACLS groups for the other consumer is missing.

I am running Kong community edition 1.2.1 with a Datastax Cassandra database.

Steps To Reproduce

  1. Start with a clean database (service kong stop;kong migrations reset; kong migrations bootstrap;service kong start)
  2. Run the following curl commands against the admin api to create the consumers and ACLS groups.
curl -s -X PUT  http://localhost:8001/consumers/web-portal         --data-urlencode username=web-portal
curl -s -X POST http://localhost:8001/consumers/web-portal/acls    --data-urlencode group=legacy-group
curl -s -X POST http://localhost:8001/consumers/web-portal/acls    --data-urlencode group=sdk-group
curl -s -X PUT  http://localhost:8001/consumers/device-portal      --data-urlencode username=device-portal
curl -s -X POST http://localhost:8001/consumers/device-portal/acls --data-urlencode group=legacy-group
curl -s -X POST http://localhost:8001/consumers/device-portal/acls --data-urlencode group=sdk-group
  1. Querying the consumers and acls endpoints give you the following config showing the 2 consumers and 4 ALCS entries (2 groups for each of the 2 consumers)
Consumers
{
  "data": [
    {
      "created_at": 1565750915,
      "custom_id": null,
      "id": "9ef86a88-a15e-4cf6-b1b9-f20f688feabf",
      "tags": null,
      "username": "device-portal"
    },
    {
      "created_at": 1565750915,
      "custom_id": null,
      "id": "f4202aac-dcf8-42c9-b45e-fcca557a2d00",
      "tags": null,
      "username": "web-portal"
    }
  ],
  "next": null
}
ACLS
{
  "data": [
    {
      "consumer": {
        "id": "9ef86a88-a15e-4cf6-b1b9-f20f688feabf"
      },
      "created_at": 1565750915,
      "group": "legacy-group",
      "id": "3794452b-bd40-4bb2-b18f-0cdaeaa0ba25"
    },
    {
      "consumer": {
        "id": "9ef86a88-a15e-4cf6-b1b9-f20f688feabf"
      },
      "created_at": 1565750915,
      "group": "sdk-group",
      "id": "db0e20cb-1310-4910-96a5-97bb4e928fee"
    },
    {
      "consumer": {
        "id": "f4202aac-dcf8-42c9-b45e-fcca557a2d00"
      },
      "created_at": 1565750915,
      "group": "sdk-group",
      "id": "8187c21a-b728-43d9-bffd-1db7349a8385"
    },
    {
      "consumer": {
        "id": "f4202aac-dcf8-42c9-b45e-fcca557a2d00"
      },
      "created_at": 1565750915,
      "group": "legacy-group",
      "id": "3a08e28e-f003-4831-af94-8d6869a2944d"
    }
  ],
  "next": null
}
  1. Start with a clean database (service kong stop;kong migrations reset; kong migrations bootstrap;service kong start)
  2. Using this declarative config file
_format_version: "1.1"
consumers:
  - username: web-portal
    acls:
      - group: legacy-group
      - group: sdk-group
  - username: device-portal
    acls:
      - group: legacy-group
      - group: sdk-group
  1. Import it using kong config db_import kong.yaml
  2. Querying the consumers and acls endpoints give you the following config, showing the same 2 consumers but now only 2 ACLS entries for 1 of the consumers and no ACLS entries for the 2nd consumer.
Consumers
{
  "data": [
    {
      "created_at": 1565751187,
      "custom_id": null,
      "id": "c16b28b9-3f7c-57cd-a250-5fea615c0b43",
      "tags": null,
      "username": "device-portal"
    },
    {
      "created_at": 1565751187,
      "custom_id": null,
      "id": "0dca855a-3cc4-5db5-ada6-47c2a5a6946d",
      "tags": null,
      "username": "web-portal"
    }
  ],
  "next": null
}
ACLs
{
  "data": [
    {
      "consumer": {
        "id": "c16b28b9-3f7c-57cd-a250-5fea615c0b43"
      },
      "created_at": 1565751187,
      "group": "legacy-group",
      "id": "725e2d56-3faf-55fa-b754-9f1972d81c23"
    },
    {
      "consumer": {
        "id": "c16b28b9-3f7c-57cd-a250-5fea615c0b43"
      },
      "created_at": 1565751187,
      "group": "sdk-group",
      "id": "28946e10-468a-5b1b-a39f-51ed32070d1d"
    }
  ],
  "next": null
}

Additional Details & Logs

  • Kong version (1.2.1)
  • Kong configuration
{
  "configuration": {
    "admin_acc_logs": "/usr/local/kong/logs/admin_access.log",
    "admin_access_log": "logs/admin_access.log",
    "admin_error_log": "logs/error.log",
    "admin_listen": [
      "0.0.0.0:8001"
    ],
    "admin_listeners": [
      {
        "http2": false,
        "ip": "0.0.0.0",
        "listener": "0.0.0.0:8001",
        "port": 8001,
        "proxy_protocol": false,
        "ssl": false,
        "transparent": false
      }
    ],
    "admin_ssl_cert_default": "/usr/local/kong/ssl/admin-kong-default.crt",
    "admin_ssl_cert_key_default": "/usr/local/kong/ssl/admin-kong-default.key",
    "admin_ssl_enabled": false,
    "anonymous_reports": true,
    "cassandra_consistency": "LOCAL_QUORUM",
    "cassandra_contact_points": [
      "oltp-cassandra-1.private.george1.do.eng.vixpulse.com"
    ],
    "cassandra_data_centers": [
      "OLTP:1"
    ],
    "cassandra_keyspace": "pulse_api_gateway",
    "cassandra_lb_policy": "RequestRoundRobin",
    "cassandra_password": "******",
    "cassandra_port": 9042,
    "cassandra_repl_factor": 1,
    "cassandra_repl_strategy": "NetworkTopologyStrategy",
    "cassandra_schema_consensus_timeout": 60000,
    "cassandra_ssl": false,
    "cassandra_ssl_verify": false,
    "cassandra_timeout": 60000,
    "cassandra_username": "pulse",
    "client_body_buffer_size": "8k",
    "client_max_body_size": "0",
    "client_ssl": false,
    "client_ssl_cert_default": "/usr/local/kong/ssl/kong-default.crt",
    "client_ssl_cert_key_default": "/usr/local/kong/ssl/kong-default.key",
    "database": "cassandra",
    "db_cache_ttl": 0,
    "db_cache_warmup_entities": [
      "services",
      "plugins"
    ],
    "db_resurrect_ttl": 30,
    "db_update_frequency": 300,
    "db_update_propagation": 2,
    "dns_error_ttl": 1,
    "dns_hostsfile": "/etc/hosts",
    "dns_no_sync": false,
    "dns_not_found_ttl": 30,
    "dns_order": [
      "LAST",
      "SRV",
      "A",
      "CNAME"
    ],
    "dns_resolver": {},
    "dns_stale_ttl": 4,
    "enabled_headers": {
      "Server": true,
      "Via": true,
      "X-Kong-Proxy-Latency": true,
      "X-Kong-Upstream-Latency": true,
      "X-Kong-Upstream-Status": false,
      "latency_tokens": true,
      "server_tokens": true
    },
    "error_default_type": "text/plain",
    "headers": [
      "server_tokens",
      "latency_tokens"
    ],
    "kong_env": "/usr/local/kong/.kong_env",
    "loaded_plugins": {
      "acl": true,
      "aws-lambda": true,
      "azure-functions": true,
      "basic-auth": true,
      "bot-detection": true,
      "correlation-id": true,
      "cors": true,
      "datadog": true,
      "file-log": true,
      "hmac-auth": true,
      "http-log": true,
      "ip-restriction": true,
      "jwt": true,
      "key-auth": true,
      "kubernetes-sidecar-injector": true,
      "ldap-auth": true,
      "loggly": true,
      "oauth2": true,
      "post-function": true,
      "pre-function": true,
      "prometheus": true,
      "proxy-cache": true,
      "rate-limiting": true,
      "request-size-limiting": true,
      "request-termination": true,
      "request-transformer": true,
      "response-ratelimiting": true,
      "response-transformer": true,
      "statsd": true,
      "syslog": true,
      "tcp-log": true,
      "udp-log": true,
      "zipkin": true
    },
    "log_level": "notice",
    "lua_package_cpath": "",
    "lua_package_path": "./?.lua;./?/init.lua;",
    "lua_socket_pool_size": 30,
    "lua_ssl_verify_depth": 1,
    "mem_cache_size": "128m",
    "nginx_acc_logs": "/usr/local/kong/logs/access.log",
    "nginx_admin_directives": {},
    "nginx_conf": "/usr/local/kong/nginx.conf",
    "nginx_daemon": "on",
    "nginx_err_logs": "/usr/local/kong/logs/error.log",
    "nginx_http_directives": [
      {
        "name": "lua_shared_dict",
        "value": "prometheus_metrics 5m"
      }
    ],
    "nginx_kong_conf": "/usr/local/kong/nginx-kong.conf",
    "nginx_kong_stream_conf": "/usr/local/kong/nginx-kong-stream.conf",
    "nginx_optimizations": true,
    "nginx_pid": "/usr/local/kong/pids/nginx.pid",
    "nginx_proxy_directives": {},
    "nginx_sproxy_directives": {},
    "nginx_stream_directives": {},
    "nginx_worker_processes": "auto",
    "origins": {},
    "pg_database": "kong",
    "pg_host": "127.0.0.1",
    "pg_max_concurrent_queries": 0,
    "pg_port": 5432,
    "pg_semaphore_timeout": 60000,
    "pg_ssl": false,
    "pg_ssl_verify": false,
    "pg_timeout": 60000,
    "pg_user": "kong",
    "plugins": [
      "bundled"
    ],
    "prefix": "/usr/local/kong",
    "proxy_access_log": "logs/access.log",
    "proxy_error_log": "logs/error.log",
    "proxy_listen": [
      "0.0.0.0:8000",
      "0.0.0.0:8443 ssl"
    ],
    "proxy_listeners": [
      {
        "http2": false,
        "ip": "0.0.0.0",
        "listener": "0.0.0.0:8000",
        "port": 8000,
        "proxy_protocol": false,
        "ssl": false,
        "transparent": false
      },
      {
        "http2": false,
        "ip": "0.0.0.0",
        "listener": "0.0.0.0:8443 ssl",
        "port": 8443,
        "proxy_protocol": false,
        "ssl": true,
        "transparent": false
      }
    ],
    "proxy_ssl_enabled": true,
    "real_ip_header": "X-Forwarded-For",
    "real_ip_recursive": "off",
    "router_consistency": "strict",
    "ssl_cert": "/usr/local/kong/ssl/kong-default.crt",
    "ssl_cert_csr_default": "/usr/local/kong/ssl/kong-default.csr",
    "ssl_cert_default": "/usr/local/kong/ssl/kong-default.crt",
    "ssl_cert_key": "/usr/local/kong/ssl/kong-default.key",
    "ssl_cert_key_default": "/usr/local/kong/ssl/kong-default.key",
    "ssl_cipher_suite": "modern",
    "ssl_ciphers": "ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256",
    "ssl_preread_enabled": true,
    "stream_listen": [
      "off"
    ],
    "stream_listeners": {},
    "trusted_ips": [
      "172.31.0.0/24"
    ],
    "upstream_keepalive": 60
  },
  "hostname": "api-gateway-1",
  "lua_version": "LuaJIT 2.1.0-beta3",
  "node_id": "bc2f7ed4-619d-438d-ac0b-f6863291c123",
  "plugins": {
    "available_on_server": {
      "acl": true,
      "aws-lambda": true,
      "azure-functions": true,
      "basic-auth": true,
      "bot-detection": true,
      "correlation-id": true,
      "cors": true,
      "datadog": true,
      "file-log": true,
      "hmac-auth": true,
      "http-log": true,
      "ip-restriction": true,
      "jwt": true,
      "key-auth": true,
      "kubernetes-sidecar-injector": true,
      "ldap-auth": true,
      "loggly": true,
      "oauth2": true,
      "post-function": true,
      "pre-function": true,
      "prometheus": true,
      "proxy-cache": true,
      "rate-limiting": true,
      "request-size-limiting": true,
      "request-termination": true,
      "request-transformer": true,
      "response-ratelimiting": true,
      "response-transformer": true,
      "statsd": true,
      "syslog": true,
      "tcp-log": true,
      "udp-log": true,
      "zipkin": true
    },
    "enabled_in_cluster": []
  },
  "prng_seeds": {
    "pid: 20436": 183701651101,
    "pid: 20446": 151229761116,
    "pid: 20447": 549715488162
  },
  "tagline": "Welcome to kong",
  "timers": {
    "pending": 6,
    "running": 0
  },
  "version": "1.2.1"
}
  • Operating system Ubuntu
@hbagdi

This comment has been minimized.

Copy link
Member

commented Aug 14, 2019

This looks another case of #4817.

With Kong 1.3.0rc1, I verified in the DB-less mode locally and it correctly creates the ACLs as expected.
Could you please test this with Kong 1.3.0rc1 and close this issue if it is solved?
Thanks!

@gchristidis

This comment has been minimized.

Copy link
Contributor Author

commented Aug 15, 2019

tested under 1.3.0rc1 and this no longer has the issue

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.