feat(acls) add admin endpoints to interact with acls

[hbagdi]

• /acls/ to paginate through all acls for all consumers
• /acls/:acl_id/consumer to retrieve the Consumer associated with an acl

Issues resolved

Fix #2188
Close #2371

[hbagdi]

Two features that are missing:

Retrieve consumers associated with a group

An endpoint such as /acls/:groupname/consumerswould be immensely helpful.
Could it be confused with /acls/:acl_id/consumer?
If not, I can add it to this PR.

Retrieve ACL group names

I feel having an endpoint GET /aclgroups to return all the group names would be pretty helpful.

The problem I'm facing, it seems there is no straightforward way of doing:

select distinct group from acls;

Is there away around of doing this, or we will need to update the core DB logic? I could take a stab at it if there is another use case for such a query.

IMHO, this would be pretty useful for admin purposes.

Another issue is we don't have group as a resource or entity really so this is not RESTful.

An inefficient implementation could be a brute-force approach, which traverses through an entire paginated set of ACLs and figure out the unique groups.

@thibaultcha Thoughts?

[thibaultcha]

@hbagdi Very nice! This is super exciting.

The issue indeed is that there is no way to distinctively select groups. The DAO doesn't support this feature, and cannot, as the limitation comes from Cassandra itself (and the data model chosen for this entity).

An inefficient implementation could be a brute-force approach, which traverses through an entire paginated set of ACLs and figure out the unique groups.

If carefully written this could be fine but it isn't ideal that's for sure. I'd rather not risk it, iterating over entire tables/column families (in Cassandra) is very much a discouraged practice. We have users and customers using those features more or less extensively, so this could cause serious performance issues (potentially slowing down an entire cluster) for those heavy users.

An endpoint such as /acls/:groupname/consumers would be immensely helpful.

If we write a migration to create an index on the group column (and make the appropriate changes in the schema), this could be fine considering the supposedly low cardinality of this attribute. The /aclsgroups endpoint would still be out of reach though.

Ideally, we come up with a better ACL solution once our new data model is completed (we are working on it).

[hbagdi]

@thibaultcha
I understand the root causes of this not being possible.
Let's push these features for a later time.
I'll open an issue just for tracking purpose.

I've been thinking a little bit about grouping or tagging of APIs/Consumers.
Group or Tag as a Kong entity to make cohorts of APIs and/or Consumers.
Such a feature could be supported by a plugin but then it would create interdependency between plugins, which is not a problem but not an ideal solution as well.

It's also possible to start an experimenting with such a plugin and then move it into the core if there's traction.

Although, supporting it from the core would give the ability to maintain/manage a group of entities with ease.

[hbagdi]

@thibaultcha Since, this PR is related with other *-auth changes in admin API, can we will include this in the upcoming release?

[thibaultcha]

I just had a look and it seems like the acls column family has such an index already for both PostgreSQL and Cassandra. It seems like we could do the /acls/:group/consumers endpoint after all!

However, I'm not sure how well this will play with Lapis and the other endpoint we register: /acls/:acl_id/consumer...

[thibaultcha]

I had a doubt and tried to run this test with Cassandra, and encountered a few errors that made it fail. Here is the patch I had to add to make this test succeed with Cassandra:

diff --git a/spec/03-plugins/19-acl/01-api_spec.lua b/spec/03-plugins/19-acl/01-api_spec.lua
index 99ad25f9..31435e59 100644
--- a/spec/03-plugins/19-acl/01-api_spec.lua
+++ b/spec/03-plugins/19-acl/01-api_spec.lua
@@ -331,7 +331,11 @@ describe("Plugin: acl (API)", function()
 
         res = assert(admin_client:send {
           method = "GET",
-          path = "/acls?size=3&offset=" .. json_1.offset,
+          path = "/acls",
+          query = {
+            size = 3,
+            offset = json_1.offset,
+          }
         })
         body = assert.res_status(200, res)
         local json_2 = cjson.decode(body)
@@ -340,7 +344,10 @@ describe("Plugin: acl (API)", function()
         assert.equal(6, json_2.total)
 
         assert.not_same(json_1.data, json_2.data)
-        assert.is_nil(json_2.offset) -- last page
+        -- Disabled: on Cassandra, the last page still returns a
+        -- next_page token, and thus, an offset proprty in the
+        -- response of the Admin API.
+        --assert.is_nil(json_2.offset) -- last page
       end)
       it("retrieves acls for a consumer_id", function()
         local res = assert(admin_client:send {

It is likely the case for other contributions we've recently commited that embed the same test.

[thibaultcha]

I just gave it a try, and, lucky for us, it seems like Lapis is perfectly able to make the distinction between the two endpoints!

  ["/acls/:acl_group/consumers"] = {
    GET = function(self, dao_factory, helpers)
      return helpers.responses.send_HTTP_OK("/acls/:acl_group/consumers")
    end
  }

$ http ":8001/acls/$(uuid)/consumer/"
HTTP/1.1 404 Not Found
...

{
    "message": "Not found"
}

$ http ":8001/acls/$(uuid)/consumers/"
HTTP/1.1 200 OK
...

{
    "message": "/acls/:acl_group/consumers"
}

[thibaultcha]

I have just opened #3051 to address the failures when using Cassandra for your previous contributions. Mind giving it a look?

Also, FYI, our merging window will be closed this Monday, before the release on Wednesday.

[hbagdi]

Regarding:

I just gave it a try, and, lucky for us, it seems like Lapis is perfectly able to make the distinction between the two endpoints!

["/acls/:acl_group/consumers"] = {
    GET = function(self, dao_factory, helpers)
      return helpers.responses.send_HTTP_OK("/acls/:acl_group/consumers")
    end
  }

I had tried the same thing and in fact the endpoint was not an issue but processing the request with pagination support.

Since, we can find Acls belonging to a single acl_group but then we need to perform a lookup for the Consumer of every Acl with acl_group.

In such a case, if a group has a few thousand Consumers then we need pagination support for the endpoint.

A hack comes to mind, where we could internally paginate based on acl_group and not Consumer.
What I mean, internally we do a paginated GET /acls?group=foo. And replace the data in response with a Consumer array instead of an Acl array.
On a request for next page, we lookup Acls and then replace Consumers.

I have bandwidth to include further changes today.

[hbagdi]

@thibaultcha Good catch with the pagination support.
I was thinking about documenting this but since this is very harmless, we can skip for better end-user understanding and experience.

I've included your patch in the updated commit on this PR.

[thibaultcha]

Yes, the pagination in such a case might be tricky and might require custom usage of the DAO (the "CRUD helpers" are, after all, just helpers for the most common tasks).

Maybe we should leave this out of 0.11.2 and I am thinking of simply taking this PR as is for now.

[hbagdi]

Yes, let's take this PR as for 0.11.2. Let me know if we need to update anything.

…

On Nov 26, 2017 1:11 PM, "Thibault Charbonnier" ***@***.***> wrote: Yes, the pagination in such a case might be tricky and might require custom usage of the DAO (the "CRUD helpers" are, after all, just helpers for the most common tasks. Maybe we should leave this out of 0.11.2 and I am thinking of simply taking this PR as is for now. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3039 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AEeiOfWExGljB5rVkFGMW77gx0iKUT6Xks5s6dPqgaJpZM4Qi3Zw> .

[thibaultcha]

I think that we are missing a self.params.acl_id = nil instruction here. Not an issue with find_consumer_by_username_or_id, but it could become one further down the road in one of the HTTP verb handlers. Better safe than sorry!

I will take care of this while merging, no worries.

[thibaultcha]

Manually merged with some minor modifications - thank you very much for your work!

[PatrikUnterstell]

Hi everyone,
I've seen all the discussion about the endpoint /acls/:acl_group/consumers.
Anyone have any news on this?
I need a lot to list all consumers from a group.
Thanks!