How to solve the huge data when I use persistent Database?

[rts-gordon]

Hi hsluoyz,

I use casbin gorm_adapter to store policies and roles in database, when I run the program, all policies will be load into memory. But if there are millions policies, I can not do this. How to solve the huge policy data? Can I check only one policy from database?

Cheers
Gordon

[hsluoyz]

Casbin supports only loading a section of policy rules via filters. See here: https://github.com/casbin/casbin#policy-enforcement-at-scale

[rts-gordon]

Cool.
My scenario is: millions users login the system, every user join different roles, and there are only 100 'p' records, but millions of 'g' records(See below image). Even I use the filtered_adapter, I can not load millions of 'g' records into memory. So can I load all 'p' records at program startup but do not load 'g' records, load one special users 'g' record only when this user access system and check the user's permission?

[hsluoyz]

For the filtered adapter solution, you can use:

import (
    "github.com/casbin/casbin"
)

enforcer := casbin.NewEnforcer()

adapter := fileadapter.NewFilteredAdapter("examples/rbac_policy.csv")
enforcer.InitWithAdapter("examples/rbac_model.conf", adapter)

filter := &fileadapter.Filter{
    P: []string{"", ""}, // load all 'p' records
    G: []string{"current_user", ""}, // load the 'g' records only for the current user.
}
enforcer.LoadFilteredPolicy(filter)

// The loaded policy now only contains the entries pertaining to "domain1".

But here's a question that your RBAC hierarchy cannot be multi-level. For example, let's say alice has role ceo, ceo has the role executives. The above code only loads: g, alice, ceo, but will not load: g, ceo, executives, as the filter only works literally. But the current filtered adapter only supports .CSV file. So if you use DB, e.g. MySQL, you need to implement the filtering feature on the current non-filtering adapter, e.g., Xorm Adapter.

However, if you need to use multi-level RBAC hierarchy, either you must preload all second level and above g records like the above g, ceo, executives. I think their size will not be too large. And then load the first-level g records (directly relates to user) like g, alice, ceo at run-time.

A more flexible way is to implement your own Role Manager, in this way, you can control all the details for your scenario: https://github.com/casbin/casbin#role-manager

---

The conclusion is:

1. If you use one-level RBAC, if you use .CSV file, use the existing Filtered File Adapter
2. If you use one-level RBAC, if you use DB, implement your own filtering-supported adapter based on existing adapters.
3. If you use multiple-level RBAC, implement your own filtering-supported adapter based on existing adapters, customize it to preload all high level (second level and above) g records, but load first-level g records at run-time.
4. If you use multiple-level RBAC, and you want to be more flexible, implement your own Role Manager and do whatever you want.

[rts-gordon]

Thanks for your excellent reply. I will try Role Manager.

Would you plan to split 'g' records from 'casbin_rule' to another table?

[hsluoyz]

Casbin loads all g records from adapter to role manager, so if you decide to use role manager, you will not need g records any more. There are only p policy rules in your adapter. And your role manager store your customized role information.

[rts-gordon]

OK, thank your help very much.

[rts-gordon]

I am writing a Role Manager which loading roles from database.
With my scenario before told, I will load all p records when program startup, and load user-role records from database only when one user access the program with RESTful call, I don't want to load p records again and again with LoadPolicy(), so when will I call SetRoleManager() and LoadPolicy()?

[hsluoyz]

LoadPolicy() will load all p and g records from database adapter to memory, and call role manager's AddLink() function to add each g record. When evaluating a request, enforce() will call role manager's HasLink() function to determine role relationship.

As you don't want to use built-in g records, so you don't need to care about AddLink(). Just load the data in your own way in your role manager, and implement HasLink() in a suitable way, so Casbin can call it.

To your questions specifically, you can call SetRoleManager() just after enforcer is created. LoadPolicy() has nothing to do with your role manager. It's just related to your p records.

[vaibhavkhl]

@Chcp Hi, did you write role manager? Would like to see your implementation.

[rts-gordon]

@vaibhavkhl Hi, sorry I have not write the role manager, it is a bit complex and need more time. If I write it, I will merge to Casbin.

Thanks.
Gordon