v1.2.0-rc3 release candidate is out with optimizations for ListObjects

[jon-whit]

We’ve just released another release candidate v1.2.0-rc3 for the upcoming release of OpenFGA. This release includes the patch fixes for the security advisory CVE-2023-35933. If you're actively evaluating the v1.2.0 release candidate, please upgrade to v1.2.0-rc3 to avoid the vulnerability reported in that CVE.

This release candidate includes experimental support for an optimized ListObjects implementation that can improve performance by a very large degree for models involving intersection (and operator in the DSL) and/or exclusion (but not operator in the DSL). Enable this new experimental feature with the --experimentals optimized-list-objects flag or by setting the OPENFGA_EXPERIMENTALS=optimized-list-objects env variable.

To run this release:

docker pull openfga/openfga:v1.2.0-rc3
docker run -e OPENFGA_EXPERIMENTALS=optimized-list-objects -p 8080:8080 openfga/openfga:v1.2.0-rc3 run

We’d love the communities help in further testing out this release candidate and give us feedback on how it improves performance for your models and query patterns. Please drop any comments/questions in the discussion below or feel free to reach out over Discord.

[xXAvoraXx]

ListObject query takes 18 seconds.

[xXAvoraXx]

ListObject query takes 2 mins.

[aaguiarz]

Hi! How long did it take with the previous version?

[xXAvoraXx]

I have sent the necessary files by mail. After getting the error below, openfga crashes completely and does not respond to calls to it, despite the fact that openfga seems to be running.

backend msg = "The request was canceled due to the configured HttpClient.Timeout of 100 seconds elapsing."
openfga console msg = {"level":"warn","timestamp":1686781801.1285214,"msg":"list objects timeout with list object configuration timeout","build.version":"v1.2.0-rc1","build.commit":"87d05a1aa665cf41be85f92b96e14e4d910c0c6a","timeout duration":"16m40s"}

[aaguiarz]

Thanks, we'll be working on it and keep you posted.

[xXAvoraXx]

Thank you, I would appreciate if you give priority to this issue. I'm getting requests about the user authorization part, I removed the relevant page from my users because openfga crashes my application.

[aaguiarz]

Hi! It's unlikely we'll have a fix in the next 5 dqys so you might need to go back to the latest released version instead of using the RC.

[xXAvoraXx]

I reverted to the latest version and now this is the result.

[shivaKhat23]

Hi, I am testing this model

model
  schema 1.1
type user
type account
  relations
    define can_assign_viewer: owner
    define can_view: owner or viewer
    define owner: [user]
    define user_in_context: [user]
    define viewer: [user]
type farm
  relations
    define user_in_context: user_in_context from parent
    define can_assign_viewer: owner or owner from parent
    define can_edit: owner
    define can_view: viewer and user_in_context
    define owner: [user] or owner from parent
    define parent: [account]
    define viewer: [user] or viewer from parent or owner

I am trying to filter farms in specific accounts using contextual tuples. It used to work but was slower before due to use of and here define can_view: viewer and user_in_context
I was expecting this to be faster because of the optimizations for ListObjects.
However, in this version, it does not work anymore and returns an empty list back. Also, once I make this request, OpenFGA stops working. Normal ListObjects request starts returning an empty list. and /stores results in deadline_exceeded error.

curl --location 'http://localhost:8080/stores/01H2WHD60GGVZ45BC5GT18ZSE9/list-objects' \
--header 'Content-Type: text/plain' \
--data '{
    "user": "user:a379e527-a975-4d19-b798-e3b9a0e67f69",
    "relation": "can_view",
    "type": "farm",
    "authorization_model_id": "01H2WHD612NPYWWQK7XQZB5MCS",
    "contextual_tuples": {
        "tuple_keys": [
            {
                "user": "user:a379e527-a975-4d19-b798-e3b9a0e67f69",
                "relation": "user_in_context",
                "object": "account:871b335a-769a-408b-b12f-8cb2a9139d63"
            }
        ]
    }
}
        '

This is the log

{"level":"warn","timestamp":1686732534.9589355,"msg":"list objects timeout with list object configuration timeout","build.version":"v1.2.0-rc1","build.commit":"87d05a1aa665cf41be85f92b96e14e4d910c0c6a","timeout duration":"3s"}

{"level":"error","timestamp":1686732545.3565893,"msg":"rpc error: code = Code(4000) desc = Internal Server Error","build.version":"v1.2.0-rc1","build.commit":"87d05a1aa665cf41be85f92b96e14e4d910c0c6a","grpc_service":"openfga.v1.OpenFGAService","grpc_method":"ListStores","grpc_type":"unary","request_id":"8da50d26-b9c1-45e7-854f-2679e1de7e8c","raw_request":{"page_size":null,"continuation_token":""},"raw_response":{"stores":[],"continuation_token":""},"peer.address":"127.0.0.1:41506","grpc_code":4000,"internal_error":"sql error: context deadline exceeded"}

[rhamzeh]

Hi @shivaKhat23 , just to verify. Have you set the --experimentals optimized-list-objects flag?

You should be seeing something like this line in your logs

🧪 experimental features enabled: [optimized-list-objects]

[shivaKhat23]

@rhamzeh yes, it is enabled.
I have this in my docker-compose

    image: openfga/openfga:v1.2.0-rc1
    container_name: openfga
    environment:
      - OPENFGA_DATASTORE_ENGINE=postgres
      - OPENFGA_DATASTORE_URI=postgres://postgres:password@postgres:5432/postgres?sslmode=disable
      - OPENFGA_LOG_FORMAT=json
      - OPENFGA_EXPERIMENTALS=optimized-list-objects

And, yes the log specifying that it was enabled was also present.

[shivaKhat23]

I tested again with openfga/openfga:v1.2.0-rc2. It works now for me.
With 1000 accounts for the above model, it used to take around 3-4 sec, now it has dropped to 600ms.
So, there is definitely performance improvement. 👍

[rhamzeh]

Thanks for the update @shivaKhat23!

[jon-whit]

We've released a new release candidate (v1.2.0-rc2) that includes some fixes to a few community-reported issues with the ListObjects changes for the upcoming v1.2.0 release.

For those of you testing out the release candidate, please pull it down and test it and let us know how it works for you 👍

docker pull openfga/openfga:v1.2.0-rc2
https://github.com/openfga/openfga/releases/tag/v1.2.0-rc2

[xXAvoraXx]

the crashing issue seems to be fixed but still the listobjects result is returning too late so I'm using redis.

[jon-whit]

@xXAvoraXx your model and the tuples in your system, unfortunately, do not benefit that significantly from the optimization included in this release candidate. The optimization benefits you a little bit, but marginally.

For the query and database dump you shared with us ListObjects(permission_access#permission_business@business:<id>) , there aren’t a lot of permission_access objects in the system in the first place, and all of the permission_access#permission_business relationships are contained under intersection (define permission_business: permission_from_plan and permission_from_resource). So we end up doing Check evaluations on all of those relationships anyways, and this is why the performance is on par with the prior implementation. The performance is better, but marginally for your specific tuples and model.

You should see an improvement to p95 and p99 latency for your ListObjects calls, even if it's marginal. If you find otherwise, we'd like to investigate with you. Please let us know if you find that your p95 and/or p99 latency has degraded with the optimized ListObjects feature in this release candidate.

[xXAvoraXx]

This is the query that the ListObjects endpoint gives the longest answer. After users log in, I fetch a list of what privileges they have in the system and this query takes a long time to respond.

            var permissions = await _fgaService.ListObjects(new ClientListObjectsRequest()
            {
                User = "user:" + userId,
                Relation = "can_access",
                Type = "permission_access",
                ContextualTuples = new List<ClientTupleKey>()
                                    {
                                        new ClientTupleKey()
                                        {
                                            User = "user:" + userId,
                                            Relation = "member_in_context",
                                            Object = "integrity:" + jti
                                        },
                                        new ClientTupleKey()
                                        {
                                            User = "user:" + userId,
                                            Relation = "user_in_context",
                                            Object = "business:" + claimBusinessId
                                        }
                                    }


            });

[jon-whit]

We've also just released a third release candidate v1.2.0-rc3 for the upcoming optimizations in the ListObjects API. This release includes the patch fixes for the security advisory CVE-2023-35933. If you're actively evaluating the v1.2.0 release candidate, please upgrade to v1.2.0-rc3 to avoid the vulnerability reported in that CVE.

docker pull openfga/openfga:v1.2.0-rc3

[xXAvoraXx]

Listobjects is currently receiving a response for the following relationship in seconds;

define can_access: can_access_member_and_individual_or_role or can_owner_access

can_access_member_and_individual_or_role: 23s
can_owner_access: 13s

Currently, these improvements have provided a serious performance increase.