Skip to content

This issue was moved to a discussion.

You can continue the conversation there. Go to discussion →

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ListObjects fails to retrieve all items #104

Closed
muntdan opened this issue Dec 23, 2022 · 8 comments
Closed

ListObjects fails to retrieve all items #104

muntdan opened this issue Dec 23, 2022 · 8 comments
Assignees
Labels
bug Something isn't working

Comments

@muntdan
Copy link

muntdan commented Dec 23, 2022

Although assert validates a read user from AdminTeam that it has read right to a certain resource,
the ListObjects method does not return all items that it actually has read access, and some times it returns different results on back to back calls.

Authorization Model (Schema v1.1)

DSL

model
  schema 1.1
type user
type organization
  relations
    define admin_team: [super_team]
    define has_delete: owner_user or has_delete from admin_team
    define has_read: has_write or read_user or has_read from admin_team
    define has_write: owner_user or write_user or has_write from admin_team
    define owner_user: [user]
    define read_user: [user]
    define write_user: [user]
type organization_resource
  relations
    define has_delete: has_delete from parent_org
    define has_read: has_write or has_read from parent_org
    define has_write: has_write from parent_org
    define parent_org: [organization]
type super_team
  relations
    define has_delete: manager_user
    define has_read: has_write or read_user
    define has_write: manager_user or write_user
    define manager_user: [user]
    define read_user: [user]
    define write_user: [user]

JSON:

{
  "schema_version": "1.1",
  "type_definitions": [{
    "type": "organization",
    "relations": {
      "admin_team": {
        "this": {}
      },
      "has_delete": {
        "union": {
          "child": [{
            "computedUserset": {
              "object": "",
              "relation": "owner_user"
            }
          }, {
            "tupleToUserset": {
              "tupleset": {
                "object": "",
                "relation": "admin_team"
              },
              "computedUserset": {
                "object": "",
                "relation": "has_delete"
              }
            }
          }]
        }
      },
      "has_read": {
        "union": {
          "child": [{
            "computedUserset": {
              "object": "",
              "relation": "has_write"
            }
          }, {
            "computedUserset": {
              "object": "",
              "relation": "read_user"
            }
          }, {
            "tupleToUserset": {
              "tupleset": {
                "object": "",
                "relation": "admin_team"
              },
              "computedUserset": {
                "object": "",
                "relation": "has_read"
              }
            }
          }]
        }
      },
      "has_write": {
        "union": {
          "child": [{
            "computedUserset": {
              "object": "",
              "relation": "owner_user"
            }
          }, {
            "computedUserset": {
              "object": "",
              "relation": "write_user"
            }
          }, {
            "tupleToUserset": {
              "tupleset": {
                "object": "",
                "relation": "admin_team"
              },
              "computedUserset": {
                "object": "",
                "relation": "has_write"
              }
            }
          }]
        }
      },
      "owner_user": {
        "this": {}
      },
      "read_user": {
        "this": {}
      },
      "write_user": {
        "this": {}
      }
    },
    "metadata": {
      "relations": {
        "admin_team": {
          "directly_related_user_types": [{
            "type": "super_team"
          }]
        },
        "has_delete": {
          "directly_related_user_types": []
        },
        "has_read": {
          "directly_related_user_types": []
        },
        "has_write": {
          "directly_related_user_types": []
        },
        "owner_user": {
          "directly_related_user_types": [{
            "type": "user"
          }]
        },
        "read_user": {
          "directly_related_user_types": [{
            "type": "user"
          }]
        },
        "write_user": {
          "directly_related_user_types": [{
            "type": "user"
          }]
        }
      }
    }
  }, {
    "type": "organization_resource",
    "relations": {
      "has_delete": {
        "tupleToUserset": {
          "tupleset": {
            "object": "",
            "relation": "parent_org"
          },
          "computedUserset": {
            "object": "",
            "relation": "has_delete"
          }
        }
      },
      "has_read": {
        "union": {
          "child": [{
            "computedUserset": {
              "object": "",
              "relation": "has_write"
            }
          }, {
            "tupleToUserset": {
              "tupleset": {
                "object": "",
                "relation": "parent_org"
              },
              "computedUserset": {
                "object": "",
                "relation": "has_read"
              }
            }
          }]
        }
      },
      "has_write": {
        "tupleToUserset": {
          "tupleset": {
            "object": "",
            "relation": "parent_org"
          },
          "computedUserset": {
            "object": "",
            "relation": "has_write"
          }
        }
      },
      "parent_org": {
        "this": {}
      }
    },
    "metadata": {
      "relations": {
        "has_delete": {
          "directly_related_user_types": []
        },
        "has_read": {
          "directly_related_user_types": []
        },
        "has_write": {
          "directly_related_user_types": []
        },
        "parent_org": {
          "directly_related_user_types": [{
            "type": "organization"
          }]
        }
      }
    }
  }, {
    "type": "super_team",
    "relations": {
      "has_delete": {
        "computedUserset": {
          "object": "",
          "relation": "manager_user"
        }
      },
      "has_read": {
        "union": {
          "child": [{
            "computedUserset": {
              "object": "",
              "relation": "has_write"
            }
          }, {
            "computedUserset": {
              "object": "",
              "relation": "read_user"
            }
          }]
        }
      },
      "has_write": {
        "union": {
          "child": [{
            "computedUserset": {
              "object": "",
              "relation": "manager_user"
            }
          }, {
            "computedUserset": {
              "object": "",
              "relation": "write_user"
            }
          }]
        }
      },
      "manager_user": {
        "this": {}
      },
      "read_user": {
        "this": {}
      },
      "write_user": {
        "this": {}
      }
    },
    "metadata": {
      "relations": {
        "has_delete": {
          "directly_related_user_types": []
        },
        "has_read": {
          "directly_related_user_types": []
        },
        "has_write": {
          "directly_related_user_types": []
        },
        "manager_user": {
          "directly_related_user_types": [{
            "type": "user"
          }]
        },
        "read_user": {
          "directly_related_user_types": [{
            "type": "user"
          }]
        },
        "write_user": {
          "directly_related_user_types": [{
            "type": "user"
          }]
        }
      }
    }
  }, {
    "type": "user",
    "relations": {}
  }]
}
@rhamzeh rhamzeh transferred this issue from openfga/api Dec 23, 2022
@rhamzeh rhamzeh added the bug Something isn't working label Dec 23, 2022
@rhamzeh
Copy link
Member

rhamzeh commented Dec 23, 2022

Thanks for raising this issue @danmunteanuevo!

Please note the ListObjects is considered very experimental and we note down some of the known issues with ListObjects here, so please take a look.

Just so that we enough data to know if this is a known issue or a new one can I ask you to share:

  • What version of OpenFGA are you running?
  • How are you running it (docker/docker-compose/binary/etc..)
  • Whether you are using any of these config values/flags, and if so, what are their values
    • listObjectsMaxResults/ Env: OPENFGA_LISTOBJECTSMAXRESULTS
    • listObjectsDeadline/ Env: OPENFGA_LISTOBJECTSDEADLINE
    • experimentals/ Env: OPENFGA_EXPERIMENTALS
  • How many relationship tuples do you have in your database?
  • How many objects do you expect to be returned in the response

@muntdan
Copy link
Author

muntdan commented Dec 23, 2022

OpenFGA Latest Docker from 4 days ago: [v0.3.1]
.net SDK
I did not use any of those custom configs.
38 Tuples.
I expect to have returned 6 and receive 5.

@muntdan
Copy link
Author

muntdan commented Dec 23, 2022

I have enabled on the docker the --experimentals="list-objects-optimized". The query are now faster for the other users/access rights and they work.
But now exactly for the same 2 users -> only for the has_read access right (which was missing items before) I now get some times not always Internal error from the api:
Exception when calling OpenFgaApi.Check: One or more errors occurred. (Internal Server Error) Status Code: OpenFga.Sdk.Exceptions.FgaApiInternalError: Internal Server Error at OpenFga.Sdk.Client.BaseClient.SendRequestAsync[T](HttpRequestMessage request, IDictionary2 additionalHeaders, String apiName, CancellationToken cancellationToken)
at OpenFga.Sdk.Client.BaseClient.SendRequestAsync[T](RequestBuilder requestBuilder, IDictionary2 additionalHeaders, String apiName, CancellationToken cancellationToken) at OpenFga.Sdk.Client.ApiClient.<>c__DisplayClass4_01.<b__0>d.MoveNext()
--- End of stack trace from previous location ---
at OpenFga.Sdk.Client.ApiClient.Retry[TResult](Func1 retryable) at OpenFga.Sdk.Client.ApiClient.SendRequestAsync[T](RequestBuilder requestBuilder, String apiName, CancellationToken cancellationToken) at OpenFga.Sdk.Api.OpenFgaApi.ListObjects(ListObjectsRequest body, CancellationToken cancellationToken)

@jon-whit jon-whit self-assigned this Jan 5, 2023
@robinmanuelthiel
Copy link

It it planned to ever support returning all items a user has a relationship with? So that I can use ListObjects to get a full list of results, even if there are more items than defined in listObjectsMaxResults (maybe supported by pagination)?

@jon-whit
Copy link
Member

jon-whit commented Jan 18, 2023

It it planned to ever support returning all items a user has a relationship with? So that I can use ListObjects to get a full list of results, even if there are more items than defined in listObjectsMaxResults (maybe supported by pagination)?

@robinmanuelthiel it already is supported, but not well documented. If you provide a zero value for that config then it won't prematurely terminate. You can also use the StreamedListObjects API for streaming semantics, which may work better for your use case.

The danger with disabling (with zero values) the listObjectsMaxResults and listObjectsDeadline configurations, and the reason those exist in the first place, is that it's possible that operation could hang for a long time. Those are meant as safeguards for the server so that a client request can't overload the server with a ListObjects request that yields an insanely large number of objects.

Unfortunately, given the nature of the query resolution (it's a recursive breadth first search of a relationship graph), it's hard to paginate this dataset because it's not a fixed dataset and the graph evaluation is done concurrently (and thus is not ordered). The query is implemented as a dynamic graph traversal over relational data. This makes it particularly challenging to paginate, and hence the reason why we've chosen these alternative tradeoffs.

I recommend giving the StreamedListObjects API a try and see how that works for you. Disable the listObjectsDeadline by setting it to a zero value and then see if that works well for ya. If you need some assistance don't hesitate to reach out over Discord 👍

@robinmanuelthiel
Copy link

robinmanuelthiel commented Jan 19, 2023

@jon-whit Thanks a lot for the detailed answer. Will give it a try and report back! Did I understand correctly, that streaming the results still requires a lot of calculation on the server and still gets throttled or limited by the listObjectsMaxResults and listObjectsDeadline variables? So streaming just gives me first results faster but internally uses the same logic, correct?

@jon-whit
Copy link
Member

jon-whit commented Jan 20, 2023

Did I understand correctly, that streaming the results still requires a lot of calculation on the server and still gets throttled or limited by the listObjectsMaxResults and listObjectsDeadline variables? So streaming just gives me first results faster but internally uses the same logic, correct?

@robinmanuelthiel That is correct, yes. That's a perfect way to describe it. It's a more responsive way to receive the result stream, whereas the alternative /list-objects or ListObjects API waits for the results (up to the max) before responding. So if, for example, you are building a UI that only needs the first 10 results, then the streaming variant may be more responsive for you.

@robinmanuelthiel
Copy link

@jon-whit Thanks for clarifying. Are there any plans to ever support pagination or something similar? My use-case is search, so I need to get all items of type Foo a user has access to and then filter my search results to only include these.

As I am building a multi-tenant system, I can't afford that one search affects the permissions-performance for all other tenants, I am looking for a solution that gives me a paginated list of all items a user can access without running into memory/time issues on the permissions server :/

@matthewpereira matthewpereira transferred this issue from openfga/openfga Feb 6, 2023
@openfga openfga locked and limited conversation to collaborators Feb 6, 2023
@rhamzeh rhamzeh converted this issue into discussion #105 Feb 6, 2023

This issue was moved to a discussion.

You can continue the conversation there. Go to discussion →

Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

4 participants