Slow bootstrap for many users and roles in security DB

[ezbc]

Description

Our project is a multi-tenant application with extremely granular security leading to thousands of users and roles in our security database. Bootstrap requests are timing out for our environment with a few thousand roles in the security DB.

To reproduce the problem:

1. Create many users and roles in the default security DB. See the Xquery script below to create users and roles.
2. Bootstrap a project with a role configured to be created.

This should lead to a slow execution of the setup:create-roles step in deploy/lib/xquery/setup.xqy.

This is because for each role to be created during the bootstrap, setup:get-roles is called to check if the role-to-be-created exists already. set:get-roles retrieves every role and transforms the elements, an expensive operation for many roles.

A solution I've devised is to retrieve only the role names outside of the role creation FLWOR statement. I will submit a PR soon.

Tech Specs

Windows 10 or Redhat 7.

MarkLogic version 8.0-6

Roxy version 1.7.5-dev

The RFE

Our project currently has enough users and roles where the target server is timing out the bootstrap request. We need to be able to deploy configuration changes to our MarkLogic server.

Script to create users and roles

Run against default security DB.

xquery version "1.0-ml";
import module namespace sec="http://marklogic.com/xdmp/security" at 
    "/MarkLogic/security.xqy";

for $i in 1 to 1000
return
 (
    try {
     xdmp:invoke-function( function() {
       sec:create-role(
        "roxy-test-role-" || $i,
        "load test role",
        ("filesystem-access"),
        (),
        ("testDocument"))
           }, <options xmlns="xdmp:eval">
           <isolation>different-transaction</isolation>
           <transaction-mode>update-auto-commit</transaction-mode>
          </options>
     )
    } catch($e) { ( "ERROR creating user" ) }
   ,
    try {
     xdmp:invoke-function( function() {

         sec:create-user(
          "roxy-test-user-" || $i,
          "load test user",
          "password",
          "roxy-test-role-" || $i,
          (xdmp:permission("security", "read")),
          ("roxy-test-user"))
      }, <options xmlns="xdmp:eval">
          <isolation>different-transaction</isolation>
           <transaction-mode>update-auto-commit</transaction-mode>
        </options>
     )
    } catch($e) { "ERROR creating user"}
 )

[dmcassel]

@ezbc, your proposed solution sounds good. We'll look forward to the PR.

[RobertSzkutak]

@ezbc Your fix sounds good. Offhand, in your code, I'd recommend getting rid of the try/catch statements unless there's a reason you'd want to continue in spite of an error. They will slow your code down a bit. If you want to keep them, you might as well throw both invokes into the same try/catch which will be a bit more efficient.

[ezbc]

@RobertSzkutak thanks for the tip!

[dmcassel]

fixed in dev. @ezbc thanks for the PR!

[ezbc]

@dmcassel thanks for the quick review!