Investigate seed script and propose fixes

[pierreTklein]

Describe the bug

We believe that the seed script causes some hackers to lose existing auth bindings. We should root-cause this and then fix the script.

To Reproduce
Steps to reproduce the behavior:
@krubenok can you successfully reproduce this?

Expected behavior

Nobody loses role-bindings after running npm run seed.

Additional context

@krubenok addtl context needed here?

[krubenok]

There's a bunch of context in the slack. I'm kinda foggy on the details though. Lemme go spelunking.

[pierreTklein]

Investigation

Terms to refresh our memory

0. Routes: an object which completely defines a permission for a user:

    post: {
        requestType: Constants.REQUEST_TYPES.POST,
        uri: "/api/hacker/"
    },

For URIs that contain a param which references some entity in the database, we define the permission as having access to ALL mongo documents in a given collection, or SELF document (the SELF document is a document that can be tied back to the requesting user):

    postAnyResumeById: {
        requestType: Constants.REQUEST_TYPES.POST,
        uri: "/api/hacker/resume/" + Constants.ROLE_CATEGORIES.ALL
    },

These routes are grouped by the collection it references (the example above would be grouped with other routes that reference the hacker collection).

1. single roles: roles that give permissions for one specific URI and request type:

let role = {
    _id: mongoose.Types.ObjectId(i),
    // routeKey is the name of the route, and routeGroupKey is the name of the collection
    name: routeKey + routeGroupKey,
    // the object mentioned in bullet 0
    routes: routeGroup[routeKey]
};

2. user roles: roles that give permissions for a collection of URIs and request types:

const hackerRole = {
    _id: mongoose.Types.ObjectId.createFromTime(2),
    name: Constants.General.HACKER,
    routes: [
        Constants.Routes.hackerRoutes.post,
        Constants.Routes.hackerRoutes.getSelfById
        ...
    ]
}

3. ObjectId: the default unique identifier for documents in mongoDB. The identifier is represented by 12 hex digits: "507f1f77bcf86cd799439011". According to the mongoDB documentation, this hex string is build by:
   (a) a 4-byte timestamp value, representing the ObjectId’s creation, measured in seconds since the Unix epoch
   (b) a 5-byte random value
   (c) a 3-byte incrementing counter, initialized to a random value

4. mongoose.Types.ObjectId(X) where X is an integer: Creates an ID with timestamp equal to X, with a 5-byte random value and 3-byte incrementing counter. Calling this function three times with X=1 would output the following:

mongoose.Types.ObjectId(1).toString();
// output: 000000017770e00000d7b8e3
mongoose.Types.ObjectId(1).toString();
// output: 000000017770e00000d7b8e4
mongoose.Types.ObjectId(1).toString();
// output: 000000017770e00000d7b8e5

where 00000001 is X, 7770e00000 is the random value, and d7b8e{3..5} is the incrementing counter. This function is not deterministic.

5. mongoose.Types.ObjectId(X) where X is a hex-string: Creates an ID with the hex-value equal to X. This function is deterministic:

mongoose.Types.ObjectId("000000017770e00000d7b8e3").toString();
// output: 000000017770e00000d7b8e3

6. mongoose.Types.createFromTime(X) where X is an integer: Creates an ID with timestamp equal to X, and the random value as well as incrementing counter are both zero. This function is deterministic.

mongoose.Types.createFromTime(1).toString();
// output: 000000010000000000000000

Root Cause

If I recall correctly, this bug appears only for "single roles" (roles that are for one specific URI and request type), not for user-roles (account role, etc.).

• All user-roles, with the exception of AccountRole, have their ID generated using mongoose.Types.createFromTime(X) (the function defined in bullet 6). Therefore, whenever we run npm run seed, the ID will always be the same.

• All single-roles are generated programmatically from the routes array. The code is basically:

function generateSingleRoles() {
    var i = 100000;
    const singleRoles = [];
    for (const routeGroup in AllRouteGroups) {
        for (const route in routeGroup) {
                let role = {
                    _id: mongoose.Types.ObjectId(i),
                    name: routeKey + routeGroupKey,
                    routes: route
                };
                let roleName = role.name;
                singleRoles[roleName] = role;
                i -= 1;
        }
    }
}

In short, we start with an integer i=100,000, and then generate ObjectIds using mongoose.Types.ObjectId(X) (bullet 4). Recall that this function is NOT DETERMINISTIC! Therefore, two will generate completely different ObjectId lists.

This is the root cause!

Solution

There are several solutions, some of which are better than others:

1. Map routeKey + routeGroupKey to a 12-byte hex value, and use that value as the id. The pro is that we don't need to pollute the route objects at all with unnecessary data (such as _ids). We don't have to care about ordering of the route objects either. The big con here is with hash-key collision: we need to make sure that the hash(X)` will generate a unique 12-byte hex value for every value of X in a deterministic way. If there is no bound on what X can be, then this is impossible, since there is a finite number of 12-byte hex values.
2. Change the structure of the routes so that they are stored in an array. Their index is used to compute their 12-byte hex value.
3. Add _id to every route object. This value is generated using createFromTime, so it is deterministic. Use this value as the _id for a single-role.

I propose we do (3).

[logan-r]

Woah, this is great!
I agree that (3) is the best option, think it's fair to say _id isn't polluting if having the primary key fixes this. (1) has a major downside and (2) seems a little inelegant

[krubenok]

If we implement (3) I recommend that we deploy to production and do the one-time database pain sooner rather than waiting until October so that it's fresh in our minds as we do this.