SDK Quota Support

[kaiwalyajoshi]

This feature requires support for enforceRole on Marathon groups found in Marathon v1.9.73 and Mesos v1.9.0 available starting DC/OS 2.0.
By default Marathon does not set enforceRole=true on group creation, and existing semantics are maintained.

Deploy new service in a group with quota enabled

Hello-World is used in the example below but this is applicable to any SDK based service.
To create a service named /dev/hello-world in group dev with quota consumed from role dev

1. Create a group with enforceRole:

cat >create-group.json<<EOF
{
    "id":"/dev",
    "enforceRole":true
}
EOF

2. Create Marathon group:

dcos marathon group add create-group.json

3. Populate the service options:

cat >hello-world-dev-options.json<<EOF
{
    "service":{
        "name":"/dev/hello-world"
    }
}
EOF

4. Install service.

dcos package install hello-world --options=namespace-options-foo-enforce-role.json --yes

5. Ensure SDK scheduler and pods have been launched under the dev role via Mesos UI.

Migrate an existing deployed service to use quota support

To upgrade an existing service to a new version of the SDK with quota support, use the following procedure.
We will use Hello-World again pre-installed in group foo in the example below but this is applicable to any SDK based service.

1. Create a file with the current service-name and the following additional options:

cat >hello-world-foo-options.json<<EOF
{
    "service":{
        "name":"/foo/hello-world",
        "role": "foo",
        "enable_role_migration": true
    }
}
EOF

• role: Specifies the quota enforced role we're migrating towards, which is foo in this example.
• enable_role_migration: Notifies the scheduler that its pods will be migrated between legacy and quota enforced roles. The scheduler subscribes with both roles when this flag is set.

2. Update the scheduler to use the quota enforced role.

dcos hello-world --name="/foo/hello-world" update start --options=hello-world-foo-options.json.json

3. At this point the scheduler will be upgraded and will use quota from the foo role. The deployed pods will be unaffected and will use their previous roles.
4. Issue pod replace commands to migrate all the pods in the service to the quota enforced role.

dcos hello-world --name="/foo/hello-world" pod replace hello-0

The hello-0 pod will be migrated to consume quota from foo
5. Create a file with the current service-name and the following options to signal the end of the migration:

cat >hello-world-foo-disable-migration.json<<EOF
{
    "service":{
        "name":"/foo/hello-world",
        "role": "foo",
        "enable_role_migration": false
    }
}
EOF

6. Update the scheduler to stop subscribing to the legacy role.

dcos hello-world --name="/foo/hello-world" update start --options=hello-world-foo-disable-migration.json.json

At this point, the scheduler and all the previous running pods have been migrated to the quota enforced role.

Strict Mode Clusters

For strict mode clusters, additional role permissions are required and must be setup before deploying the service.

1. New service in a group with enforceRole=true
   Example:

• New service with name /dev/hello-world will need permissions to the dev role

dcos security org users grant <service-account> dcos:mesos:master:reservation:role:dev create

2. Migrating an existing service to a quota enforced role.
   Example:

• Existing service with name /foo/hello-world will need permissions to both the foo and foo__hello-world-role roles

dcos security org users grant <service-account> dcos:mesos:master:reservation:role:foo create
dcos security org users grant <service-account> dcos:mesos:master:reservation:role:foo__hello-world-role create

Pod Pre-Reserved Roles

For pods which specify pre-reserved roles (eg slave_public), the scheduler will issue a hierarhical role depending on the value of role.

Example:

• Pod Pre-Reserved Role: slave_public and role=slave_public. These permissions are required:

dcos security org users grant <service-account> dcos:mesos:master:reservation:role:slave_public/dev__hello-world-role create

• Pod Pre-Reserved Role: slave_public and role=dev. These permissions are required:

dcos security org users grant <service-account> dcos:mesos:master:reservation:role:slave_public/dev create

When performing migration between legacy to enforced group roles via enable_role_migration, both permissions above will be required.

Downgrading to and older non-quota aware version of the scheduler

This section details the procedures to downgrade from a quota enforced role to a shipped non-quota enforced release.
The process is the same as migrating an existing service to the quoted role
The key difference is that role should be slave_public to indicate migration towards the legacy roles.

cat >hello-world-foo-downgrade.json<<EOF
{
    "service":{
        "name":"/foo/hello-world",
        "role": "slave_public",
        "enable_role_migration": true
    }
}
EOF

The remaining scheduler update and pod-replace operations must be issued to move the scheduler and pods into the legacy roles.
Once all the pods have been migrated, the scervice can be downgraded to an earlier release which isn't quota aware.

JIRA: DCOS-54278

[takirala]

Had couple queries, good to merge after those are answered. 🚢

Had some ideas on code cleanup in test_quota_deployment and the usage of null in SchedulerConfig. I created
suggestions.diff.txt a diff file with some suggestions. Nothing blocking though.

[takirala]

I haven't had a closer look, but would this affect the minimum number of nodes on the cluster (due to placement constraints ) - and if yes, is this within that limit ?

[kaiwalyajoshi]

Both hello and world pods have the placement constraint "[[\"hostname\", \"UNIQUE\"]]", in our SI we spin-up a cluster with 5 agents, so this fits into the normal test configuration.

[takirala]

we can delete this line

[takirala]

i think we have to add this to test_requirements.txt as well ?!

[kaiwalyajoshi]

Don't think so, pytest harness gets all of its deps from frozen_requirements.txt

[takirala]

So if this is a multi role, what would be the result of service_state["role"] - is it a non existent key OR defaulted to an empty list ?

[kaiwalyajoshi]

service_state["role"] = "*"