Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Flat queries #487

Merged
merged 7 commits into from
Feb 4, 2021
Merged

Flat queries #487

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
5de5340
create and populate effective policies
dvoet Nov 23, 2020
205f694
cleanup
dvoet Jan 19, 2021
a6d2b05
added test
dvoet Jan 25, 2021
f074891
review feedback
dvoet Jan 30, 2021
9d7a53e
test migration sql
dvoet Feb 2, 2021
3301868
a touch more feedback
dvoet Feb 2, 2021
c1b404e
consistent case
dvoet Feb 4, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
11 changes: 11 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -75,6 +75,17 @@ Group - Create, delete, read, list, add/remove users and groups. Nested groups a

Note that in this model Group is a Subject. This allows it to be used interchangeably with Users within policies.

### ERD
![Sam ERD](sam_erd.png)

The Sam schema has 3 main sections: users and groups in yellow, resources and policies in green, resource type
configuration in purple. Both groups and resources have a hierarchical model (groups can contain groups
and resources can have parents). To solve read-query performance issues these hierarchies are also stored in
a flattened representation:
[sam_group_member_flat](src/main/scala/org/broadinstitute/dsde/workbench/sam/dataAccess/PostgresGroupDAO.scala)
and
[sam_effective_*](src/main/scala/org/broadinstitute/dsde/workbench/sam/dataAccess/EffectivePolicyMutationStatements.scala) tables.

### API
[Sam APIs](http://petstore.swagger.io/?url=https://raw.githubusercontent.com/broadinstitute/sam/develop/src/main/resources/swagger/api-docs.yaml)

Expand Down
Binary file added sam_erd.png
View file
Open in desktop
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
2 changes: 2 additions & 0 deletions src/main/resources/org/broadinstitute/dsde/sam/liquibase/changelog.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,4 +14,6 @@
<include file="changesets/20201102_flattened_role_materialized_view.xml" relativeToChangelogFile="true"/>
<include file="changesets/20201117_policy_role_index.xml" relativeToChangelogFile="true"/>
<include file="changesets/20201103_add_google_project_child_for_each_billing_project.xml" relativeToChangelogFile="true"/>
<include file="changesets/20201123_effective_policy.xml" relativeToChangelogFile="true"/>
<include file="changesets/20201123_flat_group_membership.xml" relativeToChangelogFile="true"/>
</databaseChangeLog>
106 changes: 106 additions & 0 deletions .../resources/org/broadinstitute/dsde/sam/liquibase/changesets/20201123_effective_policy.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,106 @@
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<databaseChangeLog logicalFilePath="dummy"
xmlns="http://www.liquibase.org/xml/ns/dbchangelog"
xmlns:ext="http://www.liquibase.org/xml/ns/dbchangelog-ext"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog-ext http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-ext.xsd http://www.liquibase.org/xml/ns/dbchangelog http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-3.4.xsd">

<changeSet logicalFilePath="dummy" author="dvoet" id="effective_policy_table">
<createTable tableName="SAM_EFFECTIVE_RESOURCE_POLICY">
<column name="id" type="BIGINT" autoIncrement="true">
<constraints primaryKey="true"/>
</column>
<column name="resource_id" type="BIGINT">
<constraints nullable="false" foreignKeyName="FK_SERP_RESOURCE" referencedTableName="SAM_RESOURCE" referencedColumnNames="id"/>
</column>
<column name="source_policy_id" type="BIGINT">
gpolumbo-broad marked this conversation as resolved.
Show resolved Hide resolved
<constraints nullable="false" foreignKeyName="FK_SERP_SOURCE_POLICY" referencedTableName="SAM_RESOURCE_POLICY" referencedColumnNames="id" deleteCascade="true"/>
</column>
</createTable>
</changeSet>

<changeSet logicalFilePath="dummy" author="dvoet" id="index_erp_resource_id">
<createIndex tableName="SAM_EFFECTIVE_RESOURCE_POLICY" indexName="IDX_ERP_RESOURCE_ID">
<column name="resource_id"/>
</createIndex>
</changeSet>

<changeSet logicalFilePath="dummy" author="dvoet" id="index_rp_public">
<createIndex tableName="SAM_RESOURCE_POLICY" indexName="IDX_RP_PUBLIC">
<column name="public"/>
dvoet marked this conversation as resolved.
Show resolved Hide resolved
</createIndex>
</changeSet>

<changeSet logicalFilePath="dummy" author="dvoet" id="index_sr_parent_resource_id">
<createIndex tableName="SAM_RESOURCE" indexName="IDX_SR_PARENT_RESOURCE_ID">
<column name="resource_parent_id"/>
</createIndex>
</changeSet>

<changeSet logicalFilePath="dummy" author="dvoet" id="effective_policy_role_table">
<createTable tableName="SAM_EFFECTIVE_POLICY_ROLE">
<column name="effective_resource_policy_id" type="BIGINT">
<constraints nullable="false" foreignKeyName="FK_SEPR_POLICY" referencedTableName="SAM_EFFECTIVE_RESOURCE_POLICY" referencedColumnNames="id" primaryKey="true" deleteCascade="true"/>
</column>
<column name="resource_role_id" type="BIGINT">
<constraints nullable="false" foreignKeyName="FK_SEPR_ROLE" referencedTableName="SAM_RESOURCE_ROLE" referencedColumnNames="id" primaryKey="true"/>
</column>
</createTable>
</changeSet>

<changeSet logicalFilePath="dummy" author="dvoet" id="effective_policy_action_table">
<createTable tableName="SAM_EFFECTIVE_POLICY_ACTION">
<column name="effective_resource_policy_id" type="BIGINT">
<constraints nullable="false" foreignKeyName="FK_SEPA_POLICY" referencedTableName="SAM_EFFECTIVE_RESOURCE_POLICY" referencedColumnNames="id" primaryKey="true" deleteCascade="true"/>
</column>
<column name="resource_action_id" type="BIGINT">
<constraints nullable="false" foreignKeyName="FK_SEPA_ACTION" referencedTableName="SAM_RESOURCE_ACTION" referencedColumnNames="id" primaryKey="true"/>
</column>
</createTable>
</changeSet>

<changeSet logicalFilePath="dummy" author="dvoet" id="populate_effective_policies" runInTransaction="true">
<sql stripComments="true">
with recursive
ancestor_resource(resource_id, base_resource_id) as (
s-rubenstein marked this conversation as resolved.
Show resolved Hide resolved
select resource.id, resource.id
from SAM_RESOURCE resource
join SAM_RESOURCE_TYPE resourceType on resource.resource_type_id = resourceType.id
union
select parentResource.resource_parent_id, ancestorResource.base_resource_id
from SAM_RESOURCE parentResource
join ancestor_resource ancestorResource on ancestorResource.resource_id = parentResource.id where parentResource.resource_parent_id is not null
)

insert into SAM_EFFECTIVE_RESOURCE_POLICY(resource_id, source_policy_id)
select ar.base_resource_id, policy.id
from SAM_RESOURCE_POLICY policy join ancestor_resource ar on policy.resource_id = ar.resource_id
</sql>
<sql stripComments="true">
insert into SAM_EFFECTIVE_POLICY_ROLE(effective_resource_policy_id, resource_role_id)
select ep.id, role.id
from SAM_EFFECTIVE_RESOURCE_POLICY ep
join SAM_POLICY_ROLE pr on ep.source_policy_id = pr.resource_policy_id
join SAM_RESOURCE resource on ep.resource_id = resource.id
join sam_flattened_role flat_role on pr.resource_role_id = flat_role.base_role_id
dvoet marked this conversation as resolved.
Show resolved Hide resolved
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: Out of curiosity, should sam_flattened_role be in all caps here?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

doesn't matter

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If it doesn't matter, would you mind moving it to caps for consistency?

join SAM_RESOURCE_ROLE role on flat_role.nested_role_id = role.id and resource.resource_type_id = role.resource_type_id
join sam_resource_policy sourcePolicy on ep.source_policy_id = sourcePolicy.id
where (((sourcePolicy.resource_id != ep.resource_id) and (pr.descendants_only or flat_role.descendants_only))
or not ((sourcePolicy.resource_id != ep.resource_id) or pr.descendants_only or flat_role.descendants_only))
on conflict do nothing
</sql>
<sql stripComments="true">
insert into SAM_EFFECTIVE_POLICY_ACTION(effective_resource_policy_id, resource_action_id)
select ep.id, action.id
from SAM_EFFECTIVE_RESOURCE_POLICY ep
join SAM_POLICY_ACTION pa on ep.source_policy_id = pa.resource_policy_id
join SAM_RESOURCE resource on ep.resource_id = resource.id
join SAM_RESOURCE_ACTION action on pa.resource_action_id = action.id and resource.resource_type_id = action.resource_type_id
join sam_resource_policy sourcePolicy on ep.source_policy_id = sourcePolicy.id
where pa.descendants_only = (sourcePolicy.resource_id != ep.resource_id)
and resource.resource_type_id = action.resource_type_id
on conflict do nothing
</sql>
</changeSet>

</databaseChangeLog>
gpolumbo-broad marked this conversation as resolved.
Show resolved Hide resolved
67 changes: 67 additions & 0 deletions ...urces/org/broadinstitute/dsde/sam/liquibase/changesets/20201123_flat_group_membership.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,67 @@
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<databaseChangeLog logicalFilePath="dummy"
xmlns="http://www.liquibase.org/xml/ns/dbchangelog"
xmlns:ext="http://www.liquibase.org/xml/ns/dbchangelog-ext"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog-ext http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-ext.xsd http://www.liquibase.org/xml/ns/dbchangelog http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-3.4.xsd">
<changeSet logicalFilePath="dummy" author="thibault" id="flat_group_member_table">
<createTable tableName="SAM_GROUP_MEMBER_FLAT">
dvoet marked this conversation as resolved.
Show resolved Hide resolved
<column name="id" type="BIGINT" autoIncrement="true">
<constraints primaryKey="true"/>
</column>
<column name="group_id" type="BIGINT">
<constraints foreignKeyName="FK_SGM_GROUP" referencedTableName="SAM_GROUP" referencedColumnNames="id" nullable="false" deleteCascade="true"/>
</column>
<column name="member_group_id" type="BIGINT">
<constraints foreignKeyName="FK_SGM_MEMBER_GROUP" referencedTableName="SAM_GROUP" referencedColumnNames="id"/>
</column>
<column name="member_user_id" type="VARCHAR">
<constraints foreignKeyName="FK_SGM_MEMBER_USER" referencedTableName="SAM_USER" referencedColumnNames="id" deleteCascade="true"/>
</column>
<column name="group_membership_path" type="BIGINT ARRAY"/>
<column name="last_group_membership_element" type="BIGINT">
<constraints foreignKeyName="FK_SGM_LAST_GROUP_MEMBERSHIP_ELEMENT" referencedTableName="SAM_GROUP" referencedColumnNames="id" deleteCascade="true"/>
dvoet marked this conversation as resolved.
Show resolved Hide resolved
</column>
</createTable>
</changeSet>
<changeSet logicalFilePath="dummy" author="dvoet" id="index_gmf_member_user_id">
<createIndex tableName="SAM_GROUP_MEMBER_FLAT" indexName="IDX_GMF_MEMBER_USER_ID">
<column name="member_user_id"/>
</createIndex>
</changeSet>
<changeSet logicalFilePath="dummy" author="dvoet" id="index_gmf_member_group_id">
<createIndex tableName="SAM_GROUP_MEMBER_FLAT" indexName="IDX_GMF_MEMBER_GROUP_ID">
<column name="member_group_id"/>
</createIndex>
</changeSet>
<changeSet logicalFilePath="dummy" author="dvoet" id="index_gmf_group_membership_path">
<sql>
CREATE INDEX IDX_GMF_MEMBERSHIP_PATH ON SAM_GROUP_MEMBER_FLAT USING GIN (group_membership_path)
s-rubenstein marked this conversation as resolved.
Show resolved Hide resolved
</sql>
</changeSet>
<changeSet logicalFilePath="dummy" author="dvoet" id="index_gmf_group_id">
<createIndex tableName="SAM_GROUP_MEMBER_FLAT" indexName="IDX_GMF_GROUP_ID">
<column name="group_id"/>
</createIndex>
</changeSet>
<changeSet logicalFilePath="dummy" author="dvoet" id="index_gmf_last_group_membership_element">
<createIndex tableName="SAM_GROUP_MEMBER_FLAT" indexName="IDX_GMF_last_group_membership_element">
<column name="last_group_membership_element"/>
</createIndex>
</changeSet>
<changeSet logicalFilePath="dummy" author="dvoet" id="populate_flat_group_member_table">
<sql stripComments="true">
with recursive
unrolled_groups(group_id, member_group_id, member_user_id, group_membership_path, last_group_membership_element) as (
select gm.group_id, gm.member_group_id, gm.member_user_id, array[gm.group_id], gm.group_id
from sam_group_member gm
union
select ug.group_id, gm.member_group_id, gm.member_user_id, array_append(ug.group_membership_path, gm.group_id), gm.group_id
from unrolled_groups ug
join sam_group_member gm on ug.member_group_id = gm.group_id
)
insert into sam_group_member_flat(group_id, member_group_id, member_user_id, group_membership_path, last_group_membership_element)
select ug.group_id, ug.member_group_id, ug.member_user_id, ug.group_membership_path, ug.last_group_membership_element from unrolled_groups ug
</sql>
</changeSet>
</databaseChangeLog>
31 changes: 11 additions & 20 deletions src/main/scala/org/broadinstitute/dsde/workbench/sam/Boot.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,25 +18,23 @@ import javax.net.SocketFactory
import javax.net.ssl.SSLContext
import org.broadinstitute.dsde.workbench.dataaccess.PubSubNotificationDAO
import org.broadinstitute.dsde.workbench.google.GoogleCredentialModes.{Json, Pem}
import org.broadinstitute.dsde.workbench.google2.util.DistributedLock
import org.broadinstitute.dsde.workbench.google.{GoogleDirectoryDAO, GoogleKmsInterpreter, GoogleKmsService, HttpGoogleDirectoryDAO, HttpGoogleIamDAO, HttpGoogleProjectDAO, HttpGooglePubSubDAO, HttpGoogleStorageDAO}
import org.broadinstitute.dsde.workbench.google2.util.DistributedLock
import org.broadinstitute.dsde.workbench.google2.{GoogleFirestoreInterpreter, GoogleStorageInterpreter, GoogleStorageService}
import org.broadinstitute.dsde.workbench.model.{WorkbenchEmail, WorkbenchException}
import org.broadinstitute.dsde.workbench.sam.api.{SamRoutes, StandardUserInfoDirectives}
import org.broadinstitute.dsde.workbench.sam.config.{AppConfig, GoogleConfig, IdentityConcentratorConfig}
import org.broadinstitute.dsde.workbench.sam.dataAccess._
import org.broadinstitute.dsde.workbench.sam.db.DatabaseNames.DatabaseName
import org.broadinstitute.dsde.workbench.sam.db.{DatabaseNames, DbReference}
import org.broadinstitute.dsde.workbench.sam.directory._
import org.broadinstitute.dsde.workbench.sam.google._
import org.broadinstitute.dsde.workbench.sam.identityConcentrator.{IdentityConcentratorService, StandardIdentityConcentratorApi}
import org.broadinstitute.dsde.workbench.sam.model._
import org.broadinstitute.dsde.workbench.sam.openam._
import org.broadinstitute.dsde.workbench.sam.schema.JndiSchemaDAO
import org.broadinstitute.dsde.workbench.sam.service._
import org.broadinstitute.dsde.workbench.util.DelegatePool
import org.broadinstitute.dsde.workbench.util2.ExecutionContexts
import org.http4s.client.blaze.BlazeClientBuilder
import scalikejdbc.config.DBs

import scala.concurrent.ExecutionContext
import scala.concurrent.ExecutionContext.Implicits.global
Expand Down Expand Up @@ -114,11 +112,11 @@ object Boot extends IOApp with LazyLogging {
private[sam] def createAppDependencies(
appConfig: AppConfig)(implicit actorSystem: ActorSystem): cats.effect.Resource[IO, AppDependencies] =
for {
(foregroundDirectoryDAO, foregroundAccessPolicyDAO, _, registrationDAO) <- createDAOs(appConfig, DatabaseNames.Foreground, appConfig.directoryConfig.connectionPoolSize, "foreground")
(foregroundDirectoryDAO, foregroundAccessPolicyDAO, _, registrationDAO) <- createDAOs(appConfig, DatabaseNames.Write, DatabaseNames.Read, appConfig.directoryConfig.connectionPoolSize, "foreground")

// This special set of objects are for operations that happen in the background, i.e. not in the immediate service
// of an api call (foreground). They are meant to partition resources so that background processes can't crowd our api calls.
(backgroundDirectoryDAO, backgroundAccessPolicyDAO, backgroundLdapExecutionContext, _) <- createDAOs(appConfig, DatabaseNames.Background, appConfig.directoryConfig.backgroundConnectionPoolSize, "background")
(backgroundDirectoryDAO, backgroundAccessPolicyDAO, backgroundLdapExecutionContext, _) <- createDAOs(appConfig, DatabaseNames.Background, DatabaseNames.Background, appConfig.directoryConfig.backgroundConnectionPoolSize, "background")

blockingEc <- ExecutionContexts.fixedThreadPool[IO](24)

Expand Down Expand Up @@ -195,35 +193,28 @@ object Boot extends IOApp with LazyLogging {
}

private def createDAOs(appConfig: AppConfig,
dbName: DatabaseName,
writeDbName: DatabaseName,
readDbName: DatabaseName,
ldapConnectionPoolSize: Int,
ldapConnectionPoolName: String): cats.effect.Resource[IO, (DirectoryDAO, AccessPolicyDAO, ExecutionContext, RegistrationDAO)] = {
for {
dbReference <- DbReference.resource(appConfig.liquibaseConfig, dbName)
writeDbRef <- DbReference.resource(appConfig.liquibaseConfig, writeDbName)
readDbRef <- DbReference.resource(appConfig.liquibaseConfig, readDbName)
ldapConnectionPool <- createLdapConnectionPool(appConfig.directoryConfig.directoryUrl, appConfig.directoryConfig.user, appConfig.directoryConfig.password, ldapConnectionPoolSize, ldapConnectionPoolName)
ldapExecutionContext <- ExecutionContexts.fixedThreadPool[IO](appConfig.directoryConfig.connectionPoolSize)
postgresExecutionContext <- ExecutionContexts.fixedThreadPool[IO](DBs.config.getInt(s"db.${dbName.name.name}.poolMaxSize"))

directoryDAO = createDirectoryDAO(dbReference, postgresExecutionContext)
accessPolicyDAO = createAccessPolicyDAO(dbReference, postgresExecutionContext)
directoryDAO = new PostgresDirectoryDAO(writeDbRef, readDbRef)
accessPolicyDAO = new PostgresAccessPolicyDAO(writeDbRef, readDbRef)
registrationDAO = createRegistrationDAO(appConfig, ldapExecutionContext, ldapConnectionPool)
} yield (directoryDAO, accessPolicyDAO, ldapExecutionContext, registrationDAO)
}

private def createDirectoryDAO(dbReference: DbReference, postgresExecutionContext: ExecutionContext) = {
new PostgresDirectoryDAO(dbReference, postgresExecutionContext)
}

private def createRegistrationDAO(appConfig: AppConfig,
ldapExecutionContext: ExecutionContext,
ldapConnectionPool: LDAPConnectionPool): RegistrationDAO = {
new LdapRegistrationDAO(ldapConnectionPool, appConfig.directoryConfig, ldapExecutionContext)
}

private def createAccessPolicyDAO(dbReference: DbReference, postgresExecutionContext: ExecutionContext) = {
new PostgresAccessPolicyDAO(dbReference, postgresExecutionContext)
}

private[sam] def createGoogleCloudExt(
accessPolicyDAO: AccessPolicyDAO,
directoryDAO: DirectoryDAO,
Expand Down Expand Up @@ -309,7 +300,7 @@ object Boot extends IOApp with LazyLogging {
val policyEvaluatorService = PolicyEvaluatorService(config.emailDomain, resourceTypeMap, accessPolicyDAO, directoryDAO)
val resourceService = new ResourceService(resourceTypeMap, policyEvaluatorService, accessPolicyDAO, directoryDAO, cloudExtensionsInitializer.cloudExtensions, config.emailDomain)
val userService = new UserService(directoryDAO, cloudExtensionsInitializer.cloudExtensions, registrationDAO, config.blockedEmailDomains)
val statusService = new StatusService(directoryDAO, cloudExtensionsInitializer.cloudExtensions, DbReference(DatabaseNames.Foreground), 10 seconds)
val statusService = new StatusService(directoryDAO, cloudExtensionsInitializer.cloudExtensions, DbReference(DatabaseNames.Read, implicitly), 10 seconds)
dvoet marked this conversation as resolved.
Show resolved Hide resolved
val managedGroupService =
new ManagedGroupService(resourceService, policyEvaluatorService, resourceTypeMap, accessPolicyDAO, directoryDAO, cloudExtensionsInitializer.cloudExtensions, config.emailDomain)
val samApplication = SamApplication(userService, resourceService, statusService)
Expand Down
2 changes: 1 addition & 1 deletion src/main/scala/org/broadinstitute/dsde/workbench/sam/api/SamRoutes.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ import org.broadinstitute.dsde.workbench.model.{ErrorReport, WorkbenchExceptionW
import org.broadinstitute.dsde.workbench.sam._
import org.broadinstitute.dsde.workbench.sam.api.SamRoutes._
import org.broadinstitute.dsde.workbench.sam.config.{LiquibaseConfig, SwaggerConfig}
import org.broadinstitute.dsde.workbench.sam.directory.DirectoryDAO
import org.broadinstitute.dsde.workbench.sam.dataAccess.DirectoryDAO
import org.broadinstitute.dsde.workbench.sam.service._

import scala.concurrent.{ExecutionContext, Future}
Expand Down