Flat queries #487

dvoet · 2020-11-25T13:35:52Z

Ticket: https://broadworkbench.atlassian.net/browse/CA-315

There are some docs in the README, start there
I created a new package org/broadinstitute/dsde/workbench/sam/dataAccess and moved most of the DAOs there

PR checklist

I've followed the instructions if I've made any changes to the API, especially if they're breaking changes
I've updated the RC_XXX release ticket with any manual steps required to release this change
I've updated the FISMA documentation if I've made any security-related changes, including auth, encryption, or auditing

coveralls · 2020-11-25T13:49:37Z

Coverage decreased (-0.06%) to 78.707% when pulling 0aac527 on flat_queries into 0e5f1ea on develop.

dvoet · 2020-12-15T00:42:56Z

jenkins retest

dvoet · 2021-01-13T14:51:22Z

jenkins retest

dvoet · 2021-01-19T16:47:04Z

src/main/scala/org/broadinstitute/dsde/workbench/sam/db/DbReference.scala

-  )(_ => IO(DBs.close(dbName.name)))
+  def resource(liquibaseConfig: LiquibaseConfig, dbName: DatabaseName): Resource[IO, DbReference] = {
+    for {
+      dbExecutionContext <- ExecutionContexts.fixedThreadPool[IO](DBs.config.getInt(s"db.${dbName.name.name}.poolMaxSize"))


note this change moves the thread pool associated to a db connection internal to the db ref.

dvoet · 2021-01-19T16:56:59Z

src/main/scala/org/broadinstitute/dsde/workbench/sam/service/ResourceService.scala

-  private def persistResource(resourceType: ResourceType, resourceId: ResourceId, policies: Set[ValidatableAccessPolicy], authDomain: Set[WorkbenchGroupName], parentOpt: Option[FullyQualifiedResourceId], samRequestContext: SamRequestContext) =
-    for {
-      resource <- accessPolicyDAO.createResource(Resource(resourceType.name, resourceId, authDomain, parent = parentOpt), samRequestContext)
-      policies <- policies.toList.traverse(p => createOrUpdatePolicy(FullyQualifiedPolicyId(resource.fullyQualifiedId, p.policyName), p, samRequestContext))
-    } yield resource.copy(accessPolicies = policies.toSet)
+  private def persistResource(resourceType: ResourceType, resourceId: ResourceId, policies: Set[ValidatableAccessPolicy], authDomain: Set[WorkbenchGroupName], parentOpt: Option[FullyQualifiedResourceId], samRequestContext: SamRequestContext) = {
+    val accessPolicies = policies.map(constructAccessPolicy(resourceType, resourceId, _, public = false)) // can't set public at create time
+    accessPolicyDAO.createResource(Resource(resourceType.name, resourceId, authDomain, accessPolicies = accessPolicies, parent = parentOpt), samRequestContext)
+  }


change to create a resource plus all policies in a single transaction

dvoet · 2021-01-19T16:57:36Z

src/main/scala/org/broadinstitute/dsde/workbench/sam/service/ResourceService.scala


-      _ <- policiesToDelete.toList.parTraverse(p => accessPolicyDAO.deletePolicy(p.id, samRequestContext)).unsafeToFuture()
-      _ <- maybeDeleteResource(resource, samRequestContext)
+      _ <- accessPolicyDAO.deleteAllResourcePolicies(resource, samRequestContext).unsafeToFuture()


delete all policies in a single transaction

dvoet · 2021-01-19T17:01:35Z

src/test/scala/org/broadinstitute/dsde/workbench/sam/dataAccess/PostgresDirectoryDAOSpec.scala

+        val groupCount = 40
+        val userCount = 50


scaled down for the sake of elapsed time of test

dvoet · 2021-01-19T17:05:00Z

src/test/scala/org/broadinstitute/dsde/workbench/sam/service/PolicyEvaluatorServiceSpec.scala

-    policy.members.toList.parTraverse {
+    policy.members.toList.traverse {


given the small database there are too many serializable failures with parTraverse so it turns out to be slower

marctalbott

Nothing blocking, mostly just questions and nits

...ain/resources/org/broadinstitute/dsde/sam/liquibase/changesets/20201123_effective_policy.xml

src/main/scala/org/broadinstitute/dsde/workbench/sam/dataAccess/RegistrationDAO.scala

...ala/org/broadinstitute/dsde/workbench/sam/dataAccess/EffectivePolicyMutationStatements.scala

src/main/scala/org/broadinstitute/dsde/workbench/sam/dataAccess/PostgresGroupDAO.scala

src/test/scala/org/broadinstitute/dsde/workbench/sam/dataAccess/PostgresDirectoryDAOSpec.scala

...ala/org/broadinstitute/dsde/workbench/sam/dataAccess/EffectivePolicyMutationStatements.scala

...ain/resources/org/broadinstitute/dsde/sam/liquibase/changesets/20201123_effective_policy.xml

src/main/scala/org/broadinstitute/dsde/workbench/sam/util/DatabaseSupport.scala

gpolumbo-broad · 2021-01-25T21:57:09Z

src/test/scala/org/broadinstitute/dsde/workbench/sam/TestSupport.scala

@@ -137,7 +136,7 @@ object TestSupport extends TestSupport {

  def genSamRoutesWithDefault(implicit system: ActorSystem, materializer: Materializer): SamRoutes = genSamRoutes(genSamDependencies(), UserInfo(OAuth2BearerToken(""), genWorkbenchUserId(System.currentTimeMillis()), defaultUserEmail, 3600))

-  lazy val dbRef = DbReference.init(config.as[LiquibaseConfig]("liquibase"), DatabaseNames.Foreground)
+  lazy val dbRef = DbReference.init(config.as[LiquibaseConfig]("liquibase"), DatabaseNames.Read, TestSupport.blockingEc)


should this be readDbRef maybe? I'm looking at line 142 below and we're using the read pool to truncate the database. In test world, this is fine, but things might start getting confusing in the tests if we're using read and write pools willy nilly.

In unit tests there really is not a difference between read and write pools. Ideally I would not even have it. But I also want to have DatabaseNames enum and DbReference.init to use it. So the situation is a little messy and I favor having more mess on the test side than the production side (i.e. I don't want to add a new database name just for tests).

Can we leave a comment explaining this then? I think it is likely for someone to raise a question about this in the future.

src/main/scala/org/broadinstitute/dsde/workbench/sam/dataAccess/PostgresGroupDAO.scala

src/main/scala/org/broadinstitute/dsde/workbench/sam/dataAccess/PostgresAccessPolicyDAO.scala

src/main/scala/org/broadinstitute/dsde/workbench/sam/dataAccess/PostgresGroupDAO.scala

...ain/resources/org/broadinstitute/dsde/sam/liquibase/changesets/20201123_effective_policy.xml

s-rubenstein · 2021-01-28T19:42:08Z

...ain/resources/org/broadinstitute/dsde/sam/liquibase/changesets/20201123_effective_policy.xml

+            from SAM_EFFECTIVE_RESOURCE_POLICY ep
+            join SAM_POLICY_ROLE pr on ep.source_policy_id = pr.resource_policy_id
+            join SAM_RESOURCE resource on ep.resource_id = resource.id
+            join sam_flattened_role flat_role on pr.resource_role_id = flat_role.base_role_id


Nit: Out of curiosity, should sam_flattened_role be in all caps here?

doesn't matter

If it doesn't matter, would you mind moving it to caps for consistency?

...esources/org/broadinstitute/dsde/sam/liquibase/changesets/20201123_flat_group_membership.xml

...ala/org/broadinstitute/dsde/workbench/sam/dataAccess/EffectivePolicyMutationStatements.scala

s-rubenstein · 2021-01-29T22:40:45Z

...ala/org/broadinstitute/dsde/workbench/sam/dataAccess/EffectivePolicyMutationStatements.scala

+    *            F                  |             T                |                F                |    F
+    *            F                  |             F                |                T                |    F
+    *            F                  |             F                |                F                |    T
+    *


Just to make sure I am understanding the logic behind this table, it is:
I xor not(PR or FR)

If so, that may be a useful thing to put in the comment.

That may also be a simpler way to write the query? If that is the logic?

@marctalbott figured this out, if there were XOR in SQL it might be better but there isn't :(

Yeah, the query looks good, but I was wondering about putting it above the truth table in the comment, just to make it more clear at a first glance. Although since we can't represent the sql that way, I'm okay with leaving it for now.

src/main/scala/org/broadinstitute/dsde/workbench/sam/dataAccess/PostgresAccessPolicyDAO.scala

src/main/scala/org/broadinstitute/dsde/workbench/sam/dataAccess/PostgresGroupDAO.scala

dvoet · 2021-02-03T00:36:27Z

jenkins retest - strange ivy thing on rawls

s-rubenstein

LGTM at this point, don't think any of my nits should hold this up.

s-rubenstein · 2021-02-03T15:34:13Z

...ala/org/broadinstitute/dsde/workbench/sam/dataAccess/EffectivePolicyMutationStatements.scala

+    *            F                  |             T                |                F                |    F
+    *            F                  |             F                |                T                |    F
+    *            F                  |             F                |                F                |    T
+    *


Yeah, the query looks good, but I was wondering about putting it above the truth table in the comment, just to make it more clear at a first glance. Although since we can't represent the sql that way, I'm okay with leaving it for now.

...ain/resources/org/broadinstitute/dsde/sam/liquibase/changesets/20201123_effective_policy.xml

s-rubenstein · 2021-02-03T15:37:17Z

...ain/resources/org/broadinstitute/dsde/sam/liquibase/changesets/20201123_effective_policy.xml

+            from SAM_EFFECTIVE_RESOURCE_POLICY ep
+            join SAM_POLICY_ROLE pr on ep.source_policy_id = pr.resource_policy_id
+            join SAM_RESOURCE resource on ep.resource_id = resource.id
+            join sam_flattened_role flat_role on pr.resource_role_id = flat_role.base_role_id


If it doesn't matter, would you mind moving it to caps for consistency?

zbedo · 2021-02-04T14:43:09Z

👍

dvoet · 2021-02-04T16:32:09Z

jenkins retest

dvoet force-pushed the flat_queries branch from cb05e7b to c248a75 Compare January 13, 2021 14:31

dvoet commented Jan 19, 2021

View reviewed changes

dvoet requested review from marctalbott and gpolumbo-broad January 19, 2021 17:17

marctalbott reviewed Jan 22, 2021

View reviewed changes

gpolumbo-broad reviewed Jan 27, 2021

View reviewed changes

dvoet added 4 commits January 28, 2021 10:45

create and populate effective policies

5de5340

cleanup

205f694

added test

a6d2b05

review feedback

f074891

dvoet force-pushed the flat_queries branch from 0aac527 to f074891 Compare January 30, 2021 03:13

marctalbott approved these changes Feb 1, 2021

View reviewed changes

s-rubenstein reviewed Feb 2, 2021

View reviewed changes

dvoet added 2 commits February 2, 2021 11:16

test migration sql

9d7a53e

a touch more feedback

3301868

s-rubenstein approved these changes Feb 3, 2021

View reviewed changes

consistent case

c1b404e

dvoet merged commit 3096814 into develop Feb 4, 2021

dvoet deleted the flat_queries branch February 4, 2021 19:00

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Flat queries #487

Flat queries #487

dvoet commented Nov 25, 2020 •

edited

Loading

coveralls commented Nov 25, 2020 •

edited

Loading

dvoet commented Dec 15, 2020

dvoet commented Jan 13, 2021

dvoet Jan 19, 2021

dvoet Jan 19, 2021

dvoet Jan 19, 2021

dvoet Jan 19, 2021

dvoet Jan 19, 2021

marctalbott left a comment

gpolumbo-broad Jan 25, 2021

dvoet Jan 29, 2021

gpolumbo-broad Feb 1, 2021

s-rubenstein Jan 28, 2021

dvoet Feb 2, 2021

s-rubenstein Feb 3, 2021

s-rubenstein Jan 29, 2021

s-rubenstein Jan 29, 2021

dvoet Feb 2, 2021

s-rubenstein Feb 3, 2021

dvoet commented Feb 3, 2021

s-rubenstein left a comment

s-rubenstein Feb 3, 2021

s-rubenstein Feb 3, 2021

zbedo commented Feb 4, 2021

dvoet commented Feb 4, 2021

		policy.members.toList.parTraverse {
		policy.members.toList.traverse {

Flat queries #487

Flat queries #487

Conversation

dvoet commented Nov 25, 2020 • edited Loading

coveralls commented Nov 25, 2020 • edited Loading

dvoet commented Dec 15, 2020

dvoet commented Jan 13, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

marctalbott left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dvoet commented Feb 3, 2021

s-rubenstein left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zbedo commented Feb 4, 2021

dvoet commented Feb 4, 2021

dvoet commented Nov 25, 2020 •

edited

Loading

coveralls commented Nov 25, 2020 •

edited

Loading