reorganize sql manager and add CockroachDB support #87

zikes · 2017-11-01T17:57:39Z

Our company needed CockroachDB support, which is very similar to PostgreSQL. CRDB uses the same wire protocol as Postgres, and the SQL format is mostly the same. In an the attempt, I ran into some complications that motivated me to reorganize the SQL manager to be more flexible, if a little more verbose.

Database-specific SQL has been moved to a separate databases.go file, which has a map of supported databases and their configuration. This should allow future databases to be added to the manager externally, as the Databases map is exported. I also copied the SQL that happens to match between databases, rather than store them separately and referencing that, to increase readability.

aeneasr

Thank you for the changes. I'm not sure I want cockroach flavored SQL added in the main repository, as it has to be maintained officially and I have to write separate migration routines for a technology I don't use & know.

One path to resolve this would be to have a map[string]StatementConfig in SQLManager that replaced the global Databases reference. That way it would be possible to compose the SQLManager with your own, custom schema without adding it to the main repository.

However, you're addressing some things which I like. Before merging, please address the mentioned things. Depending on how to proceed, some of them might be ignored (like keeping the export of Database).

Let me know what you think.

aeneasr · 2017-11-01T18:25:19Z

manager/sql/manager_sql.go

-	for _, v := range relations {
-		for _, template := range v.p {
+	for _, rel := range relations {
+		var subject string


This should be query or something, subject implies that it's a policy subject which isn't the case

Agreed. I struggled to find the right variable name for this, and query still doesn't seem quite right, but it is at least less confusing than subject.

aeneasr · 2017-11-01T18:28:18Z

manager/sql/databases.go

+
+type Database struct {
+	Migrations               *migrate.MemoryMigrationSource
+	CreatePolicy             string


Please rename this to QueryInsertPolicy

aeneasr · 2017-11-01T18:28:38Z

manager/sql/databases.go

+	CreatePolicyResources    string
+	CreatePolicyResourcesRel string
+	CreatePolicySubjects     string
+	CreatePolicySubjectsRel  string


And subsequently for all of the above

aeneasr · 2017-11-01T18:31:16Z

manager/sql/databases.go

+
+import migrate "github.com/rubenv/sql-migrate"
+
+type Database struct {


Please rename this to sqlStatements - exporting it is not required because it's an internal struct

In order to support external additions of databases which are not officially supported, this would need to remain exported. In that case, would you prefer SQLStatements?

Yes, I think Statements should actually be enough!

aeneasr · 2017-11-01T18:39:57Z

manager/sql/manager_sql.go

-	db     *sqlx.DB
-	schema []string
+	db       *sqlx.DB
+	schema   []string


Doesn't seem like this is being used any more

Agreed. Removing this might require changing the signature of the NewSQLManager function, currently func NewSQLManager(db *sqlx.DB, schema []string) *SQLManager. I could change that to func NewSQLManager(db *sql.DB) *SQLManager or continue accepting schema and ignore it until a major version release?

aeneasr · 2017-11-02T09:55:26Z

Let me know when this is ready for another pass!

zikes · 2017-11-02T12:22:22Z

I believe I've implemented all your requests, please review and merge at your convenience.

aeneasr

Thank you, a couple more questions and also some changes I'd like to see

aeneasr · 2017-11-05T10:03:02Z

manager/sql/databases.go

+				},
+			},
+		},
+		QueryInsertPolicy:             `INSERT INTO ladon_policy(id, description, effect, conditions) SELECT $1::varchar, $2, $3, $4 WHERE NOT EXISTS (SELECT 1 FROM ladon_policy WHERE id = $1)`,


Do you know why this (::varchar) was previously not necessary?

The PostgreSQL driver's type inference was returning two different types for the first and second instances of $1 in the prepared statement, so the ::varchar settled that. It was originally unnecessary because $1 wasn't reused later in this query, the ID value was passed in twice.

aeneasr · 2017-11-05T10:05:07Z

manager/sql/databases.go

+			Migrations: []*migrate.Migration{
+				{
+					Id: "1",
+					Up: []string{


I think it makes sense to use the same up/down statements wherever possible which in this case would be that postgres and mysql have the same up/down statements for id 1. What do you think?

When I was reorganizing with the goal of adding official support for more databases, it made more sense to me to have each database's statements remain separate for readability. I can revert this since the library will only be supporting MySQL & PostgreSQL.

aeneasr · 2017-11-05T10:07:06Z

manager/sql/manager_sql.go

+		return errors.Errorf("Database %s is not supported", s.database)
+	}
+
+	if _, err = tx.Exec(Databases[s.database].QueryInsertPolicy, policy.GetID(), policy.GetDescription(), policy.GetEffect(), conditions); err != nil {


By using db.Rebind() we can actually further streamline the SQL statements and use ? everywhere instead of dialect-specific parameters.

aeneasr · 2017-11-05T10:07:21Z

manager/sql/manager_sql.go

-				break
-			default:
-				return errors.Errorf("Database driver %s is not supported", s.db.DriverName())
+			if _, err := tx.Exec(query, id, template, compiled.String(), strings.Index(template, string(policy.GetStartDelimiter())) >= -1); err != nil {


By using db.Rebind() we can actually further streamline the SQL statements and use ? everywhere instead of dialect-specific parameters.

aeneasr · 2017-11-05T10:07:25Z

manager/sql/manager_sql.go

+			if _, err := tx.Exec(query, id, template, compiled.String(), strings.Index(template, string(policy.GetStartDelimiter())) >= -1); err != nil {
+				return errors.WithStack(err)
+			}
+			if _, err := tx.Exec(queryRel, policy.GetID(), id); err != nil {


By using db.Rebind() we can actually further streamline the SQL statements and use ? everywhere instead of dialect-specific parameters.

aeneasr · 2017-11-05T10:08:03Z

manager/sql/manager_sql.go

-	default:
-		return nil, errors.Errorf("Database driver %s is not supported", s.db.DriverName())
-	}
+	query := Databases[s.database].FindRequestCandidates

 	rows, err := s.db.Query(query, r.Subject, r.Subject)


I missed db.Rebind here myself but I think we should add it here too.

zikes · 2017-11-07T21:25:33Z

This PR is ready for an additional review.

aeneasr

Awesome, thank you so much for staying with this PR and addressing all the issues. There are only two more, very minor, renames left. Once they are done I'll merge it as it LGTM!

aeneasr · 2017-11-08T09:23:25Z

manager/sql/databases.go

+	},
+}
+
+var Databases = map[string]Statements{


Rename to Migrations

aeneasr · 2017-11-08T09:24:55Z

manager/sql/databases.go

+	QueryInsertPolicyResourcesRel string
+	QueryInsertPolicySubjects     string
+	QueryInsertPolicySubjectsRel  string
+	FindRequestCandidates         string


Rename to QueryRequestCandidates

kminehart · 2017-11-08T14:21:00Z

Thanks @arekkas. I've made the 2 changes that were requested :)

This will close #86, as CRDB support will be provided via. an external library.

…ndidate)

andreis · 2017-11-08T18:28:29Z

We actually need cockroachDB support, so many thanks @kminehart and @arekkas and let me know if there's any way I can help!

kminehart · 2017-11-08T18:38:11Z

@andreis Thank @zikes ! I just changed a few variable names for him.

aeneasr · 2017-11-10T10:46:26Z

Thank you for all your hard work, sorry for the long response times, the past days have been very intense!

aeneasr · 2017-11-10T10:47:53Z

By the way, if you supply CockroachDB support in a separate repo I would be more than happy to link it in the README

zikes · 2017-11-10T21:27:37Z

Thanks! The CRDB repo is at https://github.com/wehco/ladon-crdb

aeneasr · 2017-11-12T10:45:47Z

7285d8c

Jason Hutchinson added 2 commits November 1, 2017 12:35

reorganize sql manager and add CockroachDB support

238d24c

fix issue with variable type deduction

279efcb

aeneasr requested changes Nov 1, 2017

View reviewed changes

Jason Hutchinson added 6 commits November 1, 2017 13:58

rename CreatePolicy to QueryInsertPolicy

6f3da65

rename subject to query to avoid confusion

b103a51

rename CreatePolicy to QueryInsertPolicy

eacf6f3

rename database configuration struct

3d6823e

remove cockroachdb support

e0ce7a3

remove unused schema from SQLManager struct

e598191

aeneasr reviewed Nov 5, 2017

View reviewed changes

Jason Hutchinson added 3 commits November 7, 2017 14:51

combine shared migrations

8c5de2c

trim extraneous type casting

ebe6d7b

use Rebind to normalize query parameters

49852c6

aeneasr reviewed Nov 8, 2017

View reviewed changes

FindRequestCandidates->QueryRequestCandidates, Databases->Migrations

5b31d3f

kminehart added 2 commits November 8, 2017 08:22

Fix usage of renamed variables

a3bb354

Fix usage of renamed variable (FindRequestCandidate -> QueryRequestCa…

42ce021

…ndidate)

aeneasr merged commit 5892158 into ory:master Nov 10, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

reorganize sql manager and add CockroachDB support #87

reorganize sql manager and add CockroachDB support #87

zikes commented Nov 1, 2017

aeneasr left a comment

aeneasr Nov 1, 2017

zikes Nov 1, 2017

aeneasr Nov 1, 2017

aeneasr Nov 1, 2017

aeneasr Nov 1, 2017

zikes Nov 1, 2017

aeneasr Nov 1, 2017

aeneasr Nov 1, 2017

zikes Nov 1, 2017

aeneasr commented Nov 2, 2017

zikes commented Nov 2, 2017

aeneasr left a comment

aeneasr Nov 5, 2017

zikes Nov 7, 2017

aeneasr Nov 10, 2017

aeneasr Nov 5, 2017

zikes Nov 7, 2017

aeneasr Nov 5, 2017

aeneasr Nov 5, 2017

aeneasr Nov 5, 2017

aeneasr Nov 5, 2017

zikes commented Nov 7, 2017

aeneasr left a comment

aeneasr Nov 8, 2017

aeneasr Nov 8, 2017

kminehart commented Nov 8, 2017 •

edited

Loading

andreis commented Nov 8, 2017

kminehart commented Nov 8, 2017

aeneasr commented Nov 10, 2017

aeneasr commented Nov 10, 2017

zikes commented Nov 10, 2017

aeneasr commented Nov 12, 2017


		import migrate "github.com/rubenv/sql-migrate"

		type Database struct {

reorganize sql manager and add CockroachDB support #87

reorganize sql manager and add CockroachDB support #87

Conversation

zikes commented Nov 1, 2017

aeneasr left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aeneasr commented Nov 2, 2017

zikes commented Nov 2, 2017

aeneasr left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zikes commented Nov 7, 2017

aeneasr left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kminehart commented Nov 8, 2017 • edited Loading

andreis commented Nov 8, 2017

kminehart commented Nov 8, 2017

aeneasr commented Nov 10, 2017

aeneasr commented Nov 10, 2017

zikes commented Nov 10, 2017

aeneasr commented Nov 12, 2017

kminehart commented Nov 8, 2017 •

edited

Loading