feat(query-generator): Generate INSERTs using bind parameters #9431

gazoakley · 2018-05-13T11:27:05Z

Pull Request check-list

Please make sure to review and check all of these items:

Does npm run test or npm run test-DIALECT pass with this change (including linting)?
Does the description below contain a link to an existing issue (Closes #[issue]) or a description of the issue you are solving?
Have you added new tests to prevent regressions?
Is a documentation update included (if this change modifies existing APIs, or introduces new ones)?
Did you follow the commit message conventions explained in CONTRIBUTING.md?

Description of change

This is a new spin of #9371 where I've removed the bindParams option and updated all the INSERT unit tests to match the new syntax. It's thrown up a few issues:

Data types that don't easily match up with built in JS data types (specifically GEOMETRY) cause issues since drivers can't identify them before sending them as bind parameters. They need to be sent using a correct CAST and providing the bind parameter in string form e.g. GeomFromText($1) with a valid bind param
Upsert needs updating - both PostgreSQL and Sqlite override the upsertQuery function in the query generator

sushantdhiman · 2018-05-21T12:29:15Z

lib/data-types.js

@@ -725,6 +739,9 @@ GEOGRAPHY.prototype.escape = false;
 GEOGRAPHY.prototype._stringify = function _stringify(value, options) {
  return 'GeomFromText(' + options.escape(Wkt.convert(value)) + ')';
 };
+GEOMETRY.prototype.bindParam = function bindParam(value, options) {


_bindParam to match _stringify etc stuff

sushantdhiman · 2018-05-21T12:30:23Z

lib/dialects/abstract/query-generator.js

@@ -989,11 +989,18 @@ class QueryGenerator {
    return SqlString.escape(value, this.options.timezone, this.dialect);
  }

+  bindParam(bind) {


sushantdhiman · 2018-05-21T12:32:23Z

test/integration/data-types.test.js

@@ -56,14 +56,18 @@ describe(Support.getTestDialectTeaser('DataTypes'), () => {
    });
  });

-  const testSuccess = function(Type, value) {
+  const testSuccess = function(Type, value, useBindParam) {


Third argument should be options with arbitrary configuration possibilities { useBindParam }

codecov · 2018-05-23T10:30:22Z

Codecov Report

Merging #9431 into master will decrease coverage by <.01%.
The diff coverage is 96.8%.

gazoakley · 2018-05-25T09:18:13Z

@sushantdhiman Hit a bit of a snag doing upserts with Postgres. The current upsertQuery function generates SQL along the lines of:

CREATE OR REPLACE FUNCTION pg_temp.sequelize_upsert(OUT created boolean,OUT primary_key text)  AS $func$
BEGIN
INSERT INTO "users" ("id","username","zab","createdAt","updatedAt") VALUES (1,"john","foo","1970-01-01 00:00:00.000 +00:00","1970-01-01 00:00:00.000 +00:00"); created := true;
EXCEPTION WHEN unique_violation THEN
UPDATE "users" SET "id"=42,"username"='john',"updatedAt"='1970-01-01 00:00:00.000 +00:00' WHERE ("id" = 42); created := false;
END;
$func$ LANGUAGE plpgsql;
SELECT * FROM pg_temp.sequelize_upsert();

That's fine when not using bind parameters/prepared statements, but when I generate something along the lines of:

CREATE OR REPLACE FUNCTION pg_temp.sequelize_upsert(arg1 <argtype>,arg2 <argtype>,arg3 <argtype>,arg4 <argtype>,arg5 <argtype>,OUT created boolean,OUT primary_key text)  AS $func$
BEGIN
INSERT INTO "users" ("id","username","zab","createdAt","updatedAt") VALUES (arg1,arg2,arg3,arg4,arg5); created := true;
EXCEPTION WHEN unique_violation THEN
UPDATE "users" SET "id"=42,"username"='john',"updatedAt"='1970-01-01 00:00:00.000 +00:00' WHERE ("id" = 42); created := false;
END;
$func$ LANGUAGE plpgsql;
SELECT * FROM pg_temp.sequelize_upsert($1,$2,$3,$4,$5);

I get an error back from Postgres of cannot insert multiple commands into a prepared statement - I can't string together multiple statements if using prepared statements. I think the result of upsertQuery for Postgres is going to have to return multiple statements to accomplish what its trying to do. I'm thinking of having it return along the lines of:

[
    {
        "query": "CREATE OR REPLACE FUNCTION pg_temp.sequelize_upsert(arg1 <argtype>,arg2 <argtype>,arg3 <argtype>,arg4 <argtype>,arg5 <argtype>,OUT created boolean,OUT primary_key text)  AS $$func$$ BEGIN INSERT INTO \"users\" (\"id\",\"username\",\"zab\",\"createdAt\",\"updatedAt\") VALUES (arg1,arg2,arg3,arg4,arg5); created := true; EXCEPTION WHEN unique_violation THEN UPDATE \"users\" SET \"id\"=42,\"username\"='john',\"updatedAt\"='1970-01-01 00:00:00.000 +00:00' WHERE (\"id\" = 42); created := false; END; $$func$$ LANGUAGE plpgsql;",
        "bind": []
    },
    {
        "query": "SELECT * FROM pg_temp.sequelize_upsert($1,$2,$3,$4,$5);",
        "bind": [42, "john", "foo", "1970-01-01 00:00:00.000 +00:00", "1970-01-01 00:00:00.000 +00:00"]
    }
]

And have something further up treat an array of query elements as something that must be run as a sequence of statements that must be run one after each other. Does that seem reasonable to you?

sushantdhiman · 2018-05-28T03:54:13Z

You are right, we can't use parameterized query with multiple statements. It seems we need to execute two queries with correct order, also keeping other upsert from affecting each other.

And have something further up treat an array of query elements as something that must be run as a sequence of statements that must be run one after each other. Does that seem reasonable to you?

Yeah I agree, we need to return multiple statements with their parameters.

In old approach we used to send one single query which will create / replace function and execute it at server level, but now we need to wrap Model.upsert in transaction. As two different statements are going to be executed, we dont want one upsert to be replacing other upsert function and return inconsistent results.

Another potential solution is to support ON CONFLICT but we still need to support this for Postgres < 9.5 so multiple statements solution still needs to be implemented.

So overall, we need to do this

Return array of statements from query generator
Execute them in order
Wrap upsert in transaction, so there is no cross talk between other upsert trying to replace same function

gazoakley · 2018-05-28T09:31:08Z

The tests are now passing on everything apart from Postgres which is becoming increasingly complex. There's still some fixes needed to support RANGE correctly with bind params, and I've also spotted the Postgres specific code that is returned by the abstract insertQuery which needs the same treatment.

This PR is starting to get quite large too. So that I can split this work up better, would it be acceptable to have an option to turn off bindParams? This would let me:

Turn it off by default if insertQuery is passed options.exception as true
Turn it off when calling insertQuery from Postgres upsertQuery

I can then postpone having to deal with returning multiple queries and the further complexity that entails. There's also a good opportunity at that point to remove the duplicate code in insertQuery so that it calls fn/exceptionFn in postgres/query-generator.js

janmeier · 2018-05-28T09:38:54Z

Sounds good to me :)

sushantdhiman · 2018-05-28T09:43:51Z

Sounds good if we keep it as a private option, Use it internally so we can keep converting old queries to parameter based queries and then remove it eventually

gazoakley · 2018-05-29T08:31:46Z

@janmeier @sushantdhiman Any chance either of you could try running the tests yourselves? I can't reproduce the failures in Travis 😕

janmeier · 2018-05-29T08:59:32Z

Works locally for me as well....

janmeier · 2018-05-29T09:12:18Z

You need to rebase on top of latest master (specifically this commit b66f9dd has some tests that are now failing)

I was tipped up by the fact that my local tests said 1764 passing, while travis was 1764 passing, 4 failing so I figured my local copy was missing something

gazoakley · 2018-05-29T10:08:27Z

@janmeier Ooops! I'll try and sort it over lunch.

gazoakley · 2018-05-29T12:57:45Z

Tests are all green 🎉

janmeier · 2018-05-29T13:08:38Z

test/unit/dialects/sqlite/query-generator.test.js

+          expectation: {
+            query: 'INSERT INTO `myTable` (`name`,`value`) VALUES ($1,$2);',
+            // TODO: Check
+            bind: ['foo', true]


Should be fine (I'm pretty sure sqlite takes whatever you throw at it)

janmeier · 2018-05-29T13:09:03Z

test/unit/dialects/sqlite/query-generator.test.js

-          expectation: "INSERT INTO `myTable` (`name`,`value`) VALUES ('bar',NULL);"
+          expectation: {
+            query: 'INSERT INTO `myTable` (`name`,`value`) VALUES ($1,$2);',
+            // TODO: Check


Hmm, not sure - There is another issue about ignoring undefined completely, but it should be fine for now

sushantdhiman

Looks good, some small changes

sushantdhiman · 2018-05-30T03:52:21Z

lib/dialects/abstract/query-generator.js

+    query = _.template(query, this._templateSettings)(replacements);
+    // Used by Postgres upsertQuery and calls to here with options.exception set to true
+    if (options.bindParam === false) {
+      return query;


return an object in both cases, so all calls can just use .query. In case options.bindParam: false return no bind parameters.

sushantdhiman · 2018-05-30T03:54:12Z

lib/dialects/abstract/query-generator.js

@@ -169,7 +171,14 @@ class QueryGenerator {
      }
    }

+    if (_.get(this, ['sequelize', 'options', 'dialectOptions', 'prependSearchPath']) || options.searchPath) {


Use full path sequelize.options.dialectOptions.prependSearchPath in _.get

I can change it, but it's more efficient in Lodash to use the array path form. If you use the string path form (e.g. _.get(this, 'sequelize.options.dialectOptions.prependSearchPath')) Lodash first converts it back to the array path form which currently entails a bunch of regex operations to parse the path. For a single call like this it probably doesn't matter, but do it many times and there's a big difference in performance.

@gazoakley No need to change it then :)

sushantdhiman · 2018-05-30T03:58:12Z

test/integration/dialects/postgres/dao.test.js

        return this.User.create({
          dates: []
        }, {
          logging(sql) {
-            expect(sql.indexOf('TIMESTAMP WITH TIME ZONE')).to.be.greaterThan(0);
+            expect(sql).not.to.contain('TIMESTAMP WITH TIME ZONE');
+            expect(sql).not.to.contain('DATETIME');


assert presence of bind param?

janmeier

I think all the tests asserting on SQL can be covered with unit tests instead

gazoakley · 2018-05-30T09:45:28Z

Now we just need to make UPDATE, SELECT and DELETE use bind params 🤣

kirrg001 · 2021-09-16T12:37:55Z

@gazoakley Hi there! Is there any progress on supporting bind params for select, update and delete? Thanks a lot!

sushantdhiman reviewed May 21, 2018

View reviewed changes

sushantdhiman mentioned this pull request May 25, 2018

feat(query-generator): Generate INSERTs using bind parameters #9371

Closed

5 tasks

gazoakley force-pushed the feat-insert-bind-iii branch 3 times, most recently from d3bb785 to 09fe634 Compare May 28, 2018 13:11

sushantdhiman added the wip label May 29, 2018

gazoakley force-pushed the feat-insert-bind-iii branch 3 times, most recently from 0fb148e to 8614879 Compare May 29, 2018 12:47

janmeier approved these changes May 29, 2018

View reviewed changes

gazoakley force-pushed the feat-insert-bind-iii branch from 8614879 to bf9eb5e Compare May 29, 2018 13:47

gazoakley changed the title ~~WIP: feat(query-generator): Generate INSERTs using bind parameters~~ feat(query-generator): Generate INSERTs using bind parameters May 29, 2018

sushantdhiman reviewed May 30, 2018

View reviewed changes

sushantdhiman removed the wip label May 30, 2018

janmeier approved these changes May 30, 2018

View reviewed changes

feat(query-generator): Generate INSERTs using bind parameters

8fc477a

gazoakley force-pushed the feat-insert-bind-iii branch from bf9eb5e to 8fc477a Compare May 30, 2018 08:55

sushantdhiman approved these changes May 30, 2018

View reviewed changes

sushantdhiman merged commit bce3d9a into sequelize:master May 30, 2018

intellix mentioned this pull request Aug 23, 2022

Prepared Statements and Security #1608

Closed

feat(query-generator): Generate INSERTs using bind parameters #9431

feat(query-generator): Generate INSERTs using bind parameters #9431

gazoakley commented May 13, 2018 •

edited

sushantdhiman May 21, 2018

sushantdhiman May 21, 2018

sushantdhiman May 21, 2018

codecov bot commented May 23, 2018 •

edited

gazoakley commented May 25, 2018

sushantdhiman commented May 28, 2018

gazoakley commented May 28, 2018

janmeier commented May 28, 2018 •

edited

sushantdhiman commented May 28, 2018

gazoakley commented May 29, 2018

janmeier commented May 29, 2018

janmeier commented May 29, 2018

gazoakley commented May 29, 2018

gazoakley commented May 29, 2018

janmeier May 29, 2018

janmeier May 29, 2018

sushantdhiman left a comment

sushantdhiman May 30, 2018

gazoakley May 30, 2018

sushantdhiman May 30, 2018

gazoakley May 30, 2018

sushantdhiman May 30, 2018

sushantdhiman May 30, 2018

janmeier left a comment

gazoakley commented May 30, 2018

kirrg001 commented Sep 16, 2021 •

edited

feat(query-generator): Generate INSERTs using bind parameters #9431

feat(query-generator): Generate INSERTs using bind parameters #9431

Conversation

gazoakley commented May 13, 2018 • edited

Pull Request check-list

Description of change

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented May 23, 2018 • edited

Codecov Report

gazoakley commented May 25, 2018

sushantdhiman commented May 28, 2018

gazoakley commented May 28, 2018

janmeier commented May 28, 2018 • edited

sushantdhiman commented May 28, 2018

gazoakley commented May 29, 2018

janmeier commented May 29, 2018

janmeier commented May 29, 2018

gazoakley commented May 29, 2018

gazoakley commented May 29, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sushantdhiman left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

janmeier left a comment

Choose a reason for hiding this comment

gazoakley commented May 30, 2018

kirrg001 commented Sep 16, 2021 • edited

gazoakley commented May 13, 2018 •

edited

codecov bot commented May 23, 2018 •

edited

janmeier commented May 28, 2018 •

edited

kirrg001 commented Sep 16, 2021 •

edited