feat(query): bind parameters

[javiertury]

Pull Request check-list

• Does npm run test or npm run test-DIALECT pass with this change (including linting)?
• Does the description below contain a link to an existing issue (Closes #[issue]) or a description of the issue you are solving?
• Have you added new tests to prevent regressions?
• Is a documentation update included (if this change modifies existing APIs, or introduces new ones)?
• Did you follow the commit message conventions explained in CONTRIBUTING.md?

Description of change

Issue #10285. All user-facing queries use binded parameters in all dialects, except select queries on mysql dialect.

Changes

• Select queries on mysql are not binded because of node-mysql2
  • execute API: Not possible to use custom parsers, and no hints about data type.
  • node-mysql2 execute typecast issue needs to be solved.
  • No way to convert geometries to geoJSON
  • No way to apply timezone
  • mariadb-connector-nodejs can be used as an alternative
    • Limitations of mariadb driver would apply to mysql, e.g., DECIMAL and BIGINT parsed as Numbers
• All tests of user-facing queries use binded parameters
  • Commitment to bind everything in the future.
  • Only integration tests cover escaping values in select queries for mysql
• New Composition object that holds strings (sql commands), Slot objects(binded parameters) and placeholders. This object is converted to a query in the very end.
  • QueryGenerators returns Composition objects
  • sequelize.query() accepts a Composition object as argument. Composition objects provide guarantees that make formatBindParameters() unnecessary. No need to reparse query with regex.
  • Tests must use dialect specific bind parameters, except for default dialect
• Many helper functions are converted to return Composition objects
  • Binding is delayed
  • Can choose between binding with QB.composeQuery() and or escaping with QB.composeString()
• Removes support for old SQL Server 2008 (Microsoft will discontinue extended support in 2019)
  • Offset hack is very difficult to deal with
• Fix version detection using semver, coherce non-semver compliant strings
• Logs binded parameters(useful for testing)
• Postgres searchPath query is run separately if there are values to bind
• Corner case: in postgres concurrent findOrCreate queries sharing the same transaction use ignoreDuplicates instead of exception handling
  • Now everything can be in 1 query
  • Exception handling was expensive
  • Requires postgres 9.5+ to handle this corner case with a shared transaction, user is warned
• Upsert query is run using two separate queries in sqlite
  • sqlite cannot run multiple statements in a single query if values are binded
  • Minimal delay, both queries are run in parallel and database is local
  • Both queries are idempotent, tests are satisfied
• Placeholder is now a Class, not an operator
  • More secure. Previously operator substitution searched for a string and replaced it, which can be faked by user. Users cannot fake a class instance.
  • Works better with Composition class
• Some DBs have limits on number of binded parameters
  • Previously, sqlite could insert up to 999 rows per query with default server settings. Now the limit is 999 binded parameters per query, which translates to only (999/#binds per row) rows per query with default server settings. To raise this number, tweak sqlite's config SQLITE_LIMIT_VARIABLE_NUMBER.
  • SQL server has a fixed limit of 2100 binded parameters, which translates to (2100/#binds per row) rows
• Methods sequelize.slot() and sequelize.placeholder() can be combined with sequelize.composition() to bind values and create placeholders.
  • Documentation not yet public. This is a proposed API, but perhaps there is a better way.
• For json data types, both the value and the json path are binded
  • In mysql the ->> operator didn't allow to bind the json path. Changed to json_unquote(json_extract(...))

[codecov]

Codecov Report

Merging #10284 into master will increase coverage by 0.09%.
The diff coverage is 96.07%.

@@            Coverage Diff             @@
##           master   #10284      +/-   ##
==========================================
+ Coverage   96.25%   96.35%   +0.09%     
==========================================
  Files          97       97              
  Lines        9250     9213      -37     
==========================================
- Hits         8904     8877      -27     
+ Misses        346      336      -10

Impacted Files	Coverage Δ
lib/associations/belongs-to-many.js	97.93% <ø> (ø)	⬆️
lib/sequelize.js	96.53% <ø> (ø)	⬆️
lib/data-types.js	91.98% <100%> (+0.02%)	⬆️
lib/dialects/abstract/connection-manager.js	90.51% <100%> (ø)	⬆️
lib/dialects/sqlite/query.js	98.13% <100%> (-0.06%)	⬇️
...b/dialects/abstract/query-generator/composition.js	100% <100%> (ø)	⬆️
lib/dialects/abstract/query.js	92.35% <100%> (+0.31%)	⬆️
lib/dialects/postgres/query-generator.js	93.33% <100%> (ø)	⬆️
lib/hooks.js	96.96% <100%> (ø)	⬆️
lib/model.js	96.68% <100%> (ø)	⬆️
... and 15 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3af556a...1737ccc. Read the comment docs.

[SimonSchick]

Still a little concerned with the amount and complexity of the involved changes, is there any way to split it into smaller chunks?

[javiertury]

I think it will be quite some extra work, but I will give it a try. Each commit should be able to pass tests. However some commits will have a temporary negative impact on test coverage.

The reason is, the commits will provide latent functionality that is only enabled and tested once other commits are ready. Also, some commits may temporarily introduce "duplicated" functionality

[mschipperheyn]

This is amazing. Cant wait to see this merged.

[mschipperheyn]

Looks like this PR has stalled because of the request to split into smaller chunks (which sounds like a humongous task to me). This sucks. Sequelize needs this.

[papb]

@sushantdhiman

[papb]

@javiertury This PR has conflicts, can you please rebase?

[javiertury]

@papb this feature is very time consuming. I want to get it merged, rather than to stall completely. But before investing more hours, we should first have a clear plan.

Last time it was decided to merge this little by little into a new branch, sequelize:feat/compositions. However it is stalling.

What if instead, I update and merge everything at once into a new branch(e.g. sequelize:feat/bind-params)?

The idea is to allow everyone to work simultaneously on the full implementation at the same time. If enough people commit to this plan, we can have multiple reviewers and multiple coders working simultaneously and autonomously. Reviewer will be free to create as many issues as they need, and everyone will be able to fix them.

After it becomes sufficiently polished, we can merge it into master.

[papb]

@javiertury Makes sense to me - but this kind of thing is up to @sushantdhiman 😬

By the way, thank you so much for the great work with Sequelize 👍 it wouldn't be the same without your help.

[mschipperheyn]

Anyone, anyone, Buehler, Buehler?

[mschipperheyn]

@sushantdhiman as this PR flounders, it's getting more and more conflicts with the master 👎 Really disappointed in the way this crucial change is being ignored. I remember the day when the CTO from my client saw the string interpolation on sequelize. He practically exploded and wanted to throw sequelize out then and there.

[papb]

@mschipperheyn I'm sorry... Our time is really short. Hopefully we will be able to pick up on this soon...