database/sql: missing escape functions #18478
Comments
There isn't anything here that must live in If you are interested in this, I would create a new public go package with something like this:
At that point you could suggest to drivers that they upstream your Quoter interface. You would use the Quoter interface like:
In other words, it would always be a pre-processor step to the actual DB.Query methods. I do agree MySQL is especially bad here. I can't really see this going into the std library. Have you talked to any sql driver maintainers about this or do you have any issues you can link here? |
I would argue to the contrary. Escaping primitives are needed for working securely with databases and they are inherently dependent on the SQL-server behind the connection. A package using database.sql might not even know what server it is talking to and how to properly quote and escape for that server, if it just gets passed a reference. As is, database.sql can only be used for queries known ahead of time and under the constraint, that the only dynamic entities in the query are value and never columns (think ORDER BY <user input>) or tables |
database/SQL abstracts the database interface, not the query text. You can build up the query text today with go packages that exist today. |
quoting query text is part of the database interface. |
placeholders are also abstracted. Native postgres uses $<number> not ?, oracle :<name> |
Placeholders are not abstracted. database/sql itself doesn't care at all, it just passes the query to a driver. And most drivers I know of don't abstract it away, either, requiring you to use $1 for PostgreSQL and ? for SQLite. Similarly, quoting arguments would depend on the specific DBMS. |
So proper quoting is impossible, since neither database.sql nor database.sql.driver export quoting functions and when I am sitting behind those apis my best bet is type-switching the driver to find out what I am talking to and worst case implementing that my self? That does not look solid to me. |
Maybe the current situation is not ideal, but there have been no solid proposals yet that consider all popular databases. If the driver community wants to put together a proposal, we're all ears. |
Perl/DBI does this right. Golang needs to do the same. |
@caseyallenshobe, see https://golang.org/wiki/NoMeToo If you have a proposal, please propose. |
I didn't say "me too", actually. I referenced a good implementation. Check out DBI and quote_identifier / quote_literal functions. Just like database/sql, DBI leaves platform-specific implementation up to the drivers. |
That's more helpful. For what it's worth, I lived and breathed Perl and Perl's DBI for a decade+, and that's what inspired my writing the database/sql package. I'll leave this to @kardianos, who's taken over maintenance of the package. |
Quoter defines an interface to safely quote identifiers and values. Updates golang/go#18478
I'm generally in favor of doing this. However, I'm not sure if or where in the std repo it would live. For now I've implemented a first pass https://godoc.org/github.com/golang-sql/sqlexp . |
MySQL has configuration options for quoting, so the quoting is also dependent on the connection. As a user I would expect the function to be on the sql.DB object, since I probably already have a reference to it when I am constructing queries and sql.DB already has all the information to set up the quoting. Thats also where you find them in similar packages for other languages like Perl/DB, PHP/PDO, PHP/Mysqli. |
@nefthy Good point about MySQL especially. I've updated the calls to be network friendly. You can get the driver from Here's what I'd like to see:
Perhaps it would be good to combine the escaper functions with the database name function. We'd probably want to make it easy to expand it in the future as well, which would mean changing it into a struct with methods. |
This is especially problematic for batch inserts, since database/sql doesn't currently support batch inserts, and the only way to do them today is by constructing the raw query string. |
Likewise for database-side arrays and variable-amount IN clauses. |
@hareesh-blippar Could you give an example of what you mean by batch inserts, like COPY IN or Bulk Copy? @caseyallenshobe database side arrays will be handled in a different way, by adding support for them. |
@kardianos For example, |
Batch inserts would be inserting multiple rows with a single statement, I believe. And yes, Go 1.9 promises something better, as does sqlx today. My company is still stuck on Go 1.4.3 and upgrades slowly, so it will be quite a while for us yet... Forced needs for implementing something to properly quote to overcome some shortcoming aren't the only reason either...just a good example. :) |
@hareesh-blippar I don't understand why database/sql doesn't understand multi-row value insert today. You would need to add those as parameters (which would be great for a little 10 line function to do, but other then that, it should be easy enough... Are you expecting a different API to support this? @caseyallenshobe Well, your situation would give additional weight to develop this out of tree first, then anyone can use it regardless of go version. |
@kardianos is that possible with database/sql today when the value of n in my example is variable and in the order of thousands? |
I would think so. SQL Server can only insert 1000 rows with that syntax, so built a little helper around it to break it up into statements of 1000. So I don't see why not. |
@kardianos What I meant is that you won't be able to use prepared statements to insert multiple rows today. A solution I found was http://stackoverflow.com/a/25192138/2210093, but it would be awesome if the API were to directly support bulk inserts with prepared statements similar to http://docs.oracle.com/javase/8/docs/api/java/sql/PreparedStatement.html#addBatch-- |
I don't think that will ever happen. multi-row value statements don't really need to be done in re-used baches, unless you can benchmark and tell me otherwise. But without resorting to really database specific functionality, most won't support it. Just construct the structure on the fly and add in the params and send at once. If performance is an issue, open an issue dedicated to that. |
Ignoring multi-row statements for now (off topic), what's the current state of getting a quoter into the stdlib? Is this one of those issues which might be marked as help-wanted? If I have some free time on my hands (and some experience with mentioned prior art like PHP PDO and mysqli), what are the best steps I could make to help move this along (write code, put together a proposal,...?) |
A temporary solution for strings from the Internet:
|
@akamajoris That might work for some databases in some configurations. That won't work for systems such as MS SQL Server for sure. My SQL can configure escape parameters, so that must be done with case on that system. Also, you typically need to escape two different types, identifiers and strings. |
I ran into this while looking to convert some SQL commands from dynamic to static. Painful. |
One case not yet discussed: Generating SQL without executing. For a patch generator I am in dire need of the equivalent As it stands, I have two options:
Both are not very attractive, I guess I'll go with the latter option and keep fixing it till it stops breaking? |
Even though MySQL behavior is different On the other hand, QueryString can not chose how to escape backslash without BTW, if we add QuoteString, why no FormatDate, FormatDecimal, FormatFloat, FormatGeometry, Note that QuoteIdentifier is special, because we can not use placeholder for it on some RDB including MySQL. So QuoteIdentifier will be worth enough even if @dbuteyn You can copy escaping code from |
@methane Thanks, its actually the first place I went to but I couldn't find what I was looking for at a glance. A string replace was sufficient for my needs and I didn't want to waste any more time. Obviously what I did is in no way correct and won't work for others. |
I'm new to Go so my apologies if this is extraordinarily obvious to others, but I have managed to use Sprintf to create a string for the query containing a dynamic table name for Sqlite3.
This has allowed me to query my Sqlite3 database using a dynamic table name which comes from elsewhere in my program (never as user input). Hope this helps someone though I understand this discussion is more tailored around larger DB systems. Who knows, it might work with them too! |
Reading this, I initially thought only "identifier.quoting" is non-portable. Alas, no. For example MySQL by default interprets backlash escapes inside 'strings', which violates the standard Still, I wonder — do all DBs have a set of flags that make them parse literals according to standard? |
@cben Nope, this isn't a thing. Many databases don't have such a setting, and I wouldn't recommend such a setting if there was one. Many people write SQL that make really important assumptions about quoting. The interface I would recommend would be something like this: @David00 You're on the right track and for your purposes, that might be enough. But almost inevitably, if you are talking about escaping inputs, you'll want check for values within the string as well as placing quotes around it. Also note that SQLite escaping is different then other systems. I'm honestly not sure the best direction for the quoter interface. Should there be a registry? Probably not. Could there be a per database quoter? Maybe. The problem is easy enough to frame. I'm not sure what the best solution would look like for this. |
What version of Go are you using (
go version
)?go version go1.7.4 linux/amd64
What operating system and processor architecture are you using (
go env
)?GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/nefthy/go-test/"
GORACE=""
GOROOT="/usr/lib/go"
GOTOOLDIR="/usr/lib/go/pkg/tool/linux_amd64"
CC="x86_64-pc-linux-gnu-gcc"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/home/nefthy/go-test/tmp/go-build451484149=/tmp/go-build -gno-record-gcc-switches"
CXX="x86_64-pc-linux-gnu-g++"
CGO_ENABLED="1"
What did you do?
There are situations when strings need to be escaped in queries that can not be done with placeholders. An example the following queries cannot be expressed with ? placeholders:
Using Sprintf is no option, since the identifiers need to be properly quoted. The quoting and escaping is inherently vendor specific and may even depend on configuration on a per database/connection basis (hello there MySql...).
What did you expect to see?
The driver must export Quoting which are passed along by the database/sql Api. As far as I can tell the folling functions are needed
What did you see instead?
No escaping/quoting functions
The text was updated successfully, but these errors were encountered: