dbSendQuery documentation unclear regarding when SQL request is executed vs prepared

[tomcnolan]

Hi folks, my team at Teradata is implementing a DBI Driver for the Teradata Database.

The documentation for dbSendQuery says:

The dbSendQuery() method only submits and synchronously executes the SQL query to the database engine.

The documentation for dbBind says:

For parametrized or prepared statements, the dbSendQuery() and dbSendStatement() functions can be called with statements that contain placeholders for values. The dbBind() function binds these placeholders to actual values ...

The documentation appears to be saying that the dbSendQuery method will execute a non-parameterized SQL request, and will prepare (but not execute) a parameterized SQL request.

The documentation does not indicate how a DBI Driver should distinguish between a parameterized versus non-parameterized SQL request, in order to decide how to transmit the SQL request to the backend database in the appropriate manner (prepare versus execute).

In other database APIs, such as JDBC, there are separate API calls for prepare versus execute, so expected driver behavior is obvious.

My expectation is that the dbSendQuery documentation can be augmented to explain the decision-making criteria for whether the specified SQL request is to be prepared versus executed.

In particular, the documentation should indicate whether this behavior is driver-specific. If there is a recommended way for the application to indicate to the driver whether to prepare versus execute, such as an extra named parameter for the dbSendQuery, then the documentation should mention that.

[krlmlr]

Thanks for the feedback.

RPostgres, RMariaDB, RSQLite and most likely odbc use the same code path both for parametrized and non-parametrized SQL. The SQL will be parsed for placeholders by the database library. Would that be an option for Teradata?

I remember problems with RMariaDB where some queries could not be executed as parametrized SQL, we had to work around and revert to a different code path after failure. Perhaps you could look for params = list() or params = NULL in the dbSendQuery() call?

Happy to adapt/enhance DBItest to this use case.

[tomcnolan]

The SQL will be parsed for placeholders by the database library. Would that be an option for Teradata?

We would like to avoid that approach if at all possible. We try to avoid parsing SQL in our drivers.

some queries could not be executed as parametrized SQL, we had to work around and revert to a different code path after failure.

This is also true for the Teradata Database. Some kinds of SQL requests cannot be prepared. Unfortunately, this approach won't work for us, because the same error code is returned for syntax errors and errors due to a SQL request that can't be prepared.

Perhaps you could look for params = list() or params = NULL in the dbSendQuery() call?

Yes, this approach could work. This approach would have 3 use cases:

1. dbSendQuery() with no params argument -- the driver tells the database to execute the non-parameterized SQL request.

2. dbSendQuery() with params=NULL argument -- the driver tells the database to prepare (but not execute) the parameterized SQL request. The app must subsequently execute the prepared request with dbBind()

3. dbSendQuery() with non-NULL params argument -- the driver tells the database to prepare and execute the parameterized SQL request with the specified parameter values.

[krlmlr]

Thanks. Unfortunately, the cases 1 and 2 are indistinguishable. How about:

1. dbSendQuery(params = NULL) : Prepare but wait for dbBind()

2. dbSendQuery(params = list()) : Execute right away (possibly using a different code path)

3. dbSendQuery(params = list(...)) : Prepare and bind

This would be consistent with existing implementations and, I guess, achieve what you're looking for.?

[tomcnolan]

How about:

1. dbSendQuery(params = NULL) : Prepare but wait for dbBind()
2. dbSendQuery(params = list()) : Execute right away (possibly using a different code path)
3. dbSendQuery(params = list(...)) : Prepare and bind

Yes, that should work.

The documentation does not currently talk about the expected behavior for non-parameterized SQL requests.

Can the documentation for dbSendQuery be augmented to discuss use case # 2 for immediately executing a non-parameterized SQL request?

2. dbSendQuery(params = list()) : Execute right away (possibly using a different code path)

Thanks!

[krlmlr]

Yes, we need to add test cases and implementations in the different backends. I'll consider this for the upcoming update in February.

[krlmlr]

This may be even more relevant for dbSendStatement().

[tomcnolan]

@krlmlr I have implemented dbSendQuery in the Teradata driver and have encountered a problem with our proposed solution.

Many other DBI methods such as dbExecute funnel down to calling dbSendQuery and do not explicitly specify a params argument and implicitly let params default to NULL. Those methods expect the SQL request to be executed, not just prepared without being executed.

Therefore, given how other DBI methods call dbSendQuery, the default behavior for dbSendQuery when params is not specified (meaning params is NULL) must be to execute the SQL request.

We should state generally that params must be NULL for a non-parameterized SQL request, and params must be non-NULL for a parameterized SQL request. That distinction actually makes sense, and is intuitive.

dbSendQuery behavior must be as follows:

1. dbSendQuery(params = NULL) : Execute the non-parameterized SQL request. This is the default when params is not specified.
2. dbSendQuery(params = list()) : Prepare but do not execute the parameterized SQL request. Assumes the parameterized SQL request will be executed with a subsequent call to dbBind().
3. dbSendQuery(params = list(...)) : Prepare and execute the parameterized SQL request with the specified bind parameter values.

Thanks!

[krlmlr]

A few loose thoughts:

dbExecute() should forward to dbSendStatement() -- if this is the same operation in Teradata, dbSendStatement() can just forward to dbSendQuery(). Do you really need to override dbExecute() ?

I skimmed the DBI specification, time of execution/query doesn't currently seem to be specified indeed. It can't be "too early", though: it's currently not mandatory for the caller to indicate whether the query contains placeholders. This is done with a call to dbBind() -- if that call is missing, dbFetch() or dbGetRowsAffected() work if the query doesn't contain placeholders, and throw an error otherwise.

NULL has the same semantics as "argument absent". You mentioned dbExecute() where the expectation is that the statement is executed immediately -- I'd second that, but dbExecute(params = NULL) could forward to dbSendStatement(params = list()) to achieve the desired behavior.

Agree to update the documentation once this is adopted.

[tomcnolan]

@krlmlr

dbExecute() should forward to dbSendStatement() -- if this is the same operation in Teradata,

Yes dbExecute should forward to dbSendStatement
Yes they are the same operation for Teradata.

dbSendStatement() can just forward to dbSendQuery().

Yes.

Do you really need to override dbExecute() ?

No, we do not need to override dbExecute. We do not want to override dbExecute.
We want to be able to inherit the standard dbExecute implementation in DBI.

dbExecute(params = NULL) could forward to dbSendStatement(params = list()) to achieve the desired behavior.

It could, but it does not.
If that behavior was added in a future release of DBI, then it would produce different behavior than existing old releases of DBI.

Our goal for the Teradata driver is to be compatible with DBI >= 0.8

Therefore, we need the meaning of the params argument to be consistent across all releases of DBI >= 0.8

dbExecute, dbSendStatement, and dbSendQuery behavior all must treat the params argument in the same manner:

1. params = NULL : Execute the non-parameterized SQL request. This is the default when params is not specified.

2. params = list() : Prepare but do not execute the parameterized SQL request. Assumes the parameterized SQL request is prepared with dbSendQuery and will be executed with a subsequent call to dbBind. When params = list() is specified for dbExecute or dbSendStatement the parameterized SQL request is prepared but cannot subsequently be used with dbBind, effectively acting only as a SQL syntax validation check.

3. params = list(...) : Prepare and execute the parameterized SQL request with the specified bind parameter values.

We have already implemented this behavior in the Teradata DBI driver, and it works well with DBI 0.8.

• We inherit the standard dbExecute from DBI, which forwards the params argument to dbSendStatement.
• We inherit the standard dbSendStatement from DBI, which forwards the params argument to dbSendQuery.
• We implement dbSendQuery and we interpret the params argument as listed above.

[krlmlr]

Thanks. I hear you, but the suggested treatment of params = NULL will not work because NULL has the same semantics as "argument absent". If no params are given, dbSendQuery() and dbSendStatement() still accepts a query/statement with placeholders, and will (in the general case) create a result set that waits for a dbBind() call.

Does your implementation pass the tests in DBItest?

It would be great to achieve consensus here. Can we find a way that doesn't break current implementations of RMariaDB, RPostgres and RSQLite and achieves what you are looking for?

Compatibility with DBI 0.8 will require overriding a few methods, but can be done. Alternatively, you could require a recent version of DBI.

[tomcnolan]

@krlmlr

If you are willing to change the default implementation for DBI methods, then I recommend that the distinction between preparing and executing a parameterized SQL request be made explicit by introducing two new methods: dbExecuteStatement and dbExecuteQuery.

In addition, the DBI documentation must be changed to clearly articulate the behavior of each of the DBI methods with respect to parameterized versus non-parameterized SQL.

Here is a proposal that I believe is compatible with existing DBI drivers...

(Existing method, modified documentation, modified default implementation)
dbExecute(conn, statement, ...) is intended for executing either non-parameterized or parameterized SQL requests.
dbExecute comes with a default implementation that calls dbExecuteStatement then dbGetRowsAffected followed by dbClearResult.
dbExecute has an optional params argument interpreted as follows:

• params = NULL : Execute the non-parameterized SQL request. This is the default when params is not specified.
• params = list(...) : Execute the parameterized SQL request with the specified bind parameter values.

(New method)
dbExecuteStatement(conn, statement, ...) is intended for executing either non-parameterized or parameterized SQL requests.
dbExecuteStatement comes with a default implementation that calls dbSendStatement, suitable for backends that do not distinguish between prepare and execute operations.
Backends with separate prepare versus execute operations should override dbExecuteStatement.
dbExecuteStatement has an optional params argument interpreted as follows:

• params = NULL : Execute the non-parameterized SQL request. This is the default when params is not specified.
• params = list(...) : Execute the parameterized SQL request with the specified bind parameter values.

(Existing method, modified documentation, unchanged default implementation)
dbSendStatement(conn, statement, ...) is intended for preparing or executing parameterized SQL requests.
dbSendStatement comes with a default implementation that calls dbSendQuery for backends that do not distinguish between DML statements and queries.

• params = NULL : Prepare but do not execute the parameterized SQL request. This is the default when params is not specified. The request can be executed with a subsequent call to dbBind.
• params = list(...) : Execute the parameterized SQL request with the specified bind parameter values.

(Existing method, modified documentation, modified default implementation)
dbGetQuery(conn, statement, ...) is intended for executing either non-parameterized or parameterized SQL requests.
dbGetQuery comes with a default implementation that calls dbExecuteQuery then dbFetch followed by dbClearResult.
dbGetQuery has an optional params argument interpreted as follows:

• params = NULL : Execute the non-parameterized SQL request. This is the default when params is not specified.
• params = list(...) : Execute the parameterized SQL request with the specified bind parameter values.

(New method)
dbExecuteQuery(conn, statement, ...) is intended for executing either non-parameterized or parameterized SQL requests.
dbExecuteQuery comes with a default implementation that calls dbSendQuery, suitable for backends that do not distinguish between prepare and execute operations.
Back ends with separate prepare versus execute operations should override dbExecuteQuery.
dbExecuteQuery has an optional params argument interpreted as follows:

• params = NULL : Execute the non-parameterized SQL request. This is the default when params is not specified.
• params = list(...) : Execute the parameterized SQL request with the specified bind parameter values.

(Existing method, modified documentation)
dbSendQuery(conn, statement, ...) is intended for preparing or executing parameterized SQL requests.

• params = NULL : Prepare but do not execute the parameterized SQL request. This is the default when params is not specified. The request can be executed with a subsequent call to dbBind.
• params = list(...) : Execute the parameterized SQL request with the specified bind parameter values.

[krlmlr]

Thanks. I was contemplating new interface methods, it's a bit of work but seems worth it.

I'm not sure I follow your proposal entirely, I'll draft mine and we see if we can meet halfway.

I'm proposing two new methods, dbSendQueryNow() and dbSendStatementNow().

• Can accept parameters, but guarantee immediate execution. Failure if not enough parameters supplied.
• Result set object may accept new/updated parameters through dbBind(), but isn't required to
• Default implementation calls dbSendXXX() + dbBind() (or just dbSendXXX(params = ...) )
• Default dbGetQuery() and dbExecute() are modified to call dbSendXXXNow()

Is the dbExecuteXXX() from your proposal the same as dbSendXXXNow() from mine? I'm confused by the naming scheme and the similarity to the existing dbExecute().

[tomcnolan]

@krlmlr Yes, your proposed methods sound similar to mine.
Your proposed dbSendQueryNow sounds like my proposed dbExecuteQuery.
Your proposed dbSendStatementNow sounds like my proposed dbExecuteStatement.