Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

Prepared statement support #289

Open
wants to merge 94 commits into from
@nyaxt

Using prepared statement is an effective way to prevent injection attacks and improve performance.
I tried to continue the work on the 'stmt' branch.

Now, I think its basically done but for two caveats:
1. Query results are cached expect for streaming queries.
2. There are no syntax to pass query_options when PreparedStatement#execute. I don't have an idea for nice syntax.

@leejarvis

Sweeeeet. Any idea when this might get merged into master?

Collaborator

After it fully functions. ;-)

I don't want to merge this in to master until I have it working better.

@brianmario
Owner

Heh, just noticed I meant to say "split out of Mysql2::Client"...

and others added some commits
@brianmario Merge branch 'master' into stmt ee4fd98
@brianmario Merge branch 'master' into stmt
* master:
  remove Sequel adapter as it's now in Sequel core :)
  move -Wextra to development flags area
  update AR adapter to reflect timezone setting update
  application_timezone is allowed to be nil
  default application_timezone to nil
  sync up with sequel adapter from my Sequel fork until it's officially merged in
  convert :timezone option into two new ones :database_timezone - the timezone (:utc or :local) Mysql2 will assume time/datetime fields are stored in the db. This modifies what initial timezone your Time objects will be in when creating them from libmysql in C and :application_timezone - the timezone (:utc or :local) you'd finally like the Time objects converted to before you get them
  can't call literal here because it'll try to join it's own thread
  Mysql2::Client uses the :username key, set it to :user if that was used instead
  heh
  fix typo in comment
  major refactor of Sequel adapter - it's now green in Sequel
  add :cast_booleans option for automatically casting tinyint(1) fields into true/false for ruby
  move most previously global symbols to static to prevent conflicts (thanks for catching this Eric)
  respect :symbolize_keys option for Mysql2::Result#fields if it's called before the first row is built
  initialize @active early on to prevent warnings later
  let's try that again - libmysql only allows one query be sent at a time per connection, bail early if that's attempted
  Revert "libmysql only allows one query be sent at a time per connection, bail early if that's attempted"
  libmysql only allows one query be sent at a time per connection, bail early if that's attempted
  no need to carry over options twice as we're already doing it up in rb_mysql_client_async_result
8aa1e9b
@brianmario bring over latest from master 731b456
@tenderlove tenderlove xfree or die hard a684efd
@brianmario Merge branch 'master' into stmt 9a84f88
@brianmario Merge branch 'master' into stmt 39ea9c6
@brianmario Merge branch 'master' into stmt 39a28c4
@brianmario Merge branch 'master' into stmt
* master:
  Detach before executing callbacks.
  make sure we don't hit a race condition if this EM spec is taking longer to run than normal
  was in a hurry earlier
  whoops, lost this line in a previous patch
  Dry windows configuration options
  Inject 1.8/1.9 pure-ruby entry point during xcompile
  Use MySQL 5.1.51 now from available mirror
7b6d210
@brianmario Merge branch 'master' into stmt
* master:
  avoid potential race-condition with closing a connection
  add option for setting the wait_timeout in the AR adapter (this can be done in database.yml)
  add some more defaults to the connect flags
  add connect_flags to default options and add REMEMBER_OPTIONS to that list. fix NUM2INT to be NUM2ULONG as it should be for flags
  free the client after close if we can
  forgot to remove this
  get rid of double-pointer casting
  a couple of minor updates to connection management with some specs
  check for error from mysql_affected_rows call
  change connection check symantecs
ddcd106
@brianmario Merge branch 'master' into stmt 4a5fcba
@brianmario merging in latest from master 6c4fd8a
@brianmario Merge branch 'master' into stmt
* master:
  only use wait_timeout if it's a Fixnum
  update gemspec from file updates
  no need to invalidate the FD now that we're only modifying it once, in one place
  no need to check if the FD is >= 0 now that we centralized the close/free logic. Once closed, none of that code will run again
  wording fix in readme
f6d7e87
@brianmario Merge branch 'master' into stmt 04bd4de
@brianmario bring over latest from master bdfd40b
@brianmario Merge branch 'master' into stmt 95ea994
@brianmario simplify prepared statement creation into Mysql2::Client#prepare 6428163
@brianmario Mysql2::Statement includes Enumerable 0d6c472
@brianmario Merge branch 'master' into stmt ef979d1
@brianmario Merge branch 'master' into stmt f0df399
@brianmario Merge branch 'master' into stmt df7905c
@brianmario don't warn on unused parameter e655699
@brianmario remove old create_statement function 2648b0f
@brianmario make sure we require our ruby implementation of Mysql2::Statement f1f8df1
@brianmario make sure we set STMT_ATTR_UPDATE_MAX_LENGTH when preparing the state…
…ment so we can get at the lengths later
b6217fb
@brianmario initial fleshing out of bind variable support and casting of ruby val…
…ues from result value types
ca4768a
@brianmario Merge branch 'master' into stmt 247a9a6
@brianmario Merge branch 'master' into stmt 542d25a
@brianmario Merge branch 'master' into stmt cfca80e
@brianmario Merge branch 'master' into stmt 60f5197
@brianmario Merge branch 'master' into stmt 9b0b1d6
@brianmario Merge branch 'master' into stmt 6c422c4
@brianmario Merge branch 'master' into stmt 1b06e4f
@brianmario Merge branch 'master' into stmt 450e8c2
@brianmario Merge branch 'master' into stmt 9c45e2c
@brianmario latest from master 8b2f327
@brianmario Merge branch 'master' into stmt c057f07
@brianmario Merge branch 'master' into stmt 1ce1a3a
@brianmario Merge branch 'master' into stmt cc31f58
@brianmario Merge branch '0.2.x' into stmt adaf0c3
@brianmario Merge branch 'master' into stmt 9e2e0c7
@brianmario Merge branch 'master' into stmt c87b122
@brianmario Merge branch 'master' into stmt 8b2ac86
@brianmario Merge branch 'master' into stmt e2d560e
@brianmario Merge branch 'master' into stmt c2ed15f
@brianmario Merge branch 'master' into stmt 42ba6ae
@brianmario don't use ruby's heap 278a484
@brianmario Merge branch 'master' into stmt 7c8f42b
@brianmario Merge branch 'master' into stmt ea18403
@brianmario Merge branch 'master' into stmt b1ed8bb
@nyaxt nyaxt Merge branch 'master' of https://github.com/nyaxt/mysql2 into stmt b420f80
@nyaxt nyaxt test pass eb6d3e9
@nyaxt nyaxt rename prepare_statement func 730e3c8
@nyaxt nyaxt MYSQL_STMT is now wrapped as mysql_stmt_wrapper. handle field name en…
…coding.
984df31
@nyaxt nyaxt utf8 fieldnames 50a7054
@nyaxt nyaxt Mysql2::Result modified to support prepstmt results too. rb_mysql_res…
…ult_each_stmt WIP
397158f
@nyaxt nyaxt num. result rows is working. rb_mysql_result_each_stmt WIP eb6e7f7
@nyaxt nyaxt test pass? 9d48f87
@nyaxt nyaxt remove Statement#each method 7b61ef4
@nyaxt nyaxt RSTRING_PTR free bug temporary workaround 45662c2
@nyaxt nyaxt added failing test: query with param in different encoding 10e5c7d
@nyaxt nyaxt query params encoded correctly 7d4ad12
@nyaxt nyaxt reuse result bind buffers 5009dca
@nyaxt nyaxt set active_thread for stmt execute 8b585ee
@nyaxt nyaxt check is_stream 6aef18c
@nyaxt nyaxt no gvl mysql_stmt_store_result 1af66cf
@nyaxt nyaxt raise stmt errors via rb_raise_mysql2_stmt_error 05df2bd
@nyaxt nyaxt no gvl mysql_stmt_fetch 601442f
@nyaxt nyaxt streaming works! 39a171c
@nyaxt nyaxt remove unused code c92fcac
@nyaxt nyaxt ported row data type mapping test from result_spec.rb / test pass 4fbb04c
@nyaxt nyaxt fixed some test which still used client#query / castBool impl. 5552b90
@nyaxt nyaxt added failing test: should keep its result after other query b4e9bb7
@nyaxt nyaxt test pass 89eea95
@nyaxt nyaxt port #each/#fields test from result_spec.rb / all test pass cf2bf20
@nyaxt nyaxt Merge branch brianmario/mysql2/master into stmt e581cf0
@nyaxt nyaxt whitespace f1f90d7
@travisbot

This pull request fails (merged f1f90d7 into 2ca69b1).

@travisbot

This pull request fails (merged 20d72bb into 2ca69b1).

@travisbot

This pull request fails (merged 8bd9725 into 2ca69b1).

@nyaxt nyaxt closed this
@nyaxt

sorry, it seems to need more work on 1.8.7 support. I would like to request again when its done.

@nyaxt nyaxt reopened this
@nyaxt

should work on 1.8.7

@travisbot

This pull request passes (merged bb32c56 into 2ca69b1).

@travisbot

This pull request passes (merged f481190 into 2ca69b1).

@brianmario
Owner

Would you mind giving us a brief API breakdown of what's introduced here?

At first blush it looks like prepare takes s single SQL string argument then execute is where you pass in the query parameters and a Mysql2::Result is returned.
What do you think about allowing execute to take an optional hash of query options as it's last parameter as well?
Also I saw in the tests that the current implementation will ignore query options passed to Mysql2::Result#each after execute is called because of the rows being cached. I think I ran into that issue as well back when I'd last worked on this. I'd definitely like to figure out a solution where we could lazily build the rows via Mysql2::Result#each. Maybe we need to set a flag telling the result class what kind of result it is (regular vs prepared statement)?

@nyaxt

APIs introduced:

Mysql2::Client#prepare(sql) => Mysql2::Statement

This creates and prepares a server-side prepared statement from a single sql given in sql. sql may contain placeholders ? to be filled on Statement#execute

Mysql2::Statement#execute(*params) # => Mysql2::Result

This binds parameters given in params and execute the prepared statement. It returns a Mysql2::Result which can be used to retrieve results just like Mysql2::Client#query.

Mysql2::Statement#param_count # => Integer

This returns number of placeholders (?) in the prepared statement.

Mysql2::Statement#field_count # => Integer

This returns the number of fields of the result set the prepared statement will create when executed.

Mysql2::Statement#fields # => Array[String]

This returns the name of fields of the result set the prepared statement will create when executed.

@nyaxt

What do you think about allowing execute to take an optional hash of query options as it's last parameter as well?

Like this? stmt.execute("param1", "param2", "param3", :as => :array)
Maybe we can see in Statement#execute if argv[argc-1] is a Hash and take that as query options.

Also I saw in the tests that the current implementation will ignore query options passed to Mysql2::Result#each after execute is called because of the rows being cached. I think I ran into that issue as well back when I'd last worked on this. I'd definitely like to figure out a solution where we could lazily build the rows via Mysql2::Result#each. Maybe we need to set a flag telling the result class what kind of result it is (regular vs prepared statement)?

Yes. The current implementation forces to create row cache on Statement#execute by calling Result#each internally. This disallows specifying options in Mysql2::Result#each.

The underlying problem is a difference in mysql_fetch_row and mysql_stmt_fetch. mysql_fetch_row takes MYSQL_RES* and this allows the result to be acquired lazily if mysql_store_result has been called. However, mysql_stmt_fetch only takes MYSQL_STMT as an argument and provides no way to specify which invocation of #execute we want to create result set from.

I was thinking of binding all results to a temporary buffer when #execute is invoked, and delay conversion to Ruby types until Result#each, but this will really make the code complicated in rb_mysql_result_stmt_fetch_row.

@sodabrew
Collaborator

If do a version 0.4.0, we should include this branch! What's the status?

@brianmario
Owner
@GreySyntax

Are there any updates on this being merged? If getting this merged means chipping i'm willing to help just point me in the direction of what needs working on.

@sodabrew
Collaborator

I think we're basically waiting to do a release with current material in master, which is somewhat blocked on #321, and then looking at this work for a next major version. One thing that would be helpful at your end is rebasing the commits up to current master (rather than merging changes from master into your branch, rebasing makes your commits sit at the top of the change history) to make it easier to review. Thanks!

@sodabrew
Collaborator

Just did a rebase of the stmt branch. If this is the best continuation of the work, I'll pull it into my rebase. See #405

@nyaxt

Hello. I am happy to continue the work if merging became realistic. I would update the API interfaces if necessary.

As my current employer has a special policy for contributing to OSS, I'm contacting them for approval now.

@tagrudev

Any progresss on that ?

@johncant johncant referenced this pull request
Closed

Stmt rebase rebase rebase #477

@johncant

#405 #477 #476 @nyaxt @brianmario @sodabrew @tenderlove

I've started rebasing @nyaxt 's branch onto @sodabrew 's most recent rebase. It is proving to be really hard, but basically I'm going to try and match the test successes/failures at every stage with @nyaxt 's branch, or the failures will get out of hand. They kind of already have, but hey this is rebasing and I can go back and sort it out....

Here is my WIP rebase: https://github.com/johncant/mysql2/commits/nyaxt_rebase1

Please let me know if I've missed anything major.

@justincase
>> stmt = client.prepare("SELECT * FROM messages WHERE uuid = ?")
=> #<Mysql2::Statement:0x007fc082086728>
>> stmt.execute("7bfbc83b-0007-4572-90c7-0d7151699bc8")
ruby(27539,0x7fff7a1c8960) malloc: *** error for object 0x7faf0113cac0: pointer being freed was not allocated
*** set a breakpoint in malloc_error_break to debug
Abort trap: 6

Occurs on all branches I've tried so far.

@johncant

@justincase I think I know the cause of the above issue ^. I fixed it, and you can see the test and fix in my closed pull requests, but I didn't use the correct base branch. Looking at @nyaxt's code, it looks like he's fixed it, but I haven't made sure yet. I'm currently attempting to rebase his branch https://github.com/johncant/mysql2/tree/nyaxt_rebase1 - any help would be appreciated.

Next time I get a chance to look at it will most likely be Sunday, and I plan on rebasing my rebase again to replicate @nyaxt's TDD set of passing/failing specs at each commit so I only have to debug incrementally and in obvious places.

TL;DR: Please no-one use my branch as a base branch (although feel free to rebase it).

Sounds like we need a meta git to deal with source code history history!

@JoshMcKin

Pretty stale topic, but could really use this feature.

@simi

I'm not sure about status. Can anyone describe? I would like to finish this on my own. But I don't know what and which branch(es) to use.

@sodabrew
Collaborator

This is the most recent rebase PR: #405 but it is incomplete.

It's been on my radar to revive this branch and begin a push towards a 0.4.0 series.

@simi

Ahh, I understand now. Thanks for info. Ping if you'll need help.

@justincase

One of @johncant 's branches was working alright for me in testing. All the rebasing and different PRs are a tad disorienting though.

@sodabrew
Collaborator

Agreed, it's going to take a day of sorting out which commits are where and then getting them into a consistent rebase state to current master. Help appreciated!

@justincase

Hi, @sodabrew. Have any new thoughts on this? @johncant's stmt-johncant branch (johncant@452799d) passes all specs with minimal issues but doesn't include @nyaxt commits yet.

@justincase

I've created a new branch from scratch that contains everything up to 50a7054 + fixes and tests from @johncant.

https://github.com/justincase/mysql2/commits/prepared_mysql2

@sodabrew
Collaborator

@justincase This is awesome! I need some time to review the branches. I probably won't be able to look at this until next week. But, let me suggest the following: If you put together a PR that definitively supersedes all prior prepared statement PRs, I'll focus on review and merge of your PR.

@johncant

Nice one! I was rebasing @nyaxt 's branch commit by commit, trying to make the same tests pass/fail as they did in @nyaxt 's own branch, but gave up a few commits in and lost interest due to a tricky error and changing to postgresql unrelatedly. @nyaxt got his prepared statement support working so long ago that the code has changed quite a bit.

@justincase
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on Jul 9, 2010
  1. @tenderlove

    we can prepare statements!

    tenderlove authored
  2. @tenderlove
  3. @tenderlove
  4. @tenderlove
  5. @tenderlove
  6. @tenderlove
  7. @tenderlove

    adding a gdb rake task

    tenderlove authored
  8. @tenderlove

    adding Stmt#fields

    tenderlove authored
  9. @tenderlove
  10. @tenderlove
Commits on Jul 12, 2010
  1. @tenderlove

    Merge branch 'master' into stmt

    tenderlove authored
    * master:
      check for and support field-level encodings
      on second thought, we should make sure we were given a string earlier on
      no need to Check_Type in these spots since we're using StringValuePtr as well
Commits on Jul 30, 2010
Commits on Aug 2, 2010
Commits on Aug 6, 2010
  1. Merge branch 'master' into stmt

    authored
    * master:
      remove Sequel adapter as it's now in Sequel core :)
      move -Wextra to development flags area
      update AR adapter to reflect timezone setting update
      application_timezone is allowed to be nil
      default application_timezone to nil
      sync up with sequel adapter from my Sequel fork until it's officially merged in
      convert :timezone option into two new ones :database_timezone - the timezone (:utc or :local) Mysql2 will assume time/datetime fields are stored in the db. This modifies what initial timezone your Time objects will be in when creating them from libmysql in C and :application_timezone - the timezone (:utc or :local) you'd finally like the Time objects converted to before you get them
      can't call literal here because it'll try to join it's own thread
      Mysql2::Client uses the :username key, set it to :user if that was used instead
      heh
      fix typo in comment
      major refactor of Sequel adapter - it's now green in Sequel
      add :cast_booleans option for automatically casting tinyint(1) fields into true/false for ruby
      move most previously global symbols to static to prevent conflicts (thanks for catching this Eric)
      respect :symbolize_keys option for Mysql2::Result#fields if it's called before the first row is built
      initialize @active early on to prevent warnings later
      let's try that again - libmysql only allows one query be sent at a time per connection, bail early if that's attempted
      Revert "libmysql only allows one query be sent at a time per connection, bail early if that's attempted"
      libmysql only allows one query be sent at a time per connection, bail early if that's attempted
      no need to carry over options twice as we're already doing it up in rb_mysql_client_async_result
Commits on Aug 16, 2010
Commits on Aug 20, 2010
  1. @tenderlove

    xfree or die hard

    tenderlove authored
Commits on Sep 24, 2010
Commits on Oct 5, 2010
  1. Merge branch 'master' into stmt

    authored
    * master:
      Detach before executing callbacks.
      make sure we don't hit a race condition if this EM spec is taking longer to run than normal
      was in a hurry earlier
      whoops, lost this line in a previous patch
      Dry windows configuration options
      Inject 1.8/1.9 pure-ruby entry point during xcompile
      Use MySQL 5.1.51 now from available mirror
Commits on Oct 15, 2010
  1. Merge branch 'master' into stmt

    authored
    * master:
      avoid potential race-condition with closing a connection
      add option for setting the wait_timeout in the AR adapter (this can be done in database.yml)
      add some more defaults to the connect flags
      add connect_flags to default options and add REMEMBER_OPTIONS to that list. fix NUM2INT to be NUM2ULONG as it should be for flags
      free the client after close if we can
      forgot to remove this
      get rid of double-pointer casting
      a couple of minor updates to connection management with some specs
      check for error from mysql_affected_rows call
      change connection check symantecs
Commits on Oct 16, 2010
Commits on Oct 17, 2010
Commits on Oct 18, 2010
  1. Merge branch 'master' into stmt

    authored
    * master:
      only use wait_timeout if it's a Fixnum
      update gemspec from file updates
      no need to invalidate the FD now that we're only modifying it once, in one place
      no need to check if the FD is >= 0 now that we centralized the close/free logic. Once closed, none of that code will run again
      wording fix in readme
Commits on Feb 12, 2011
Commits on Apr 15, 2011
Commits on Apr 16, 2011
Commits on Apr 27, 2011
Commits on May 5, 2011
  1. make sure we set STMT_ATTR_UPDATE_MAX_LENGTH when preparing the state…

    authored
    …ment so we can get at the lengths later
Commits on May 6, 2011
Commits on Jun 15, 2011
Commits on Jun 16, 2011
  1. latest from master

    authored
Commits on Jun 17, 2011
Commits on Aug 24, 2011
Commits on Oct 13, 2011
  1. don't use ruby's heap

    authored
Commits on Oct 31, 2011
Commits on Nov 15, 2011
Commits on Dec 6, 2011
Commits on Jul 28, 2012
  1. @nyaxt
  2. @nyaxt

    test pass

    nyaxt authored
Commits on Jul 29, 2012
  1. @nyaxt

    rename prepare_statement func

    nyaxt authored
  2. @nyaxt
  3. @nyaxt

    utf8 fieldnames

    nyaxt authored
Commits on Aug 5, 2012
  1. @nyaxt
Commits on Aug 8, 2012
  1. @nyaxt
  2. @nyaxt

    test pass?

    nyaxt authored
  3. @nyaxt

    remove Statement#each method

    nyaxt authored
  4. @nyaxt
  5. @nyaxt
  6. @nyaxt

    query params encoded correctly

    nyaxt authored
Commits on Aug 9, 2012
  1. @nyaxt

    reuse result bind buffers

    nyaxt authored
Commits on Aug 10, 2012
  1. @nyaxt
  2. @nyaxt

    check is_stream

    nyaxt authored
  3. @nyaxt

    no gvl mysql_stmt_store_result

    nyaxt authored
  4. @nyaxt
  5. @nyaxt

    no gvl mysql_stmt_fetch

    nyaxt authored
  6. @nyaxt

    streaming works!

    nyaxt authored
  7. @nyaxt

    remove unused code

    nyaxt authored
  8. @nyaxt
  9. @nyaxt
  10. @nyaxt
  11. @nyaxt

    test pass

    nyaxt authored
  12. @nyaxt
  13. @nyaxt
  14. @nyaxt

    whitespace

    nyaxt authored
  15. @nyaxt

    revert changes to extconf.rb

    nyaxt authored
  16. @nyaxt

    support ruby w/o Encoding

    nyaxt authored
  17. @nyaxt
  18. @nyaxt
  19. @nyaxt
  20. @nyaxt

    fix typo: rb_warn

    nyaxt authored
Something went wrong with that request. Please try again.