New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jdbc mode #599

Open
wants to merge 65 commits into
base: master
from

Conversation

Projects
None yet
5 participants
@iaddict

iaddict commented May 31, 2017

  • Drop strict gem dependency on tiny_tds.
  • Add gem dependency for tests on new (un-published) gem sqljdbc4 which bundles the MS jdbc driver.
  • Implement a thin ruby jdbc wrapper that handles queries, stored procedures and type conversion to ruby.
  • Use jdbc wrapper methods where required in a conditional manner.

This pull request resembles the discussion from #579.

@metaskills

Thanks! I got plenty of time now so I should be much more responsive. Can you rebase to latest master and take a look at my other notes? Looking forward to this in 5.1.

when :jdbc
result = begin
@connection.fetch(sql).all
rescue => e

This comment has been minimized.

@metaskills

metaskills Jun 16, 2017

Contributor

This rescue feels risky. Why would an empty result set to fetch raise an exception? Is it due to the .all method call? If so, can we avoid the rescue trap?

@metaskills

metaskills Jun 16, 2017

Contributor

This rescue feels risky. Why would an empty result set to fetch raise an exception? Is it due to the .all method call? If so, can we avoid the rescue trap?

This comment has been minimized.

@iaddict

iaddict Jun 22, 2017

The problem is, that I can't detect if the sql statement returns a result set or not.
#fetch uses jdbcs executeQuery (https://docs.oracle.com/javase/7/docs/api/java/sql/Statement.html#executeQuery(java.lang.String)) which throws an exception when the statement does not yield a result (like ddl statements, in which case the execute(sql) method should be used).

The statement is executed none the less.

@iaddict

iaddict Jun 22, 2017

The problem is, that I can't detect if the sql statement returns a result set or not.
#fetch uses jdbcs executeQuery (https://docs.oracle.com/javase/7/docs/api/java/sql/Statement.html#executeQuery(java.lang.String)) which throws an exception when the statement does not yield a result (like ddl statements, in which case the execute(sql) method should be used).

The statement is executed none the less.

results =
begin
handle.all(query_options)
rescue => e

This comment has been minimized.

@metaskills

metaskills Jun 16, 2017

Contributor

Same trap on rescue, would you mind letting me know why no results returns an error. Can that be avoided? I'd really like to know why the JDBC connection wold do that.

@metaskills

metaskills Jun 16, 2017

Contributor

Same trap on rescue, would you mind letting me know why no results returns an error. Can that be avoided? I'd really like to know why the JDBC connection wold do that.

This comment has been minimized.

@iaddict

iaddict Jun 22, 2017

See above.

@iaddict
@@ -20,11 +20,11 @@ def current_database
end
def charset
select_value "SELECT DATABASEPROPERTYEX(DB_NAME(), 'SqlCharSetName')"
select_value "SELECT CAST(DATABASEPROPERTYEX(DB_NAME(), 'SqlCharSetName') AS NVARCHAR(128))"

This comment has been minimized.

@metaskills

metaskills Jun 16, 2017

Contributor

Nice change. But I think we need to update sql_counter_sqlserver.rb with the ignored SQL format too right?

@metaskills

metaskills Jun 16, 2017

Contributor

Nice change. But I think we need to update sql_counter_sqlserver.rb with the ignored SQL format too right?

This comment has been minimized.

@iaddict

iaddict Jun 22, 2017

Not sure why this needs to be done. Isn't this statement execute in isolation all the time and never with count?

@iaddict

iaddict Jun 22, 2017

Not sure why this needs to be done. Isn't this statement execute in isolation all the time and never with count?

@@ -194,7 +196,7 @@ def reset!
def tables_with_referential_integrity
schemas_and_tables = select_rows <<-SQL.strip_heredoc
SELECT s.name, o.name
SELECT s.name sc, o.name tab

This comment has been minimized.

@metaskills

metaskills Jun 16, 2017

Contributor

Another nice change. Did JDBC throw a warning here or did you notice the ambiguity?

@metaskills

metaskills Jun 16, 2017

Contributor

Another nice change. Did JDBC throw a warning here or did you notice the ambiguity?

This comment has been minimized.

@iaddict

iaddict Jun 22, 2017

The method handle.all which is used to retrieve the result within DatabaseStatements#handle_to_names_and_values_jdbc returns a hash with {column: value}.
Without the aliases there would only be one column.

@iaddict

iaddict Jun 22, 2017

The method handle.all which is used to retrieve the result within DatabaseStatements#handle_to_names_and_values_jdbc returns a hash with {column: value}.
Without the aliases there would only be one column.

Show outdated Hide outdated lib/activerecord_sqlserver_adapter/jdbc.rb
Show outdated Hide outdated lib/activerecord_sqlserver_adapter/jdbc/dataset.rb
Show outdated Hide outdated lib/activerecord_sqlserver_adapter/jdbc/dataset.rb
Show outdated Hide outdated lib/activerecord_sqlserver_adapter/jdbc/type_converter.rb

Thomas Steinhausen and others added some commits Mar 22, 2017

Make adapter work on jruby with sequel as raw_connection provider.
Sequel is used as the connection provider over jdbc.
Set schema_reflection for sequel on connect.
This has to be set on connect, because sequel only handles this on connect.
Require tiny_sql/java
This file then requires sequel and the sqljdbc jar.
Gracefully handle empty result sets
Sequel / Jdbc barks if no result set is returned.
Fix stored procedure execution for sequel mode
At least the code now returns multiple rows returned by a procedure.
Still no multiple results supported.
Ensure ftom and to_table are strings
Sequel returns symbols which yield exceptions.
Skip some tests that do not work as expected with sequel for now
Some of them make no sense with mode sequel, others need to be looked at.
Do not test datetimeoffset with mode sequel
Sequel does not cast this type to a ruby type, which lets several tests fail.
Especially the migration tests fail.
Handle empty sequel results.
JDBC raises when a stored procedure is called and no result is returned.
We now rescue exactly this and return an empty array instead.

iaddict added some commits Jun 22, 2017

@iaddict

This comment has been minimized.

Show comment
Hide comment
@iaddict

iaddict Jun 22, 2017

The rebase is done. The test suite is not yet totaly green:

$ ONLY_SQLSERVER=1 jruby -S bundle exec rake test:jdbc
#=> 218 runs, 1349 assertions, 1 failures, 4 errors, 5 skips

$ jruby -S bundle exec rake test:jdbc
#=> 5012 runs, 14857 assertions, 2 failures, 16 errors, 10 skips

Especially the uuid identity columns do not work for insert. It looks like the 'select identity' statements do not yield the generated ids.

Do you have a hint how to get these? I didn't find any code the handles this for tiny_tds which confuses me because this seems to be an issue with SQL-Server in general.

iaddict commented Jun 22, 2017

The rebase is done. The test suite is not yet totaly green:

$ ONLY_SQLSERVER=1 jruby -S bundle exec rake test:jdbc
#=> 218 runs, 1349 assertions, 1 failures, 4 errors, 5 skips

$ jruby -S bundle exec rake test:jdbc
#=> 5012 runs, 14857 assertions, 2 failures, 16 errors, 10 skips

Especially the uuid identity columns do not work for insert. It looks like the 'select identity' statements do not yield the generated ids.

Do you have a hint how to get these? I didn't find any code the handles this for tiny_tds which confuses me because this seems to be an issue with SQL-Server in general.

iaddict added some commits Jun 22, 2017

Remove unused code
Especially remove Jdbc#synchronize because AR already does connection pooling and threadsafety stuff.
Handle varbinary DML
Guess from string encoding if this should by handled as a varbinary.
Fixes tests for varbinary column inserts.
@iaddict

This comment has been minimized.

Show comment
Hide comment
@iaddict

iaddict Jun 27, 2017

$ONLY_SQLSERVER=1 jruby -S bundle exec rake test:jdbc

The tests now look even better. Two tests still fail, but the errors stem from the same issue.
I have a problem to get the last inserted id for uuid columns. @metaskills Do you have a hint how to get the generated uuid on insert?

The 'select identity' statements do not yield the generated ids.

$ ONLY_SQLSERVER=1 jruby -S bundle exec rake test:jdbc

  1) Error:
SQLServerUuidTest#test_0003_can create other uuid column on reload:
ActiveRecord::RecordNotFound: Couldn't find SSTestUuid without an ID
    .../activerecord/lib/active_record/relation/finder_methods.rb:448:in `find_with_ids'
    .../activerecord/lib/active_record/relation/finder_methods.rb:66:in `find'
    .../activerecord/lib/active_record/querying.rb:3:in `find'
    .../activerecord/lib/active_record/core.rb:171:in `find'
    .../activerecord/lib/active_record/persistence.rb:462:in `block in reload'
    .../activerecord/lib/active_record/scoping/default.rb:35:in `block in unscoped'
    .../activerecord/lib/active_record/relation.rb:336:in `scoping'
    .../activerecord/lib/active_record/scoping/default.rb:35:in `unscoped'
    .../activerecord/lib/active_record/persistence.rb:462:in `reload'
    .../activerecord/lib/active_record/attribute_methods/dirty.rb:50:in `reload'
    .../activerecord/lib/active_record/associations.rb:283:in `reload'
    .../activerecord/lib/active_record/autosave_association.rb:236:in `reload'
    .../activerecord/lib/active_record/aggregations.rb:13:in `reload'
    .../activerecord-sqlserver-adapter/test/cases/uuid_test_sqlserver.rb:21:in `block in test_0003_can create other uuid column on reload'


  2) Failure:
SQLServerUuidTest#test_0002_can create with a new pk [.../activerecord-sqlserver-adapter/test/cases/uuid_test_sqlserver.rb:15]:
Expected nil to be present?.

218 runs, 1352 assertions, 1 failures, 1 errors, 5 skips

iaddict commented Jun 27, 2017

$ONLY_SQLSERVER=1 jruby -S bundle exec rake test:jdbc

The tests now look even better. Two tests still fail, but the errors stem from the same issue.
I have a problem to get the last inserted id for uuid columns. @metaskills Do you have a hint how to get the generated uuid on insert?

The 'select identity' statements do not yield the generated ids.

$ ONLY_SQLSERVER=1 jruby -S bundle exec rake test:jdbc

  1) Error:
SQLServerUuidTest#test_0003_can create other uuid column on reload:
ActiveRecord::RecordNotFound: Couldn't find SSTestUuid without an ID
    .../activerecord/lib/active_record/relation/finder_methods.rb:448:in `find_with_ids'
    .../activerecord/lib/active_record/relation/finder_methods.rb:66:in `find'
    .../activerecord/lib/active_record/querying.rb:3:in `find'
    .../activerecord/lib/active_record/core.rb:171:in `find'
    .../activerecord/lib/active_record/persistence.rb:462:in `block in reload'
    .../activerecord/lib/active_record/scoping/default.rb:35:in `block in unscoped'
    .../activerecord/lib/active_record/relation.rb:336:in `scoping'
    .../activerecord/lib/active_record/scoping/default.rb:35:in `unscoped'
    .../activerecord/lib/active_record/persistence.rb:462:in `reload'
    .../activerecord/lib/active_record/attribute_methods/dirty.rb:50:in `reload'
    .../activerecord/lib/active_record/associations.rb:283:in `reload'
    .../activerecord/lib/active_record/autosave_association.rb:236:in `reload'
    .../activerecord/lib/active_record/aggregations.rb:13:in `reload'
    .../activerecord-sqlserver-adapter/test/cases/uuid_test_sqlserver.rb:21:in `block in test_0003_can create other uuid column on reload'


  2) Failure:
SQLServerUuidTest#test_0002_can create with a new pk [.../activerecord-sqlserver-adapter/test/cases/uuid_test_sqlserver.rb:15]:
Expected nil to be present?.

218 runs, 1352 assertions, 1 failures, 1 errors, 5 skips
@iaddict

This comment has been minimized.

Show comment
Hide comment
@iaddict

iaddict Jun 29, 2017

Ok. I fixed the uuid issue. The ONLY_SQLSERVER tests are now all green!

$ ONLY_SQLSERVER=1 jruby -S bundle exec rake test:jdbc 

Finished in 36.358056s, 5.9959 runs/s, 37.2957 assertions/s.

218 runs, 1356 assertions, 0 failures, 0 errors, 5 skips

iaddict commented Jun 29, 2017

Ok. I fixed the uuid issue. The ONLY_SQLSERVER tests are now all green!

$ ONLY_SQLSERVER=1 jruby -S bundle exec rake test:jdbc 

Finished in 36.358056s, 5.9959 runs/s, 37.2957 assertions/s.

218 runs, 1356 assertions, 0 failures, 0 errors, 5 skips
@iaddict

This comment has been minimized.

Show comment
Hide comment
@iaddict

iaddict Jul 3, 2017

Hi. I worked some more on the jdbc mode. Travisci now also runs with jruby.
Only one AR test case is still failing, the rest is green. 🎉

$ jruby -S bundle exec rake test

  1) Failure:
TransactionTest#test_rollback_when_thread_killed [.../jruby-9.1.12.0/lib/ruby/gems/shared/bundler/gems/rails-8e7fff4c2122/activerecord/test/cases/transactions_test.rb:541]:
First shouldn't have been approved

5031 runs, 14918 assertions, 1 failures, 0 errors, 10 skips

This test fails due to a bug in jruby, see: jruby/jruby#4705

Do you like what you see so far? Please lets talk about the next steps.

iaddict commented Jul 3, 2017

Hi. I worked some more on the jdbc mode. Travisci now also runs with jruby.
Only one AR test case is still failing, the rest is green. 🎉

$ jruby -S bundle exec rake test

  1) Failure:
TransactionTest#test_rollback_when_thread_killed [.../jruby-9.1.12.0/lib/ruby/gems/shared/bundler/gems/rails-8e7fff4c2122/activerecord/test/cases/transactions_test.rb:541]:
First shouldn't have been approved

5031 runs, 14918 assertions, 1 failures, 0 errors, 10 skips

This test fails due to a bug in jruby, see: jruby/jruby#4705

Do you like what you see so far? Please lets talk about the next steps.

@iaddict

This comment has been minimized.

Show comment
Hide comment
@iaddict

iaddict Sep 6, 2017

Hi @metaskills
Did you already find some time to check this pull request?

iaddict commented Sep 6, 2017

Hi @metaskills
Did you already find some time to check this pull request?

@metaskills

This comment has been minimized.

Show comment
Hide comment
@metaskills

metaskills Sep 6, 2017

Contributor

I'll be spending this week climbing into all PRs and issues.

Contributor

metaskills commented Sep 6, 2017

I'll be spending this week climbing into all PRs and issues.

@iaddict

This comment has been minimized.

Show comment
Hide comment
@iaddict

iaddict Nov 20, 2017

Hi @metaskills.
I did some investigation and found the cause for the above mentioned issue on jruby with killed threads and rollbacks. This has been fixed on jruby v9.1.14 (jruby/jruby@d335c0c) and thus almost all tests are green on rails 5.1.4 (with your ONE_AS_ONE patch applied 👍). Only one rails AR-test fails (test_serialized_attribute_works_under_concurrent_initial_access).
Maybe this one has already been fixed with your further development on the adapter.
On rails 5.1.0-5.1.2 all tests are green.

I would merge my branch with master and see how it works, but I would first like to hear your plans concerning this PR.

iaddict commented Nov 20, 2017

Hi @metaskills.
I did some investigation and found the cause for the above mentioned issue on jruby with killed threads and rollbacks. This has been fixed on jruby v9.1.14 (jruby/jruby@d335c0c) and thus almost all tests are green on rails 5.1.4 (with your ONE_AS_ONE patch applied 👍). Only one rails AR-test fails (test_serialized_attribute_works_under_concurrent_initial_access).
Maybe this one has already been fixed with your further development on the adapter.
On rails 5.1.0-5.1.2 all tests are green.

I would merge my branch with master and see how it works, but I would first like to hear your plans concerning this PR.

@iaddict

This comment has been minimized.

Show comment
Hide comment
@iaddict

iaddict Mar 29, 2018

Hi @metaskills. New year, new chance ;)
Have you already had time to look into this PR?

iaddict commented Mar 29, 2018

Hi @metaskills. New year, new chance ;)
Have you already had time to look into this PR?

@seizmo

This comment has been minimized.

Show comment
Hide comment
@seizmo

seizmo May 24, 2018

Hi @iaddict and @metaskills, great job in the project! Since the MS SQL Support of activerecord-jdbc-adapter has been in decline for a while (cf. jruby/activerecord-jdbc-adapter#784) and I don't think that will change anytime soon, having activerecord-sqlserver-adapter as a high quality option for JRuby users would be great!

seizmo commented May 24, 2018

Hi @iaddict and @metaskills, great job in the project! Since the MS SQL Support of activerecord-jdbc-adapter has been in decline for a while (cf. jruby/activerecord-jdbc-adapter#784) and I don't think that will change anytime soon, having activerecord-sqlserver-adapter as a high quality option for JRuby users would be great!

@rdubya

This comment has been minimized.

Show comment
Hide comment
@rdubya

rdubya Jul 23, 2018

Hey all, has there been any more movement on this? We're stuck at Rails 4.2 until there is a viable MSSQL adapter. So, if there's anything I can do to help move it along, I'd be more than happy to put some time into it.

rdubya commented Jul 23, 2018

Hey all, has there been any more movement on this? We're stuck at Rails 4.2 until there is a viable MSSQL adapter. So, if there's anything I can do to help move it along, I'd be more than happy to put some time into it.

@iaddict

This comment has been minimized.

Show comment
Hide comment
@iaddict

iaddict Aug 2, 2018

I would like to see this pull request also to make it into the adapter.
I already tested it with a medium sized rails 4.2 project which I did upgrade to rails 5.1 and it worked fine.

What can we do to make this a reality?

iaddict commented Aug 2, 2018

I would like to see this pull request also to make it into the adapter.
I already tested it with a medium sized rails 4.2 project which I did upgrade to rails 5.1 and it worked fine.

What can we do to make this a reality?

@neojin

This comment has been minimized.

Show comment
Hide comment
@neojin

neojin Sep 18, 2018

@iaddict Is it possible to use the jdbc-mode branch in our Gemfiles while waiting for this to get merged?

neojin commented Sep 18, 2018

@iaddict Is it possible to use the jdbc-mode branch in our Gemfiles while waiting for this to get merged?

@seizmo

This comment has been minimized.

Show comment
Hide comment
@seizmo

seizmo Sep 18, 2018

@iaddict Would you consider to fork activerecord-sqlserver-adapter with you jdbc changes and upload your own gem? The maintainer seems to be MIA and there are lots of people waiting for it to be easily integratable in their projects (esp. to be able to upgrade to Rails 5.x).

seizmo commented Sep 18, 2018

@iaddict Would you consider to fork activerecord-sqlserver-adapter with you jdbc changes and upload your own gem? The maintainer seems to be MIA and there are lots of people waiting for it to be easily integratable in their projects (esp. to be able to upgrade to Rails 5.x).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment