Fix failing Netezza symbols core tests #579

javornikolov · 2017-05-27T08:22:53Z

Resolves #578

Parameterize CalcLength parameters in Symbols acceptance tests
Create CalcLength stored procedure for Netezza
Implement support for Neteza functions return values
Fix a small old typo in 001_add_objects_needed_for_acceptance_test.sql

javornikolov · 2017-05-27T08:27:39Z

@MMatten, I hope that should solve the Netezza tests failure. Possibly we may need to also rename the input parameter (e.g. to $1) but I'm not quite sure yet.
Would you be able to test it?

javornikolov · 2017-05-27T08:38:04Z

Possibly we may need to also rename the input parameter (e.g. to $1) but I'm not quite sure yet.

Pretty much looks like we should follow the $n naming so I just changed and force pushed it that way.

MMatten · 2017-05-27T20:10:58Z

I'm getting a failure in CoreTests.CommonSuite.Symbols whilst executing CalcLength: -

java.sql.SQLException: Cannot find parameter ""
	at dbfit.api.DbStoredProcedure.findAccessorForParamWithName(DbStoredProcedure.java:42)
	at dbfit.api.DbStoredProcedure.getDbParameterAccessor(DbStoredProcedure.java:29)
	at dbfit.fixture.DbObjectExecutionFixture.getAccessors(DbObjectExecutionFixture.java:121)
	at dbfit.fixture.DbObjectExecutionFixture.getAccessors(DbObjectExecutionFixture.java:108)
	at dbfit.fixture.DbObjectExecutionFixture.doRows(DbObjectExecutionFixture.java:71)
	at fit.Fixture.doTable(Fixture.java:156)
	at fitlibrary.traverse.AlienTraverseHandler.doTable(AlienTraverseHandler.java:21)
	at fitlibrary.traverse.workflow.DoTraverseInterpreter.interpretWholeTable(DoTraverseInterpreter.java:104)
	at fitlibrary.traverse.workflow.DoTraverseInterpreter.interpretWholeTable(DoTraverseInterpreter.java:89)
	at fitlibrary.DoFixture.interpretWholeTable(DoFixture.java:73)
	at fitlibrary.suite.InFlowPageRunner.run(InFlowPageRunner.java:27)
	at fitlibrary.DoFixture.interpretTables(DoFixture.java:42)
	at dbfit.DatabaseTest.interpretTables(DatabaseTest.java:26)
	at fit.Fixture.doTables(Fixture.java:81)
	at fit.FitServer.process(FitServer.java:81)
	at fit.FitServer.run(FitServer.java:56)
	at fit.FitServer.main(FitServer.java:41)

MMatten · 2017-05-27T20:22:25Z

Note that in the _v_procedure system view this another column called RETURNS. It specifies the data type of the RETURNS clause for a procedure.

Perhaps we can use this to return the OUT param via the SQL query.

javornikolov · 2017-05-27T20:29:07Z

Indeed - Netezza dbfit environment doesn't support return values yet. Perhaps RETURNS column is what we need to form the return value descriptor.

MMatten · 2017-05-28T10:50:10Z

@javornikolov let me know if you need me to run any SQL queries, tests, etc, etc.

javornikolov · 2017-05-28T11:59:34Z

@MMatten, I just pushed an attempt to add Netezza function return value support. May we test with it please. (It might worth adding some specific acceptance test for that but for a start we may use the Symbols tests).

MMatten · 2017-05-28T12:17:49Z

I got a build error: -

:dbfit-java:netezza:compileJava
/var/dbfit/dbfit-java/netezza/src/main/java/dbfit/environment/NetezzaEnvironment.java:258: error: variable paramName might not have been initialized
            allParams.put(NameNormaliser.normaliseName(paramName), dbp);
                                                       ^
1 error
:dbfit-java:netezza:compileJava FAILED

FAILURE: Build failed with an exception.

javornikolov · 2017-05-28T12:19:14Z

Yes, sorry, I saw that too. I just force-pushed a version which should compile OK.

MMatten · 2017-05-28T13:23:02Z

In FlowMode.CallProcedures for the Inspect Procedure of sp_procedure I get the error: -

org.netezza.error.NzSQLException: Column Index out of range: 2
	at org.netezza.sql.NzResultSet.getPgString(NzResultSet.java:2806)
	at org.netezza.sql.NzResultSet.getString(NzResultSet.java:1378)
	at dbfit.environment.NetezzaEnvironment.getAllProcedureParameters(NetezzaEnvironment.java:220)
	at dbfit.fixture.Inspect.inspectProcedure(Inspect.java:70)
	at dbfit.fixture.Inspect.doTable(Inspect.java:44)
	at fitlibrary.traverse.AlienTraverseHandler.doTable(AlienTraverseHandler.java:21)
	at fitlibrary.traverse.workflow.DoTraverseInterpreter.interpretWholeTable(DoTraverseInterpreter.java:104)
	at fitlibrary.traverse.workflow.DoTraverseInterpreter.interpretWholeTable(DoTraverseInterpreter.java:89)
	at fitlibrary.DoFixture.interpretWholeTable(DoFixture.java:73)
	at fitlibrary.suite.InFlowPageRunner.run(InFlowPageRunner.java:27)
	at fitlibrary.DoFixture.interpretTables(DoFixture.java:42)
	at dbfit.DatabaseTest.interpretTables(DatabaseTest.java:26)
	at fit.Fixture.doTables(Fixture.java:81)
	at fit.FitServer.process(FitServer.java:81)
	at fit.FitServer.run(FitServer.java:56)
	at fit.FitServer.main(FitServer.java:41)

MMatten · 2017-05-28T13:24:29Z

Executing CalcLength I get: -

org.netezza.error.NzSQLException: Column Index out of range: 2
	at org.netezza.sql.NzResultSet.getPgString(NzResultSet.java:2806)
	at org.netezza.sql.NzResultSet.getString(NzResultSet.java:1378)
	at dbfit.environment.NetezzaEnvironment.getAllProcedureParameters(NetezzaEnvironment.java:220)
	at dbfit.api.DbStoredProcedure.getAllParams(DbStoredProcedure.java:49)
	at dbfit.api.DbStoredProcedure.findAccessorForParamWithName(DbStoredProcedure.java:40)
	at dbfit.api.DbStoredProcedure.getDbParameterAccessor(DbStoredProcedure.java:29)
	at dbfit.fixture.DbObjectExecutionFixture.getAccessors(DbObjectExecutionFixture.java:121)
	at dbfit.fixture.DbObjectExecutionFixture.getAccessors(DbObjectExecutionFixture.java:108)
	at dbfit.fixture.DbObjectExecutionFixture.doRows(DbObjectExecutionFixture.java:71)
	at fit.Fixture.doTable(Fixture.java:156)
	at fitlibrary.traverse.AlienTraverseHandler.doTable(AlienTraverseHandler.java:21)
	at fitlibrary.traverse.workflow.DoTraverseInterpreter.interpretWholeTable(DoTraverseInterpreter.java:104)
	at fitlibrary.traverse.workflow.DoTraverseInterpreter.interpretWholeTable(DoTraverseInterpreter.java:89)
	at fitlibrary.DoFixture.interpretWholeTable(DoFixture.java:73)
	at fitlibrary.suite.InFlowPageRunner.run(InFlowPageRunner.java:27)
	at fitlibrary.DoFixture.interpretTables(DoFixture.java:42)
	at dbfit.DatabaseTest.interpretTables(DatabaseTest.java:26)
	at fit.Fixture.doTables(Fixture.java:81)
	at fit.FitServer.process(FitServer.java:81)
	at fit.FitServer.run(FitServer.java:56)
	at fit.FitServer.main(FitServer.java:41)

javornikolov · 2017-05-28T13:37:20Z

That was because I forgot to add retuns column to the SQL query... Fixed now.

MMatten · 2017-05-28T14:55:59Z

FlowMode.CallProcedures is now passing.

CoreTests.CommonSuite.Symbols now fails with: -

org.netezza.sql.NzCallableStatement$OutParameterNorRegisteredException: out parameter not registered
	at org.netezza.sql.NzCallableStatement.getObject(NzCallableStatement.java:684)
	at dbfit.fixture.StatementExecution.getObject(StatementExecution.java:31)
	at dbfit.util.DbParameterAccessor.get(DbParameterAccessor.java:118)
	at dbfit.util.DbParameterAccessorTypeAdapter.get(DbParameterAccessorTypeAdapter.java:18)
	at dbfit.util.SymbolAccessQueryBinding.doCell(SymbolAccessQueryBinding.java:15)
	at dbfit.fixture.DbObjectExecutionFixture.executeStatementAndEvaluateOutputs(DbObjectExecutionFixture.java:188)
	at dbfit.fixture.DbObjectExecutionFixture.runRow(DbObjectExecutionFixture.java:155)
	at dbfit.fixture.DbObjectExecutionFixture.doRows(DbObjectExecutionFixture.java:79)
	at fit.Fixture.doTable(Fixture.java:156)
	at fitlibrary.traverse.AlienTraverseHandler.doTable(AlienTraverseHandler.java:21)
	at fitlibrary.traverse.workflow.DoTraverseInterpreter.interpretWholeTable(DoTraverseInterpreter.java:104)
	at fitlibrary.traverse.workflow.DoTraverseInterpreter.interpretWholeTable(DoTraverseInterpreter.java:89)
	at fitlibrary.DoFixture.interpretWholeTable(DoFixture.java:73)
	at fitlibrary.suite.InFlowPageRunner.run(InFlowPageRunner.java:27)
	at fitlibrary.DoFixture.interpretTables(DoFixture.java:42)
	at dbfit.DatabaseTest.interpretTables(DatabaseTest.java:26)
	at fit.Fixture.doTables(Fixture.java:81)
	at fit.FitServer.process(FitServer.java:81)
	at fit.FitServer.run(FitServer.java:56)
	at fit.FitServer.main(FitServer.java:41)

javornikolov · 2017-05-28T18:34:18Z

Hmm... I just added in FlowMode.CallProcedures another test for sp_procedure return value and Inspect Procedure and direct calls to CalcLength.

javornikolov · 2017-05-28T18:36:10Z

Gee, looks like we're already in debt with that (#11): https://www.ibm.com/developerworks/community/forums/html/threadTopic?id=6ab238ce-f4d2-48f9-ae4e-f9e648fff9fe

MMatten · 2017-05-28T21:08:33Z

Gee, looks like we're already in debt with that (#11): https://www.ibm.com/developerworks/community/forums/html/threadTopic?id=6ab238ce-f4d2-48f9-ae4e-f9e648fff9fe

No, no out params in stored procs and no function return values?

May be the what we created for Informix would suit this.

Let me test your updates now.

javornikolov · 2017-05-28T21:13:47Z

I decompiled the JDBC driver to see what's going on inside. Looks like it has some support for return values but not through getObject but using getInt, getString, etc... (Internally these are implemented via getResultSet). Besides getBigDecimal looks a bit buggy at first glimpse.

I'm just working on a getObject override to see if we can trick it in that way. I'll push it in a few minutes. Later we may try the getResultSet approach too.

javornikolov · 2017-05-28T21:23:58Z

@MMatten, I just pushed the patched Netezza environment.

MMatten · 2017-05-28T21:43:47Z

CallProcedures fails with: -

Everything else seems to be passing.

javornikolov · 2017-05-28T21:49:09Z

Everything else seems to be passing.

Great :-)
I fixed the other part too (I didn't pay attention that the function call had side effect in populating a table).

javornikolov · 2017-05-28T21:50:36Z

At first glimpse the Informix approach for function calls should work here out of the box. I'll change that shortly to see if we can get rid of my getObject overrides.

javornikolov · 2017-05-28T21:54:47Z

At first glimpse the Informix approach for function calls should work here out of the box. I'll change that shortly to see if we can get rid of my getObject overrides.

Done - pushed with 4bee54e.

MMatten · 2017-05-28T22:04:56Z

In CallProcedures where we're testing procs and looking for the output I'm getting: -

org.netezza.error.NzSQLException: Value can be set for IN/INOUT parameter type only
	at org.netezza.sql.NzCallableStatement.setObject(NzCallableStatement.java:2031)
	at dbfit.fixture.StatementExecution.setObject(StatementExecution.java:26)
	at dbfit.fixture.StatementExecutionCapturingResultSetValue.setObject(StatementExecutionCapturingResultSetValue.java:32)
	at dbfit.util.DbParameterAccessor.set(DbParameterAccessor.java:110)
	at dbfit.util.DbParameterAccessorTypeAdapter.set(DbParameterAccessorTypeAdapter.java:22)
	at fit.Binding$SetBinding.doCell(Binding.java:217)
	at dbfit.util.SymbolAccessSetBinding.doCell(SymbolAccessSetBinding.java:22)
	at dbfit.fixture.DbObjectExecutionFixture.runRow(DbObjectExecutionFixture.java:151)
	at dbfit.fixture.DbObjectExecutionFixture.doRows(DbObjectExecutionFixture.java:79)
	at fit.Fixture.doTable(Fixture.java:156)
	at fitlibrary.traverse.AlienTraverseHandler.doTable(AlienTraverseHandler.java:21)
	at fitlibrary.traverse.workflow.DoTraverseInterpreter.interpretWholeTable(DoTraverseInterpreter.java:104)
	at fitlibrary.traverse.workflow.DoTraverseInterpreter.interpretWholeTable(DoTraverseInterpreter.java:89)
	at fitlibrary.DoFixture.interpretWholeTable(DoFixture.java:73)
	at fitlibrary.suite.InFlowPageRunner.run(InFlowPageRunner.java:27)
	at fitlibrary.DoFixture.interpretTables(DoFixture.java:42)
	at dbfit.DatabaseTest.interpretTables(DatabaseTest.java:26)
	at fit.Fixture.doTables(Fixture.java:81)
	at fit.FitServer.process(FitServer.java:81)
	at fit.FitServer.run(FitServer.java:56)
	at fit.FitServer.main(FitServer.java:41)

MMatten · 2017-05-28T22:08:08Z

I'm still seeing this.

javornikolov · 2017-05-28T22:31:34Z

I'm still seeing this.

My bad. I moved more calls than needed. should be OK now.

org.netezza.error.NzSQLException: Value can be set for IN/INOUT parameter type only

Perhaps we sholdn't adjust the parameter index as we do for Informix. I changed that. Let's see if that helps.

MMatten · 2017-05-28T22:51:57Z

Awesome. A clean bill of health again. No test failures.

javornikolov · 2017-05-29T08:12:15Z

Nice :-) I need to tidy up some bits here but looks like that we finally managed to get Netezza functions working.

MMatten · 2017-05-29T08:15:09Z

Great work!

Interestingly there are also functions in Netezza in addition to stored procs that return values. I think they have to be built in C++ and compiled into the DB. One for another day!

* Parameterize CalcLength parameters in Symbols acceptance tests in order to be able to configure it in an Netezza-compatible way. * Create CalcLength stored procedure for Netezza

Netezza JDBC driver doesn't support CallableStatement.getObject. We handle that limitation by retrieving return value through the statement's ResultSet.

Split getAllProcedureParameters into several routines using the same approach (and copy of some code) as in Postgres environment

javornikolov · 2017-05-29T21:47:48Z

@MMatten, I did some refactoring. Can we test it again?
And if it looks good enough to you - I think we can merge it (maybe slicing the last commit with the refactoring into a separate PR).

MMatten · 2017-06-03T09:07:18Z

We're still missing the CalcLength stored routine for SQL Server.

But I'll test all the others (I just have Teradata to do).

MMatten · 2017-06-03T21:59:03Z

I'm now seeing Netezza failures in FlowMode.CallProcedures: -

Length specifiers are dropped

javornikolov · 2017-06-06T19:16:10Z

@MMatten, could you test that again. I hope we have that error fixed now.

MMatten · 2017-06-10T13:14:31Z

All tests are passing now.

javornikolov · 2017-06-10T13:29:01Z

Wonderful :-) Thanks for verifying, @MMatten! I guess we're ready to merge that PR?

MMatten · 2017-06-10T14:09:04Z

Yes. I think so. I've nothing else to add.

javornikolov force-pushed the topic/fix-netezza-symbols-test branch from de3fac2 to fc6edd4 Compare May 27, 2017 08:36

MMatten mentioned this pull request May 28, 2017

Deploy Netezza acceptance test objects via dbdeploy #580

Merged

4 tasks

javornikolov force-pushed the topic/fix-netezza-symbols-test branch from 606ec11 to 74b24ce Compare May 28, 2017 12:14

javornikolov force-pushed the topic/fix-netezza-symbols-test branch from 74b24ce to 737dd75 Compare May 28, 2017 13:35

javornikolov added 4 commits May 29, 2017 23:13

Fix typo in netezza schema objects setup script

53c38d0

Fix Netezza symbols acceptance tests

fc7a132

* Parameterize CalcLength parameters in Symbols acceptance tests in order to be able to configure it in an Netezza-compatible way. * Create CalcLength stored procedure for Netezza

Netezza function return value support

311d864

Netezza JDBC driver doesn't support CallableStatement.getObject. We handle that limitation by retrieving return value through the statement's ResultSet.

Refactor Netezza function parameters retrieval

76aa8b9

Split getAllProcedureParameters into several routines using the same approach (and copy of some code) as in Postgres environment

javornikolov force-pushed the topic/fix-netezza-symbols-test branch from c4609c7 to 76aa8b9 Compare May 29, 2017 21:39

Fix Netezza procedure parameter type names parsing

a2bdc65

Length specifiers are dropped

javornikolov added bugfix enhancement labels Jun 6, 2017

javornikolov added this to the Next milestone Jun 6, 2017

Refactor Netezza parameter accessor construction

03bbd3f

javornikolov merged commit fe7e983 into master Jun 10, 2017

javornikolov deleted the topic/fix-netezza-symbols-test branch June 10, 2017 17:15

Fix failing Netezza symbols core tests #579

Fix failing Netezza symbols core tests #579

Conversation

javornikolov commented May 27, 2017 • edited

javornikolov commented May 27, 2017

javornikolov commented May 27, 2017

MMatten commented May 27, 2017

MMatten commented May 27, 2017

javornikolov commented May 27, 2017

MMatten commented May 28, 2017

javornikolov commented May 28, 2017

MMatten commented May 28, 2017

javornikolov commented May 28, 2017

MMatten commented May 28, 2017 • edited

MMatten commented May 28, 2017

javornikolov commented May 28, 2017

MMatten commented May 28, 2017

javornikolov commented May 28, 2017

javornikolov commented May 28, 2017 • edited

MMatten commented May 28, 2017

javornikolov commented May 28, 2017

javornikolov commented May 28, 2017

MMatten commented May 28, 2017 • edited

javornikolov commented May 28, 2017

javornikolov commented May 28, 2017

javornikolov commented May 28, 2017

MMatten commented May 28, 2017

MMatten commented May 28, 2017

javornikolov commented May 28, 2017

MMatten commented May 28, 2017

javornikolov commented May 29, 2017

MMatten commented May 29, 2017

javornikolov commented May 29, 2017

MMatten commented Jun 3, 2017

MMatten commented Jun 3, 2017

javornikolov commented Jun 6, 2017

MMatten commented Jun 10, 2017

javornikolov commented Jun 10, 2017

MMatten commented Jun 10, 2017

javornikolov commented May 27, 2017 •

edited

MMatten commented May 28, 2017 •

edited

javornikolov commented May 28, 2017 •

edited

MMatten commented May 28, 2017 •

edited