APEXMALHAR-2172: Updates to JDBC Poll Input Operator

[DT-Priyanka]

Changes include:

1. Using jooq query to construct sql queries
2. Update logic to query based on row numbers instead of primary key column values
3. Small fixes
4. Code refactoring to make it more readable
5. Update operator to emit pojo class constructed from table input fields
6. Update to test cases, to abstract out generic test code and add more tests

[DT-Priyanka]

@bhupeshchawda and @devtagare please review

[tweise]

Multiple file headers and import statements mixed.

[tweise]

How does 2. work? The query needs to be repeatable, so you will still need to sort on the primary key?

[tweise]

The jooq license isn't one of those listed here:

http://www.apache.org/legal/resolved.html#category-a

Where is the analysis on license compatibility?

[bhupeshchawda]

Building the range query is currently based on limit and offset.
This kind of querying might create problems since the order of fetching data might change depending on various factors like presence of indexes on that column or also might depend on what plan is generated for the query. The output being a set, we cannot assume the order. In order for it to be repeatable, we have the following options:

1. Explicitly use "order by" with limit and offset. - Might be costly for larger tables. Although this is worth exploring as we can assume a unique key index on this column and improve the performance.
2. Explicitly identify bounds for the query and change the query format to where key >= low and key <= high. This should work in a repeatable manner; However, the performance here is again questionable in absence of indexes. Perhaps we'll need to assume an index in this case as well for performance.
3. Use the current query (limit and offset) using something like RowNumber as in oracle databases. However, need to check if such functionality is present in other databases.

[DT-Priyanka]

Reading further, I guess right thing to do will be use "order by" and also avoid using "offset".

Here are some good reads about it:
http://www.iheavy.com/2013/06/19/3-ways-to-optimize-for-paging-in-mysql/
http://www.slideshare.net/slideshow/view?login=Eweaver&preview=no&slideid=1&title=efficient-pagination-using-mysql

I will check if I can modify our query to be more efficient, also would clarify assumptions about "key" column if any.

[DT-Priyanka]

Thomas, using "order by" key column in queries to ensure right results.

Would evaluate options to optimize performance by avoiding use of "offset"

[bhupeshchawda]

Can lower, lastPolledBound and lastEmittedRecord be int instead of Integer?

[DT-Priyanka]

I am doing a null check and those variables are compared and assigned to each other so won't do that.

[tweise]

What if the user wants to use RDBMS specific SQL?

[DT-Priyanka]

Thomas,
In JDBCPojoInputOperator, we do allow user to directly specify raw query, but we haven't done same here, it would be little complicated considering we are firing range queries here. If user doesn't specify such range queries our operator can misbehave.

[bhupeshchawda]

Can you make lowerBound, lastPolledBound and lastEmittedRecord as int instead of Integer

[bhupeshchawda]

Can we remove explicit mention for MySQL? I think it is generic as of now..

[bhupeshchawda]

@DT-Priyanka I have a few outstanding comments. Please address them.
Additionally also add documentation to data members and getters and setters along with proper doc tags. Fix documentation for refactored classes.
Also, please check a few places where store.disconnect() is not called in case of an exception.

[bhupeshchawda]

while(!emitQueue.offer(getTuple(result))) { Thread.sleep(10); } ?

[bhupeshchawda]

Also 1000 sounds too much?

[DT-Priyanka]

Should I change it to 500, as it's our default window size?

[DT-Priyanka]

Sorry had missed couple of comments before, addressing remaining comments.

[bhupeshchawda]

Looks good to me. If there are no other comments, will merge by tomorrow.

[DT-Priyanka]

File Splitter test is failing reopening to re-run tests.