Skip to content
This repository has been archived by the owner on Jul 23, 2024. It is now read-only.

HAWQ-1461. Improve partition parameters validation for PXF-JDBC plugin #1249

Closed
wants to merge 2 commits into from
Closed

Conversation

lavjain
Copy link

@lavjain lavjain commented Jun 1, 2017

No description provided.

pxf/build.gradle Outdated
@@ -477,7 +481,7 @@ def buildNumber() {
}

task wrapper(type: Wrapper) {
gradleVersion = '2.11'
gradleVersion = '2.14.1'
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we need to update Gradle version?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We are currently using 2 different versions 2.11 and 2.13 (pxf/gradle/wrapper/gradle-wrapper.properties)

*/
public void retrieveStats() throws Exception;

/**
* Returns next tuple based on statistics information without actual reading of data
* @return only one row
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@return next row without reading it from disk

try {
range = inConf.getUserProperty("RANGE").split(":");
if (inConf.getUserProperty("RANGE") != null) {
range = inConf.getUserProperty("RANGE").split(":");
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

calling same function twice, would be better to call it once saving result and then using the result for further logic

try {
range = inConf.getUserProperty("RANGE").split(":");
if (inConf.getUserProperty("RANGE") != null) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"RANGE" should be an enum value or at least a constant somewhere

if (inConf.getUserProperty("RANGE") != null) {
range = inConf.getUserProperty("RANGE").split(":");
if (range.length == 1 && partitionType != PartitionType.ENUM)
throw new UserDataException("The parameter 'RANGE' does not specify '[:end_value]'");
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why did we drop support for open-ended range ?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we want to support open-ended range on fragmenter phase - we might have the last fragment having more data than rest of fragments, which is acceptable and very user-friendly so +1 for having open-ended range.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

in the similar fashion, we might support open ended lower range as well -- in cases where min value is not known or changes dynamically.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Supporting open-ended range would require changes to the JDBC code. The current implementation does not support it.

@@ -144,7 +153,7 @@ public JdbcPartitionFragmenter(InputData inConf) throws UserDataException {
rangeEnd.setTime(df.parse(range[1]));
}
} catch (ParseException e) {
throw new UserDataException("The parameter 'RANGE' include invalid date format.");
throw new UserDataException("The parameter 'RANGE' has invalid date format.");
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should we also say what expected format is, such as : "yyyy-MM-dd" ?

@@ -164,8 +163,8 @@ public void inValidParameterFormat() throws Exception {
when(inputData.getUserProperty("RANGE")).thenReturn("100:200");
try {
new JdbcPartitionFragmenter(inputData);
fail("Expected an ArrayIndexOutOfBoundsException");
} catch (ArrayIndexOutOfBoundsException ex) {
fail("Expected an UserDataException");
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

use @test(expected = UserDataException.class) so that there is no need for try / catch / fail lines : https://github.com/junit-team/junit4/wiki/exception-testing


JdbcPartitionFragmenter fragment = new JdbcPartitionFragmenter(inputData);
List<Fragment> fragments = fragment.getFragments();
assertEquals(fragments.size(), 11);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the order of params for assertEquals usually goes (expected, actual) : http://junit.sourceforge.net/javadoc/org/junit/Assert.html#assertEquals(java.lang.Object, java.lang.Object) as it will report errors as such, so it might be confusing when error happens what was actually expected and what was returned.

pxf/build.gradle Outdated
@@ -111,6 +111,10 @@ subprojects { subProject ->
force 'org.codehaus.jackson:jackson-mapper-asl:1.9.13'
force 'junit:junit:4.11'
}
configurations {
exclude group: 'org.json', module: 'json'
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

are we sure it will run OK without it ?

@lavjain lavjain closed this Jun 10, 2017
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants