Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

NIFI-5572 #3343

Closed
wants to merge 10 commits into from
Closed

NIFI-5572 #3343

wants to merge 10 commits into from

Conversation

gideonkorir
Copy link
Contributor

Add Decimal support

  • RecordFieldType.DECIMAL
  • DecimalDataType
  • Fix conversions for Avro + ORC
  • Fix broken tests

@MikeThomsen
Copy link
Contributor

@gideonkorir can you rebase and force push to your branch?

@gideonkorir
Copy link
Contributor Author

@MikeThomsen I've just done that again after fixing the style issues

@joewitt
Copy link
Contributor

joewitt commented Mar 13, 2019

@gideonkorir there is a conflict and ideally the commits would be squashed to allow for reviews to result in easier to follow changes.

@gideonkorir
Copy link
Contributor Author

@joewitt I've fixed the conflicts, I assumed it's a squash merge according to contributor guide.

@gideonkorir gideonkorir force-pushed the NIFI-5572 branch 2 times, most recently from 78029c0 to bed5f88 Compare March 18, 2019 11:34
DecimalDataType hashcode fix

AvroTypeUtil conversions include decimal now :)

Read/Write tests are passing

- Added missing overrides to DecimalDataType
- Add scale to DecimalDataType
- Fix broken Avro tests
- Fix broken ORC tests
- Fix broken Csv tests
- Fix broken Xml tests

tests are passing

fix checkstyle issues

added missing license header

look for getAsDouble and handle decimal conversions

fixed bug with TestPutKudu.java

decimal support for query record
@gideonkorir
Copy link
Contributor Author

Realized I was naive in my initial PR, I've gone through all the references to RecordFieldType and made the following changes:

  • Update PutCassandra processor to handle decimal and added tests for that
  • Update PutElasticSearch processor to handle decimal and added tests for that. Caveat, I'm using BigDecimal.toPlainString() so that we don't have scientific notation in the json output; not sure if OK
  • Added decimal tests for HBase
  • Added decimal tests for Kudu
  • Update PutMongoRecord processor to handle decimal, added codec package that adds the necessary codecs for that.
  • Update StandardSchemaValidator, support decimal in validate, added tests for that
  • ResultSetRecordSet has decimal support, factoring in precision and scale of the returned decimals
  • JsonPathRowRecordReader has decimal support, added tests for that
  • JsonTreeRowRecordReader has decimal support, added tests for that

@@ -265,6 +270,19 @@ public DataType getDataType(final String format) {
return new DataType(this, format);
}

/**
* Returns a Data Type that represents a "Decimal" with given precision
* @param precision number of digits to the right of point (.)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The scale is actually the number of digits to the right of the point. The precision is the number of digits in the number.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You right don't know what I was thinking when I wrote the comment, will fix that. thanks

@@ -172,6 +180,8 @@ public static Object convertType(final Object value, final DataType dataType, fi

return convertType(value, chosenDataType, fieldName, charset);
}
case DECIMAL:
return toDecimal(value, fieldName).setScale(((DecimalDataType)dataType).getScale(), RoundingMode.HALF_UP);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not pass the scale (and ideally precision) to the toDecimal method and set the scale and rounding mode (and check precision) within that method?

if(value instanceof Number){
return new BigDecimal(((Number)value).doubleValue());
}

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would also check for precision here and throw an IllegalTypeConversionException if the required precision is less than the precision of the converted BigDecimal.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good feedback, will add that

@@ -11,6 +11,7 @@
"time" : "17:00:00",
"timestamp" : "2017-01-01 17:00:00",
"char" : "c",
"decimal" : "45.76",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would rather have this as a number to be honest. There is a specific case in the WriteJsonResult class.

@pvillard31 pvillard31 changed the title Nifi 5572 NIFI-5572 Nov 21, 2019
@github-actions
Copy link

github-actions bot commented May 2, 2021

We're marking this PR as stale due to lack of updates in the past few months. If after another couple of weeks the stale label has not been removed this PR will be closed. This stale marker and eventual auto close does not indicate a judgement of the PR just lack of reviewer bandwidth and helps us keep the PR queue more manageable. If you would like this PR re-opened you can do so and a committer can remove the stale tag. Or you can open a new PR. Try to help review other PRs to increase PR review bandwidth which in turn helps yours.

@github-actions github-actions bot added the Stale label May 2, 2021
@github-actions github-actions bot closed this May 18, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
4 participants