NIFI-6925: Fixed JoltTransformRecord for RecordReaders, improved MockProcessSession #3913
Conversation
|
I believe the change to MockProcessSession has found a bug in PutElasticsearchHttpRecord, as TestPutElasticsearchHttpRecord is failing. I'll take a look at that processor and update it as necessary. Hopefully the change is for the better and will find such bugs that unit tests haven't previously detected. |
| } catch (final Exception ex) { | ||
| logger.error("Unable to transform {} due to {}", new Object[]{original, ex.toString(), ex}); | ||
| session.transfer(original, REL_FAILURE); | ||
| session.remove(transformed); |
There was a problem hiding this comment.
Recommend adding a null check here. In testing I saw a case where if an exception is encountered the transformed object is null at this point. When session.remove is called on the object it hits a NullPointer exception.
| transformed = session.putAllAttributes(transformed, attributes); | ||
| logger.info("{} had no Records to transform", new Object[]{original}); | ||
| } else { | ||
|
|
||
| final JoltTransform transform = getTransform(context, original); | ||
| final Record transformedFirstRecord = transform(firstRecord, transform); |
There was a problem hiding this comment.
If the transform fails here transformedFirstRecord is returned and causes a NullPointerException on the next line for writeSchema. Recommend adding a null check.
|
Was able to configure a setup to validate in Hive. Appreciate your patience (and help) @mattyb149 . Also fixes for elasticsearch were confirmed as well +1 Will merge shortly |
…ProcessSession (apache#3913) * NIFI-6925: Fixed JoltTransformRecord for RecordReaders, improved MockProcessSession * Fixed logic for no records, added unit test * Fixed PutElasticsearchHttpRecord and PutHive3Streaming, same bug as JoltTransformRecord * Added null checks
…ProcessSession (apache#3913) * NIFI-6925: Fixed JoltTransformRecord for RecordReaders, improved MockProcessSession * Fixed logic for no records, added unit test * Fixed PutElasticsearchHttpRecord and PutHive3Streaming, same bug as JoltTransformRecord * Added null checks
Thank you for submitting a contribution to Apache NiFi.
Please provide a short description of the PR here:
Description of PR
For RecordReader implementations that don't close the underlying input stream when nextRecord() returns no records, JoltTransformRecord fails the flowfile instead of sending the input flowfile to the "original" relationship. Unit tests didn't catch it because MockProcessSession wasn't keeping track of the open stream counts.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
For all changes:
Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
Does your PR title start with NIFI-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
Has your PR been rebased against the latest commit within the target branch (typically
master)?Is your initial contribution a single, squashed commit? Additional commits in response to PR reviewer feedback should be made on this branch and pushed to allow change tracking. Do not
squashor use--forcewhen pushing to allow for clean monitoring of changes.For code changes:
mvn -Pcontrib-check clean installat the rootnififolder?LICENSEfile, including the mainLICENSEfile undernifi-assembly?NOTICEfile, including the mainNOTICEfile found undernifi-assembly?.displayNamein addition to .name (programmatic access) for each of the new properties?For documentation related changes:
Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.