NIFI-4545: Improve Hive processors provenance transit URL#2239
NIFI-4545: Improve Hive processors provenance transit URL#2239ijokarumawak wants to merge 2 commits intoapache:masterfrom
Conversation
|
Reviewing... |
mattyb149
left a comment
There was a problem hiding this comment.
This PR is awesome!!! Great information being added here for provenance and in general. I had just one issue during my testing, for tables that may be input and output tables during execution of Hive statement(s). Other than that, things are looking and working well!
| } | ||
|
|
||
| @Test | ||
| public void parseInsert() throws Exception { |
There was a problem hiding this comment.
These tests pass for me, but when I tried on a real NiFi against a Hive with two tables (prov_events and t), running a single flow file into PutHiveQL containing the following:
insert into t values (11,"eleven");
insert into prov_events select * from prov_events where durationmillis = 10;
insert into t values (0,"zero");
I correctly get query.input_tables = "prov_events", but query.output_tables = "t" when it should be "t,prov_events" right?
| throw new IllegalStateException("TOK_TABNAME does not have expected children, childCount=" + childCount); | ||
| } | ||
| // If parent is TOK_TABREF, then it is an input table. | ||
| tableName.setInput("TOK_TABREF".equals(tree.getParent().getText())); |
There was a problem hiding this comment.
We may need setInput and setOutput (and the getters) here, since a table could technically be input and output, either in the same statement or multiple statements. See my other comment for an example, also this one generates nothing in query.input_tables:
insert into t values (8,"eight");
create table if not exists t3 (id int, name string);
insert overwrite table t3 select distinct id,name from t;
- Added 'input' to equals() method so that the same table name can appear as input and output tables.
|
@mattyb149 Thanks for reviewing, glad you like it :) I've updated |
|
+1 LGTM, ran build with contrib-check and unit tests, also tried various HiveQL commands, verified the input and output tables attributes are being correctly populated. Thanks for this improvement! Merging to master |
Thank you for submitting a contribution to Apache NiFi.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
For all changes:
Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
Does your PR title start with NIFI-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
Has your PR been rebased against the latest commit within the target branch (typically master)?
Is your initial contribution a single, squashed commit?
For code changes:
For documentation related changes:
Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.