Hive: Preparation to enable Hive writes with Tez engine

[marton-bod]

In order to enable Hive writes using the Tez engine, we have to make a few modifications to the OutputCommitter due to the inner workings of Tez. Couple of main reasons for the changes:

1. Tez: Arbitrary inclusion/exclusion of vertexId in TaskAttemptID: there's a difference in Tez on how TaskAttemptIDs are constructed - in some places the vertexId is included in the TaskAttemptID, in other places it's not, which leads to differences in the IDs and therefore to issues when retrieving the Writer from the cache based on this ID.
2. Tez: taskType (reducer/mapper) being part of TaskAttemptID's equals/hashcode: this prevents a reducer from retrieving a Writer object that was previously cached by a mapper for example.

Enabling the unit tests to run on Tez will be done in a future PR. For that work, we'll need to release a new version of Hive and Tez containing the necessary patches (mainly HIVE-24629 and TEZ-4264) and update the dependencies here.

[marton-bod]

@pvary @lcspinter could you please review when you get the chance? Thanks!

[pvary]

+1 from my side, but I would like to see @rdblue's opinion about pushing code which we can not create tests for.

@rdblue: A bit of context: Internally we were able to run write tests with Tez, but there are multiple unreleased fixes needed for Hive and Tez too. The fixes are available on the apache repos but they are not present on any of the releases ATM. Releasing both components would be a slow process because of the dependencies, and would greatly delay adding code here. Shall we wait for those releases or we can put code here which could only be used by patched versions of Hive and Tez?

Thanks,
Peter

[rdblue]

@pvary, @marton-bod, as long as we are passing the existing tests, I think it is fine to add code that will be more thoroughly tested later. Better to get it in sooner.

[rdblue]

Will merge when tests pass. Thanks for updating this, @marton-bod!

[marton-bod]

Thanks a lot for your review, @rdblue!