HIVE-17410 : repl load task during subsequent DAG generation does notstart from the last partition processed #240
Conversation
| tracker.addTask(tasksForAddPartition(table, addPartitionDesc)); | ||
| ReplicationState currentReplicationState = | ||
| new ReplicationState(new PartitionState(table.getTableName(), addPartitionDesc)); | ||
| updateReplicationState(currentReplicationState); |
There was a problem hiding this comment.
Need to set the replication state only if max number of tasks reached and still have more partitions to load. Or else, need to set it to null if all partitions are added within same DAG. This change is needed for REPL progress log feature.
| Map<String, String> partSpec = addPartitionDesc.getPartition(0).getPartSpec(); | ||
| Partition ptn = context.hiveDb.getPartition(table, partSpec, false); | ||
| if (ptn == null) { | ||
| if (!replicationSpec.isMetadataOnly()) { |
There was a problem hiding this comment.
metadataOnly flag is always false in case of bootstrap. So, shall avoid checking it here.
There was a problem hiding this comment.
for view it will still be isMetadataOnly though it should not come to partitions load for it, i looked at the code and looks like we can remove a larger part of the code now with this condition not true, will create a bug for it as it will require going through some code branches to figure this out. HIVE-17484
| // the destination ptn's repl.last.id is older than the replacement's. | ||
| if (replicationSpec.allowReplacementInto(ptn.getParameters())) { | ||
| if (replicationSpec.isMetadataOnly()) { | ||
| tracker.addTask(alterSinglePartition(addPartitionDesc, replicationSpec, ptn)); |
There was a problem hiding this comment.
Dead code as metadata only flag is always false for bootstrap. May be simplify this by safely assuming that "ptn exists" case never occur for bootstrap. Shall add a common method which takes iterator as input and iterates overs the iterator and load partitions. This method can be called by both forNewtable and forExistingTable.
There was a problem hiding this comment.
Same as above HIVE-17484
| if (replicationSpec.isMetadataOnly()) { | ||
| tracker.addTask(alterSinglePartition(addPartitionDesc, replicationSpec, ptn)); | ||
| if (!tracker.canAddMoreTasks()) { | ||
| tracker.setReplicationState( |
There was a problem hiding this comment.
If this is the last partition in the iterator, then should not set the replication state. Shall check this as well.
There was a problem hiding this comment.
Since this comes under dead code, it will not be executed and will be taken care of as part of HIVE-17484
… start from the last partition processed
… start from the last partition processed setting up the replicationState Correctly.
No description provided.