HIVE-21029: External table replication for existing deployments running incremental replication. #523

sankarh · 2019-01-29T11:21:24Z

No description provided.

maheshk114 · 2019-01-30T09:26:52Z

ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/Utils.java

@@ -204,7 +211,7 @@ public static boolean shouldReplicate(NotificationEvent tableForEvent,
              .getTableName(), e);
      return false;
    }
-    return shouldReplicate(replicationSpec, table, hiveConf);
+    return shouldReplicate(replicationSpec, table, false, hiveConf);


here true should be passed as this is called for incremental dump. Better to change the prototype of this function also to avoid misuse

maheshk114 · 2019-01-30T09:45:16Z

ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplLoadTask.java

+      // If incremental events are already applied, then check and perform if need to bootstrap any tables.
+      if (!builder.hasMoreWork() && !work.getPathsToCopyIterator().hasNext()) {
+        if (work.hasBootstrapLoadTasks()) {
+          LOG.debug("Current incremental dump have tables to be bootstrapped. Switching to bootstrap "


this can run in parallel to incremental load. For external table both incremental and bootstrap can not be there

Cannot be done. We have to bootstrap external tables only after finish applying events. This is to avoid problems if some non-external table uses same name as external table and it has alter/drop table event in the current dump.
Anyways, bootstrap in itself loads all external tables in parallel. Also, it is metadata-only operations and most of the work is with copy of external tables data which anyways done in parallel with incremental events load.

maheshk114 · 2019-01-30T09:46:20Z

ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplLoadTask.java

@@ -351,13 +366,23 @@ public StageType getType() {

  private int executeIncrementalLoad(DriverContext driverContext) {
    try {
+      IncrementalLoadTasksBuilder builder = work.incrementalLoadTasksBuilder();


if (loadWork.getPathsToCopyIterator().hasNext()) {
taskChainTail.addDependentTask(TaskFactory.get(loadWork, conf));
}

This can be removed from IncrementalLoadTasksBuilder.build method

Yes. Will remove it.

maheshk114 · 2019-01-30T09:48:13Z

ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplLoadTask.java

+      // Update last repl ID of the database only if the current dump is not incremental. If bootstrap
+      // is combined with incremental dump, it contains only tables to bootstrap. So, needn't change
+      // last repl ID of the database.
+      if (!iterator.hasNext() && !constraintIterator.hasNext() && !work.isIncrementalLoad()) {


no checkpoint support for external table bootstrap ?

Checkpoint support is there. It is done in table/partition level for inc-bootstrap case. Database object should be unaltered and hence no need to set checkpoint flag on DB.

maheshk114 · 2019-01-30T09:50:25Z

common/src/java/org/apache/hadoop/hive/conf/HiveConf.java

+          + " repl dump to bootstrap all external tables."),
+    REPL_BOOTSTRAP_EXTERNAL_TABLES("hive.repl.bootstrap.external.tables", false,
+        "Indicates if repl dump should bootstrap the information about external tables during incremental \n"
+          + "phase of replication. It should be used in conjunction with 'hive.repl.include.external.tables' \n"


i think this should not be a config value as it may cause repeated bootstrap of external table. This can be passed as some parameter in dump command.

Yes, it should be set in WITH clause of REPL DUMP. Also, as the description says, it should be set to true only once for repl dump after enabling external tables replication. No issues having it as config provided user sets it properly.

i think it should not be a config ..should be a flag. Or atleast block it some how , so that it can be given only using repl dump command.

As discussed, updated description of this config to highlight that it should be set only via WITH clause of REPL DUMP.

maheshk114 · 2019-01-30T11:00:31Z

ql/src/java/org/apache/hadoop/hive/ql/exec/repl/bootstrap/load/table/LoadPartitions.java

-    // This in-turn applicable for partitions creation as well.
-    if ((parentDb != null) && (!event.replicationSpec().allowReplacementInto(parentDb.getParameters()))) {
-      return tracker;
-    }


why is it removed ?

Due to this check, external tables are not bootstrapped as DB is later (higher last_repl_id) than table itself. Also, this check doesn't make sense as it is bootstrap flow where we set DB's last_repl_id as 0 before triggering tables bootstrap. It will be always true. Also, in retry case, we depend on checkpoints to ignore replicated tables not this.

maheshk114 · 2019-01-30T13:09:25Z

ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplDumpTask.java

+    if (conf.getBoolVar(HiveConf.ConfVars.REPL_INCLUDE_EXTERNAL_TABLES)
+        && (!conf.getBoolVar(HiveConf.ConfVars.REPL_DUMP_METADATA_ONLY)
+        || conf.getBoolVar(HiveConf.ConfVars.REPL_BOOTSTRAP_EXTERNAL_TABLES))) {
+      Path dbRoot = getBootstrapDbRoot(dumpRoot, dbName, true);
      try (Writer writer = new Writer(dumpRoot, conf)) {
        for (String tableName : Utils.matchesTbl(hiveDb, dbName, work.tableNameOrPattern)) {
          Table table = hiveDb.getTable(dbName, tableName);
          if (TableType.EXTERNAL_TABLE.equals(table.getTableType())) {
            writer.dataLocationDump(table);


dataLocationDump should be done only if its not metadata only dump

It is validated inside dataLocationDump method and it is noop if metadata-only.

maheshk114 · 2019-01-30T13:10:38Z

ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplDumpTask.java

-        !conf.getBoolVar(HiveConf.ConfVars.REPL_DUMP_METADATA_ONLY)) {
+    if (conf.getBoolVar(HiveConf.ConfVars.REPL_INCLUDE_EXTERNAL_TABLES)
+        && (!conf.getBoolVar(HiveConf.ConfVars.REPL_DUMP_METADATA_ONLY)
+        || conf.getBoolVar(HiveConf.ConfVars.REPL_BOOTSTRAP_EXTERNAL_TABLES))) {


the condition is very confusing ..add a comment

maheshk114 · 2019-01-30T13:13:22Z

ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplDumpTask.java

+    if (conf.getBoolVar(HiveConf.ConfVars.REPL_INCLUDE_EXTERNAL_TABLES)
+        && (!conf.getBoolVar(HiveConf.ConfVars.REPL_DUMP_METADATA_ONLY)
+        || conf.getBoolVar(HiveConf.ConfVars.REPL_BOOTSTRAP_EXTERNAL_TABLES))) {
+      Path dbRoot = getBootstrapDbRoot(dumpRoot, dbName, true);


getBootstrapDbRoot should be called only if inc-bootstrap

Why is that? getBootstrapDbRoot takes flag to decide if it is for inc-bootstrap or normal one.

maheshk114 · 2019-01-30T13:19:22Z

ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplLoadWork.java

+      FileSystem fs = incBootstrapDir.getFileSystem(hiveConf);
+      if (fs.exists(incBootstrapDir)) {
+        this.bootstrapIterator = new BootstrapEventsIterator(incBootstrapDir.toString(), dbNameToLoadIn, hiveConf);
+        this.constraintsIterator = new ConstraintEventsIterator(dumpDirectory, hiveConf);


this.constraintsIterator ..why this is required ?

executeBootstrapLoad method expects it to be not null even if constraints are not there. Also, I set it here so that in future if bootstrapped table (such as ACID tables) have constraints, it works fine.

…ng incremental replication.

…only in REPL DUMP via WITH clause.

sankarh force-pushed the HIVE-21029 branch 3 times, most recently from a7ea7d9 to 859b07e Compare January 30, 2019 06:51

maheshk114 reviewed Jan 30, 2019

View reviewed changes

sankarh added 3 commits January 31, 2019 09:44

HIVE-21029: External table replication for existing deployments runni…

db71510

…ng incremental replication.

Fixed checkstyle issues and rebased with master.

0efc01e

Fixed review comments from Mahesh

a7ef9d7

sankarh force-pushed the HIVE-21029 branch from 9a1b61c to a7ef9d7 Compare January 31, 2019 04:14

Added description for hive.repl.bootstrap.external.tables to be used …

36fdf96

…only in REPL DUMP via WITH clause.

sankarh closed this Feb 1, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HIVE-21029: External table replication for existing deployments running incremental replication. #523

HIVE-21029: External table replication for existing deployments running incremental replication. #523

sankarh commented Jan 29, 2019

maheshk114 Jan 30, 2019

sankarh Jan 30, 2019

maheshk114 Jan 30, 2019

sankarh Jan 30, 2019 •

edited

Loading

maheshk114 Jan 30, 2019

sankarh Jan 30, 2019 •

edited

Loading

maheshk114 Jan 30, 2019

sankarh Jan 30, 2019

maheshk114 Jan 30, 2019

sankarh Jan 30, 2019 •

edited

Loading

maheshk114 Jan 31, 2019

sankarh Jan 31, 2019

maheshk114 Jan 30, 2019

sankarh Jan 30, 2019

maheshk114 Jan 30, 2019

sankarh Jan 30, 2019

maheshk114 Jan 30, 2019

sankarh Jan 30, 2019

maheshk114 Jan 30, 2019

sankarh Jan 30, 2019

maheshk114 Jan 30, 2019

sankarh Jan 30, 2019

HIVE-21029: External table replication for existing deployments running incremental replication. #523

HIVE-21029: External table replication for existing deployments running incremental replication. #523

Conversation

sankarh commented Jan 29, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sankarh Jan 30, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sankarh Jan 30, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sankarh Jan 30, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sankarh Jan 30, 2019 •

edited

Loading

sankarh Jan 30, 2019 •

edited

Loading

sankarh Jan 30, 2019 •

edited

Loading