Permalink
Browse files

Adding uploadLocalFileIfChanged to allow HBase & ESearch jobs to run …

…in parallel
  • Loading branch information...
temujin9 committed Sep 7, 2011
1 parent 224926a commit 38a662f554ac20eda5898cabdc40c05230caae34
Showing with 2 additions and 2 deletions.
  1. +2 −2 src/main/java/com/infochimps/elasticsearch/pig/ElasticSearchStorage.java
@@ -256,10 +256,10 @@ private void elasticSearchSetup(String location, Job job) {
Path hdfsConfigPath = new Path(ES_CONFIG_HDFS_PATH);
Path hdfsPluginsPath = new Path(ES_PLUGINS_HDFS_PATH);
- HadoopUtils.uploadLocalFile(new Path(LOCAL_SCHEME+esConfig), hdfsConfigPath, job.getConfiguration());
+ HadoopUtils.uploadLocalFileIfChanged(new Path(LOCAL_SCHEME+esConfig), hdfsConfigPath, job.getConfiguration());
HadoopUtils.shipFileIfNotShipped(hdfsConfigPath, job.getConfiguration());
- HadoopUtils.uploadLocalFile(new Path(LOCAL_SCHEME+esPlugins), hdfsPluginsPath, job.getConfiguration());
+ HadoopUtils.uploadLocalFileIfChanged(new Path(LOCAL_SCHEME+esPlugins), hdfsPluginsPath, job.getConfiguration());
HadoopUtils.shipArchiveIfNotShipped(hdfsPluginsPath, job.getConfiguration());
} catch (Exception e) {

0 comments on commit 38a662f

Please sign in to comment.