Peon cannot write segments to GCS but can write index logs

[dgregoire]

Peon cannot write segments to GCS but can write index logs.

Affected Version

0.14.0-incubating

The Druid version where the problem was encountered.

Description

At the end of a hadoop indexation job the peon errors out but then the middle manager is able to write the indexation logs.

2019-04-21T04:53:36,122 INFO [forking-task-runner-1-[index_hadoop_supply_2019-04-21T04:52:45.180Z]] org.apache.druid.indexing.overlord.ForkingTaskRunner - Process exited with status[0] for task: index_hadoop_supply_2019-04-21T04:52:45.180Z
2019-04-21T04:53:36,123 INFO [forking-task-runner-1] org.apache.druid.storage.hdfs.tasklog.HdfsTaskLogs - Writing task log to: gs://druid-XXXXXXXX/indexing-logs/index_hadoop_supply_2019-04-21T04_52_45.180Z
2019-04-21T04:53:37,445 INFO [forking-task-runner-1] org.apache.druid.storage.hdfs.tasklog.HdfsTaskLogs - Wrote task log to: gs://druid-XXXXXXXX/indexing-logs/index_hadoop_supply_2019-04-21T04_52_45.180Z
2019-04-21T04:53:37,445 INFO [forking-task-runner-1] org.apache.druid.storage.hdfs.tasklog.HdfsTaskLogs - Writing task reports to: gs://druid-XXXXXXXX/indexing-logs/index_hadoop_supply_2019-04-21T04_52_45.180Z.reports.json
2019-04-21T04:53:38,651 INFO [forking-task-runner-1] org.apache.druid.storage.hdfs.tasklog.HdfsTaskLogs - Wrote task reports to: gs://druid-XXXXXXXX/indexing-logs/index_hadoop_supply_2019-04-21T04_52_45.180Z.reports.json
2019-04-21T04:53:38,652 INFO [forking-task-runner-1] org.apache.druid.indexing.overlord.TaskRunnerUtils - Task [index_hadoop_supply_2019-04-21T04:52:45.180Z] status changed to [FAILED].
2019-04-21T04:53:38,653 INFO [forking-task-runner-1] org.apache.druid.indexing.overlord.ForkingTaskRunner - Removing task directory: var/druid/task/index_hadoop_supply_2019-04-21T04:52:45.180Z
2019-04-21T04:53:38,666 INFO [WorkerTaskManager-NoticeHandler] org.apache.druid.indexing.worker.WorkerTaskManager - Job's finished. Completed [index_hadoop_supply_2019-04-21T04:52:45.180Z] with status [FAILED]

Peon is definitely producing output

2019-04-21T02:32:00,232 INFO [pool-34-thread-1] org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter - Saved output of task 'attempt_local152430381_0001_r_001394_0' to file:/var/druid/hadoop-tmp/supply/2019-04-21T023121.679Z_d3d8ec21cead4e3ea61a9e05af02f36e/_temporary/0/task_local152430381_0001_r_001394

but finishes with this error

Could not find job job_local812015859_0001
Finished peon task

Middle manager config looks like this:

druid.service=druid/middleManager
druid.port=8091

# Number of tasks per middleManager
druid.worker.capacity=4

# Task launch parameters
druid.indexer.runner.javaOpts=-server -Xmx4g -XX:MaxDirectMemorySize=3g -Duser.timezone=UTC -Dfile.encoding=UTF-8 -XX:+ExitOnOutOfMemoryError -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager
druid.indexer.task.baseTaskDir=var/druid/task

# HTTP server threads
druid.server.http.numThreads=25

# Processing threads and buffers on Peons
druid.indexer.fork.property.druid.processing.buffer.sizeBytes=536870912
druid.indexer.fork.property.druid.processing.numThreads=2

# Hadoop indexing
druid.indexer.task.hadoopWorkingPath=var/druid/hadoop-tmp

Is this a problem with the user (me) or some subtle path bug that I can't figure out?

[dgregoire]

was a configuration issue, closing

[stepanovD]

hello. I have same case. How you solved problem?

[dgregoire]

hello. I have same case. What you solved problem?

Using the version of the gcs hdfs connector that was recommended at that time (almost a year ago) did not work and I had switched to using the latest.

There's been developments more recently and there is a GCS extension now, perhaps things have changed?

I'm not working on this project anymore, but I have theses excerpts to share.

I was using this from a dockerfile to add the gcs connector.

# GCS connector
ADD https://storage.googleapis.com/hadoop-lib/gcs/gcs-connector-hadoop2-latest.jar /opt/druid/extensions/druid-hdfs-storage

core-site.xml placed in _common, I had a startup script replacing the KV of the keys GCS_PROJECT_ID and GCS_KEY_FILE.

<configuration>  
  <property>
    <name>fs.gs.impl</name>
    <value>com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem</value>
    <description>The FileSystem for gs: (GCS) uris.</description>
  </property>
  <property>
    <name>fs.AbstractFileSystem.gs.impl</name>
    <value>com.google.cloud.hadoop.fs.gcs.GoogleHadoopFS</value>
    <description>
      The AbstractFileSystem for gs: (GCS) uris. Only necessary for use with Hadoop 2.
    </description>
  </property>
  <property>
    <name>fs.gs.project.id</name>
    <value>GCS_PROJECT_ID</value>
    <description>
      Required. Google Cloud Project ID with access to configured GCS buckets.
    </description>
  </property>
  <property>
    <name>google.cloud.auth.service.account.enable</name>
    <value>true</value>
    <description>
      Whether to use a service account for GCS authorizaiton. If an email and
      keyfile are provided (see google.cloud.auth.service.account.email and
      google.cloud.auth.service.account.keyfile), then that service account
      willl be used. Otherwise the connector will look to see if it running on
      a GCE VM with some level of GCS access in it's service account scope, and
      use that service account.
    </description>
  </property>
  <property>
    <name>google.cloud.auth.service.account.json.keyfile</name>
    <value>GCS_KEY_FILE</value>
    <description>
      The JSON key file of the service account used for GCS
      access when google.cloud.auth.service.account.enable is true.
    </description>
  </property>
  <property>
    <name>hadoop.tmp.dir</name>
    <value>/var/druid</value>
  </property>
</configuration>

and common.runtime.properties including the hdfs storage extension.

# This is not the full list of Druid extensions, but common ones that people often use. You may need to change this list
# based on your particular setup.
druid.extensions.directory=/opt/druid/extensions
druid.extensions.loadList=["druid-histogram", "druid-datasketches", "mysql-metadata-storage", "druid-hdfs-storage", "druid-kafka-indexing-service", "druid-protobuf-extensions"]