BnP not working #420
Comments
|
Okay, so I just picked up the warning about fetcher class assuming local file. Fixed that in (I can provide more context if necessary.) ...turns out that you must add this to readonly.keytab.path=
readonly.kerberos.user= |
|
For posterity, here are the configurations that worked for me-- BnP config: type=java
job.class=voldemort.store.readonly.mr.azkaban.VoldemortBuildAndPushJob
avro.key.field=name
avro.value.field=content
build.input.path=/user/hadoop/avro-data/
build.output.dir=/tmp
build.replication.factor=2
build.type.avro=true
push.cluster=tcp://localhost:6666
push.store.name=test1
push.store.description="Testing avro build and push"
push.store.owners=myemail@myworkplace.comExample node # The ID of *this* particular cluster node
node.id=0
max.threads=100
############### DB options ######################
http.enable=true
socket.enable=true
#NIO connector settings.
enable.nio.connector=true
request.format=vp3
storage.configs=voldemort.store.readonly.ReadOnlyStorageConfiguration
enable.readonly.engine=true
bdb.enable=false
slop.enable=false
# This is the important part
file.fetcher.class=voldemort.store.readonly.fetcher.HdfsFetcher
readonly.hadoop.config.path=/etc/hadoop/conf
readonly.keytab.path=
readonly.kerberos.user=Example node <store>
<name>test1</name>
<description>Testing avro build and push</description>
<owners>myemail@myworkplace.com</owners>
<persistence>read-only</persistence>
<routing-strategy>consistent-routing</routing-strategy>
<routing>client</routing>
<replication-factor>2</replication-factor>
<required-reads>1</required-reads>
<required-writes>1</required-writes>
<key-serializer>
<type>avro-generic</type>
<schema-info version="0">{"type":"string","avro.java.string":"String"}</schema-info>
</key-serializer>
<value-serializer>
<type>avro-generic</type>
<schema-info version="0">{"type":"bytes"}</schema-info>
</value-serializer>
</store> |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Hey guys,
I'm trying to run BnP on AWS EMR. My input data is Avro where the
namefield has the key andcontenthas the value.For testing purposes, I've got a tiny EMR cluster (1 master, 2 nodes). The master also runs Voldemort with
config/single_node_cluster. Here's my BnP config:Here's the output from running
run-bnp.sh:As you can see, it boils down to:
Looks like it mixes up between
/tmpin the local FS and HDFS.Have you seen this before? Any ideas on how to get past it?
Thanks!
The text was updated successfully, but these errors were encountered: