Cassandra 5234 PIG CQL3 support #3

Closed
wants to merge 2 commits into
from

Conversation

Projects
None yet
3 participants
@alexliu68
Owner

alexliu68 commented May 30, 2013

No description provided.

@jschappet

This comment has been minimized.

Show comment
Hide comment
@jschappet

jschappet May 31, 2013

Alex, thanks for this work. Between this and the Hadoop changes, DSP-1954, I have hope, I will be able to Use CQL3 Tables with Hadoop and PIG.

Any idea if these will get promoted to trunk?

I have downloaded, and run the Hadoop Example, work almost right out of the box. Just a little change to the readme will fix it, keyspace name is: cql3_worldcount, not wordcount.

Alex, thanks for this work. Between this and the Hadoop changes, DSP-1954, I have hope, I will be able to Use CQL3 Tables with Hadoop and PIG.

Any idea if these will get promoted to trunk?

I have downloaded, and run the Hadoop Example, work almost right out of the box. Just a little change to the readme will fix it, keyspace name is: cql3_worldcount, not wordcount.

@alexliu68

This comment has been minimized.

Show comment
Hide comment
@alexliu68

alexliu68 May 31, 2013

Owner

James,

We are working on get them merged into 1.2.6 release. It may take a few weeks for the worst case.

Thanks for the feedback.

Alex


From: James Schappet notifications@github.com
To: alexliu68/cassandra cassandra@noreply.github.com
Cc: Alex Liu alex_liu68@yahoo.com
Sent: Friday, May 31, 2013 10:03 AM
Subject: Re: [cassandra] Cassandra 5234 PIG CQL3 support (#3)

Alex, thanks for this work. Between this and the Hadoop changes, DSP-1954, I have hope, I will be able to Use CQL3 Tables with Hadoop and PIG.
Any idea if these will get promoted to trunk?
I have downloaded, and run the Hadoop Example, work almost right out of the box. Just a little change to the readme will fix it, keyspace name is: cql3_worldcount, not wordcount.

Reply to this email directly or view it on GitHub.

Owner

alexliu68 commented May 31, 2013

James,

We are working on get them merged into 1.2.6 release. It may take a few weeks for the worst case.

Thanks for the feedback.

Alex


From: James Schappet notifications@github.com
To: alexliu68/cassandra cassandra@noreply.github.com
Cc: Alex Liu alex_liu68@yahoo.com
Sent: Friday, May 31, 2013 10:03 AM
Subject: Re: [cassandra] Cassandra 5234 PIG CQL3 support (#3)

Alex, thanks for this work. Between this and the Hadoop changes, DSP-1954, I have hope, I will be able to Use CQL3 Tables with Hadoop and PIG.
Any idea if these will get promoted to trunk?
I have downloaded, and run the Hadoop Example, work almost right out of the box. Just a little change to the readme will fix it, keyspace name is: cql3_worldcount, not wordcount.

Reply to this email directly or view it on GitHub.

@lsimoneau

This comment has been minimized.

Show comment
Hide comment
@lsimoneau

lsimoneau Jun 12, 2013

Trying to test this, as we'd like to start using Cassandra with CQL3 and Pig so we want to get familiar with the integration as soon as possible. But checking out the branch and running ant results in a build error for me:

build-project:
     [echo] apache-cassandra: /home/vagrant/cassandra/build.xml
    [javac] Compiling 41 source files to /home/vagrant/cassandra/build/classes/thrift
    [javac] Note: Some input files use unchecked or unsafe operations.
    [javac] Note: Recompile with -Xlint:unchecked for details.
    [javac] Compiling 447 source files to /home/vagrant/cassandra/build/classes/main
    [javac] /home/vagrant/cassandra/src/java/org/apache/cassandra/io/util/Memory.java:20: warning: Unsafe is internal proprietary API and may be removed in a future release
    [javac] import sun.misc.Unsafe;
    [javac]                ^
    [javac] /home/vagrant/cassandra/src/java/org/apache/cassandra/hadoop/cql3/ColumnFamilyRecordReader.java:514: error: incompatible types
    [javac]             return result = null ? "" : result;
    [javac]                             ^
    [javac]   required: boolean
    [javac]   found:    <null>
    [javac] /home/vagrant/cassandra/src/java/org/apache/cassandra/io/util/Memory.java:29: warning: Unsafe is internal proprietary API and may be removed in a future release
    [javac]     private static final Unsafe unsafe;
    [javac]                          ^
    [javac] /home/vagrant/cassandra/src/java/org/apache/cassandra/io/util/Memory.java:35: warning: Unsafe is internal proprietary API and may be removed in a future release
    [javac]             Field field = sun.misc.Unsafe.class.getDeclaredField("theUnsafe");
    [javac]                                   ^
    [javac] /home/vagrant/cassandra/src/java/org/apache/cassandra/io/util/Memory.java:37: warning: Unsafe is internal proprietary API and may be removed in a future release
    [javac]             unsafe = (sun.misc.Unsafe) field.get(null);
    [javac]                               ^
    [javac] Note: Some input files use or override a deprecated API.
    [javac] Note: Recompile with -Xlint:deprecation for details.
    [javac] Note: Some input files use unchecked or unsafe operations.
    [javac] Note: Recompile with -Xlint:unchecked for details.
    [javac] 1 error
    [javac] 4 warnings

Am I missing something? Or should I just wait for the 1.2.6 release?

Trying to test this, as we'd like to start using Cassandra with CQL3 and Pig so we want to get familiar with the integration as soon as possible. But checking out the branch and running ant results in a build error for me:

build-project:
     [echo] apache-cassandra: /home/vagrant/cassandra/build.xml
    [javac] Compiling 41 source files to /home/vagrant/cassandra/build/classes/thrift
    [javac] Note: Some input files use unchecked or unsafe operations.
    [javac] Note: Recompile with -Xlint:unchecked for details.
    [javac] Compiling 447 source files to /home/vagrant/cassandra/build/classes/main
    [javac] /home/vagrant/cassandra/src/java/org/apache/cassandra/io/util/Memory.java:20: warning: Unsafe is internal proprietary API and may be removed in a future release
    [javac] import sun.misc.Unsafe;
    [javac]                ^
    [javac] /home/vagrant/cassandra/src/java/org/apache/cassandra/hadoop/cql3/ColumnFamilyRecordReader.java:514: error: incompatible types
    [javac]             return result = null ? "" : result;
    [javac]                             ^
    [javac]   required: boolean
    [javac]   found:    <null>
    [javac] /home/vagrant/cassandra/src/java/org/apache/cassandra/io/util/Memory.java:29: warning: Unsafe is internal proprietary API and may be removed in a future release
    [javac]     private static final Unsafe unsafe;
    [javac]                          ^
    [javac] /home/vagrant/cassandra/src/java/org/apache/cassandra/io/util/Memory.java:35: warning: Unsafe is internal proprietary API and may be removed in a future release
    [javac]             Field field = sun.misc.Unsafe.class.getDeclaredField("theUnsafe");
    [javac]                                   ^
    [javac] /home/vagrant/cassandra/src/java/org/apache/cassandra/io/util/Memory.java:37: warning: Unsafe is internal proprietary API and may be removed in a future release
    [javac]             unsafe = (sun.misc.Unsafe) field.get(null);
    [javac]                               ^
    [javac] Note: Some input files use or override a deprecated API.
    [javac] Note: Recompile with -Xlint:deprecation for details.
    [javac] Note: Some input files use unchecked or unsafe operations.
    [javac] Note: Recompile with -Xlint:unchecked for details.
    [javac] 1 error
    [javac] 4 warnings

Am I missing something? Or should I just wait for the 1.2.6 release?

@alexliu68

This comment has been minimized.

Show comment
Hide comment
@alexliu68

alexliu68 Jun 13, 2013

Owner

I will post the latest patch based on CASSANDRA-4421 latest changes soon. The patch is a little out of date.

Owner

alexliu68 commented Jun 13, 2013

I will post the latest patch based on CASSANDRA-4421 latest changes soon. The patch is a little out of date.

@alexliu68 alexliu68 closed this Jun 13, 2013

@alexliu68

This comment has been minimized.

Show comment
Hide comment
@alexliu68

alexliu68 Jun 14, 2013

Owner

@lsimoneau I have posted the latest patch for cassandra-1.2 branch to CASSANDRA-5234 ticket, you can apply the patch to cassandra-1.2 branch and test it.

Owner

alexliu68 commented Jun 14, 2013

@lsimoneau I have posted the latest patch for cassandra-1.2 branch to CASSANDRA-5234 ticket, you can apply the patch to cassandra-1.2 branch and test it.

@jschappet

This comment has been minimized.

Show comment
Hide comment
@jschappet

jschappet Jul 5, 2013

@alexliu68 I have downloaded the latest version of Source. It builds and runs just fine. However when running the the pig cql://, I am getting a NoSuchElementException.

For example:
CC = LOAD 'cql://PigTest/Bytes' USING CqlStorage();
dump CC;

Pig Stack Trace

ERROR 1066: Unable to open iterator for alias CC

org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to open iterator for alias CC
at org.apache.pig.PigServer.openIterator(PigServer.java:857)
at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:682)
at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:303)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:189)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:165)
at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:84)
at org.apache.pig.Main.run(Main.java:555)
at org.apache.pig.Main.main(Main.java:111)
Caused by: java.util.NoSuchElementException
at java.util.ArrayList$Itr.next(ArrayList.java:794)
at org.apache.cassandra.hadoop.pig.AbstractCassandraStorage.getKeysMeta(AbstractCassandraStorage.java:690)
at org.apache.cassandra.hadoop.pig.CqlStorage.getColumnMetadata(CqlStorage.java:369)
at org.apache.cassandra.hadoop.pig.AbstractCassandraStorage.getCfDef(AbstractCassandraStorage.java:559)
at org.apache.cassandra.hadoop.pig.AbstractCassandraStorage.initSchema(AbstractCassandraStorage.java:448)
at org.apache.cassandra.hadoop.pig.CqlStorage.setLocation(CqlStorage.java:164)
at org.apache.cassandra.hadoop.pig.CqlStorage.getSchema(CqlStorage.java:198)
at org.apache.pig.newplan.logical.relational.LOLoad.getSchemaFromMetaData(LOLoad.java:151)
at org.apache.pig.newplan.logical.relational.LOLoad.getSchema(LOLoad.java:110)
at org.apache.pig.newplan.logical.relational.LOStore.getSchema(LOStore.java:68)
at org.apache.pig.newplan.logical.visitor.SchemaAliasVisitor.validate(SchemaAliasVisitor.java:60)
at org.apache.pig.newplan.logical.visitor.SchemaAliasVisitor.visit(SchemaAliasVisitor.java:84)
at org.apache.pig.newplan.logical.relational.LOStore.accept(LOStore.java:77)
at org.apache.pig.newplan.DependencyOrderWalker.walk(DependencyOrderWalker.java:75)
at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:50)
at org.apache.pig.PigServer$Graph.compile(PigServer.java:1617)
at org.apache.pig.PigServer$Graph.compile(PigServer.java:1611)
at org.apache.pig.PigServer$Graph.access$200(PigServer.java:1334)
at org.apache.pig.PigServer.storeEx(PigServer.java:951)
at org.apache.pig.PigServer.store(PigServer.java:919)
at org.apache.pig.PigServer.openIterator(PigServer.java:832)
... 7 more

@alexliu68 I have downloaded the latest version of Source. It builds and runs just fine. However when running the the pig cql://, I am getting a NoSuchElementException.

For example:
CC = LOAD 'cql://PigTest/Bytes' USING CqlStorage();
dump CC;

Pig Stack Trace

ERROR 1066: Unable to open iterator for alias CC

org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to open iterator for alias CC
at org.apache.pig.PigServer.openIterator(PigServer.java:857)
at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:682)
at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:303)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:189)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:165)
at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:84)
at org.apache.pig.Main.run(Main.java:555)
at org.apache.pig.Main.main(Main.java:111)
Caused by: java.util.NoSuchElementException
at java.util.ArrayList$Itr.next(ArrayList.java:794)
at org.apache.cassandra.hadoop.pig.AbstractCassandraStorage.getKeysMeta(AbstractCassandraStorage.java:690)
at org.apache.cassandra.hadoop.pig.CqlStorage.getColumnMetadata(CqlStorage.java:369)
at org.apache.cassandra.hadoop.pig.AbstractCassandraStorage.getCfDef(AbstractCassandraStorage.java:559)
at org.apache.cassandra.hadoop.pig.AbstractCassandraStorage.initSchema(AbstractCassandraStorage.java:448)
at org.apache.cassandra.hadoop.pig.CqlStorage.setLocation(CqlStorage.java:164)
at org.apache.cassandra.hadoop.pig.CqlStorage.getSchema(CqlStorage.java:198)
at org.apache.pig.newplan.logical.relational.LOLoad.getSchemaFromMetaData(LOLoad.java:151)
at org.apache.pig.newplan.logical.relational.LOLoad.getSchema(LOLoad.java:110)
at org.apache.pig.newplan.logical.relational.LOStore.getSchema(LOStore.java:68)
at org.apache.pig.newplan.logical.visitor.SchemaAliasVisitor.validate(SchemaAliasVisitor.java:60)
at org.apache.pig.newplan.logical.visitor.SchemaAliasVisitor.visit(SchemaAliasVisitor.java:84)
at org.apache.pig.newplan.logical.relational.LOStore.accept(LOStore.java:77)
at org.apache.pig.newplan.DependencyOrderWalker.walk(DependencyOrderWalker.java:75)
at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:50)
at org.apache.pig.PigServer$Graph.compile(PigServer.java:1617)
at org.apache.pig.PigServer$Graph.compile(PigServer.java:1611)
at org.apache.pig.PigServer$Graph.access$200(PigServer.java:1334)
at org.apache.pig.PigServer.storeEx(PigServer.java:951)
at org.apache.pig.PigServer.store(PigServer.java:919)
at org.apache.pig.PigServer.openIterator(PigServer.java:832)
... 7 more

@alexliu68

This comment has been minimized.

Show comment
Hide comment
@alexliu68

alexliu68 Jul 5, 2013

Owner

Can you test it by checkout cassandra-1.2 branch from https://github.com/apache/cassandra? Please raise the issue if it occurs in that branch.

This branch is a little old.

Owner

alexliu68 commented Jul 5, 2013

Can you test it by checkout cassandra-1.2 branch from https://github.com/apache/cassandra? Please raise the issue if it occurs in that branch.

This branch is a little old.

@jschappet

This comment has been minimized.

Show comment
Hide comment
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment