ycsb cassandra fails on GCE

[vladzcloudius]

How to reproduce the issue ?

Run cassandra on osv vm with 4 cpu and 3.8 G memory.
Run ycsb from Linux vm (https://github.com/cloudius-systems/osv/wiki/Benchmarking-Cassandra-and-other-NoSQL-databases-with-YCSB).
I usually ran workloadf with -threads 100 -p operationcount=1000000 -p recordcount=800000 so my load command line looks like that:
./bin/ycsb load cassandra-10 -threads 100 -p operationcount=1000000 -p recordcount=800000 -p hosts=osv-vm-ip -P workloads/workloadf -s

Then one will see that ops per second become 0 and shortly the client will fail with a "broken pipe" error.

[dorlaor]

How does it fails?
Could it be there is not enough memory? Try w/ larger guest

On Thu, Jul 31, 2014 at 5:40 PM, vladzcloudius notifications@github.com
wrote:

1. How to reproduce the issue ?
   Run cassandra on osv vm with 4 cpu and 3.8 G memory.
   Run ycsb from Linux vm (
   https://github.com/cloudius-systems/osv/wiki/Benchmarking-Cassandra-and-other-NoSQL-databases-with-YCSB
   ).
   I usually ran workloadf with -threads 100 -p operationcount=1000000 -p
   recordcount=800000 so my load command line
   looks like that:
   ./bin/ycsb load cassandra-10 -threads 100 -p operationcount=1000000 -p
   recordcount=800000 -p hosts=osv-vm-ip -P workloads/workloadf -s

After a while one will see the

—
Reply to this email directly or view it on GitHub
#429.

[vladzcloudius]

It's been noticed that this test blows off the Guest storage quite quickly (in about 1100-1800 seconds of the test) and eventually causes the following cassandra assert:

ERROR 12:06:11,457 Exception in thread Thread[CompactionExecutor:18,1,main]
FSWriteError in /var/lib/cassandra/data/usertable/data/usertable-data-tmp-jb-218-Filter.db
at org.apache.cassandra.io.sstable.SSTableWriter$IndexWriter.close(SSTableWriter.java:478)
at org.apache.cassandra.io.util.FileUtils.closeQuietly(FileUtils.java:212)
at org.apache.cassandra.io.sstable.SSTableWriter.abort(SSTableWriter.java:304)
at org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:209)
at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60)
at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:198)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: java.io.FileNotFoundException: /var/lib/cassandra/data/usertable/data/usertable-data-tmp-jb-218-Filter.db (No space left on device)
at java.io.FileOutputStream.open(Native Method)
at java.io.FileOutputStream.(FileOutputStream.java:221)
at java.io.FileOutputStream.(FileOutputStream.java:110)
at org.apache.cassandra.io.sstable.SSTableWriter$IndexWriter.close(SSTableWriter.java:469)
... 13 more

So, this issue is a false alarm.