New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Writebatch call crashes #262

Closed
naveenatceg opened this Issue Sep 4, 2014 · 4 comments

Comments

Projects
None yet
5 participants
@naveenatceg

naveenatceg commented Sep 4, 2014

Hi,
I have been trying WriteBatch method to do puts, I keep running into these two types of crashes when I try to use writeBatch (everything works fine when I do just puts instead of writebatch).

org.rocksdb.RocksDBException: Corruption: unknown WriteBatch tag
    at org.rocksdb.RocksDB.write(Native Method)
    at org.rocksdb.RocksDB.write(RocksDB.java:187)
    at org.apache.samza.storage.kv.RocksDbKeyValueStore.putAll(RocksDbKeyValueStore.scala:100)
    at org.apache.samza.storage.kv.SerializedKeyValueStore.putAll(SerializedKeyValueStore.scala:57)
    at org.apache.samza.storage.kv.CachedStore.flush(CachedStore.scala:162)
    at org.apache.samza.storage.kv.CachedStore.put(CachedStore.scala:146)
    at org.apache.samza.storage.kv.NullSafeKeyValueStore.put(NullSafeKeyValueStore.scala:42)
    at org.apache.samza.storage.kv.KeyValueStorageEngine.put(KeyValueStorageEngine.scala:49)
    at com.linkedin.samza.test.perf.TestStoreAllPerformance.process(TestStoreAllPerformance.java:41)
    at org.apache.samza.container.TaskInstance.process(TaskInstance.scala:133)
    at org.apache.samza.container.RunLoop$$anonfun$process$2.apply(RunLoop.scala:93)
    at org.apache.samza.util.TimerUtils$class.updateTimer(TimerUtils.scala:37)
    at org.apache.samza.container.RunLoop.updateTimer(RunLoop.scala:36)
    at org.apache.samza.container.RunLoop.process(RunLoop.scala:79)
    at org.apache.samza.container.RunLoop.run(RunLoop.scala:65)
    at org.apache.samza.container.SamzaContainer.run(SamzaContainer.scala:553)
    at org.apache.samza.container.SamzaContainer$.safeMain(SamzaContainer.scala:106)
    at org.apache.samza.container.SamzaContainer$.main(SamzaContainer.scala:85)
    at org.apache.samza.container.SamzaContainer.main(SamzaContainer.scala)
java version "1.8.0_05"
Java(TM) SE Runtime Environment (build 1.8.0_05-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.5-b02, mixed mode)
Exception in thread "main" org.rocksdb.RocksDBException: Corruption: malformed WriteBatch (too small)
    at org.rocksdb.RocksDB.write(Native Method)
    at org.rocksdb.RocksDB.write(RocksDB.java:187)
    at org.apache.samza.storage.kv.RocksDbKeyValueStore.putAll(RocksDbKeyValueStore.scala:100)
    at org.apache.samza.storage.kv.SerializedKeyValueStore.putAll(SerializedKeyValueStore.scala:57)
    at org.apache.samza.storage.kv.CachedStore.flush(CachedStore.scala:162)
    at org.apache.samza.storage.kv.NullSafeKeyValueStore.flush(NullSafeKeyValueStore.scala:69)
    at org.apache.samza.storage.kv.KeyValueStorageEngine.flush(KeyValueStorageEngine.scala:113)
    at org.apache.samza.storage.kv.KeyValueStorageEngine.close(KeyValueStorageEngine.scala:125)
    at org.apache.samza.storage.kv.KeyValueStorageEngine.stop(KeyValueStorageEngine.scala:119)
    at org.apache.samza.storage.TaskStorageManager$$anonfun$stop$2.apply(TaskStorageManager.scala:132)
    at org.apache.samza.storage.TaskStorageManager$$anonfun$stop$2.apply(TaskStorageManager.scala:132)
    at scala.collection.Iterator$class.foreach(Iterator.scala:727)
    at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)

source code on how WriteBatch is used:

  def putAll(entries: java.util.List[Entry[Array[Byte], Array[Byte]]]) {
    val batch = new WriteBatch()
    val iter = entries.iterator
    var wrote = 0
    var deletes = 0
    while (iter.hasNext) {
      wrote += 1
      val curr = iter.next()
      if (curr.getValue == null) {
        deletes += 1
        batch.remove(curr.getKey)
      } else {
        val key = curr.getKey
        val value = curr.getValue
        metrics.bytesWritten.inc(key.size + value.size)
        batch.put(key, value)
      }
    }
    db.write(new WriteOptions(), batch)
    metrics.puts.inc(wrote)
    metrics.deletes.inc(deletes)
    deletesSinceLastCompaction += deletes
  }
@yhchiang

This comment has been minimized.

Show comment
Hide comment
@yhchiang

yhchiang Sep 5, 2014

Contributor

Hey naveenatceg, I found I am not able to reproduce such problem when I use RocksDB Java directly (as in this under-reviewed diff https://reviews.facebook.net/D22983). Can I first know whether you are able to run the RocksDBSample.java in D22983 properly?

Contributor

yhchiang commented Sep 5, 2014

Hey naveenatceg, I found I am not able to reproduce such problem when I use RocksDB Java directly (as in this under-reviewed diff https://reviews.facebook.net/D22983). Can I first know whether you are able to run the RocksDBSample.java in D22983 properly?

@dhruba

This comment has been minimized.

Show comment
Hide comment
@dhruba

dhruba Nov 2, 2014

Contributor

Hi Naveen, any new findings on this task?

Contributor

dhruba commented Nov 2, 2014

Hi Naveen, any new findings on this task?

@dhruba

This comment has been minimized.

Show comment
Hide comment
@dhruba

dhruba Jan 7, 2015

Contributor

Hi Naveen, any new information on this task?

Contributor

dhruba commented Jan 7, 2015

Hi Naveen, any new information on this task?

@igorcanadi

This comment has been minimized.

Show comment
Hide comment
@igorcanadi

igorcanadi Jan 24, 2015

Contributor

Closing due to inactivity. @naveenatceg feel free to reopen if you still need support from us.

Contributor

igorcanadi commented Jan 24, 2015

Closing due to inactivity. @naveenatceg feel free to reopen if you still need support from us.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment