Using TimestampSkipping iterator in RollbackCheckIterator #962
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
@llvieira there is a bit more that needs to be done for this issue. In the while loop in the seek method, need to use the timestampskipping iterator to jump ahead. When the colType is TX_DONE, can do the following : if (colType == ColumnConstants.TX_DONE_PREFIX) {
# jump to WRITE skipping all TX_DONE
source.skipToPrefix(curCol, ColumnConstants.WRITE_PREFIX);
} else if (colType == ColumnConstants.WRITE_PREFIX) {For The background on this is that the column prefixs are in sorted order (descending sort of signed long and the prefix is the 1st three bits). There could be lots of column prefixes and once we determine that nothing else in the current prefix is useful then we can seek to the next possible prefix. This has the potential to seek over a lot of prefixes. The code is currently doing a linear scan of all column prefixes. |
|
@keith-turner thanks for the help and for the clarification. I made the required changes, can you see if it is ok? Now I saw that the changes caused some issues, but before I submit the commit, I ran the tests locally and all worked fine I guess. I still don't know why this happened. |
|
@llvieira I figured out the problem. Calling |
|
Oh yeah, I'll fix it. |
|
Now I'm having a non-deterministic behavior when I run the tests, because locally all tests run successfully but in the tarvis an error is raised. Tarvis: Locally: Any ideas on what's happening, @keith-turner? |
|
Still need call to source.next. only call source.next when not calling
skip. Could exec continue after skip.
Sent from phone. Please excuse typos and brevity.
…On Oct 27, 2017 9:26 PM, "Lucas Lima Vieira" ***@***.***> wrote:
Now I'm having a non-deterministic behavior when I ran the tests, because
locally all tests run successfully but in the *tarvis* an error is raised.
Tarvis:
Running org.apache.fluo.integration.impl.TransactorIT
2017-10-28 00:52:28,350 [imps.CuratorFrameworkImpl] ERROR: Background exception was not retry-able or retry gave up
java.lang.IllegalStateException: Client is not started
at com.google.common.base.Preconditions.checkState(Preconditions.java:149)
at org.apache.curator.CuratorZookeeperClient.getZooKeeper(CuratorZookeeperClient.java:113)
at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:806)
at org.apache.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:490)
at org.apache.curator.framework.imps.CreateBuilderImpl.pathInBackground(CreateBuilderImpl.java:663)
at org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:439)
at org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:44)
at org.apache.curator.framework.recipes.nodes.PersistentEphemeralNode.createNode(PersistentEphemeralNode.java:340)
at org.apache.curator.framework.recipes.nodes.PersistentEphemeralNode.access$000(PersistentEphemeralNode.java:53)
at org.apache.curator.framework.recipes.nodes.PersistentEphemeralNode$1.process(PersistentEphemeralNode.java:70)
at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:61)
at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505)
2017-10-28 00:52:28,352 [imps.CuratorFrameworkImpl] ERROR: Background exception was not retry-able or retry gave up
java.lang.IllegalStateException: Client is not started
at com.google.common.base.Preconditions.checkState(Preconditions.java:149)
at org.apache.curator.CuratorZookeeperClient.getZooKeeper(CuratorZookeeperClient.java:113)
at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:806)
at org.apache.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:490)
at org.apache.curator.framework.imps.CreateBuilderImpl.pathInBackground(CreateBuilderImpl.java:663)
at org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:439)
at org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:44)
at org.apache.curator.framework.recipes.nodes.PersistentEphemeralNode.createNode(PersistentEphemeralNode.java:340)
at org.apache.curator.framework.recipes.nodes.PersistentEphemeralNode.access$000(PersistentEphemeralNode.java:53)
at org.apache.curator.framework.recipes.nodes.PersistentEphemeralNode$1.process(PersistentEphemeralNode.java:70)
at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:61)
at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505)
Tests run: 4, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 29.869 sec <<< FAILURE! - in org.apache.fluo.integration.impl.TransactorIT
testTransactorAndCache(org.apache.fluo.integration.impl.TransactorIT) Time elapsed: 5.66 sec <<< FAILURE!
java.lang.AssertionError
at org.apache.fluo.integration.impl.TransactorIT.testTransactorAndCache(TransactorIT.java:64)
Locally:
[INFO] Reactor Summary:
[INFO]
[INFO] Apache Fluo Project ................................ SUCCESS [ 14.261 s]
[INFO] Apache Fluo API .................................... SUCCESS [ 21.973 s]
[INFO] Apache Fluo Accumulo Extensions .................... SUCCESS [ 17.406 s]
[INFO] Apache Fluo Core ................................... SUCCESS [ 27.737 s]
[INFO] Apache Fluo Cluster ................................ SUCCESS [ 11.530 s]
[INFO] Apache Fluo Command ................................ SUCCESS [ 7.784 s]
[INFO] Apache Fluo Mini ................................... SUCCESS [ 5.707 s]
[INFO] Apache Fluo Integration ............................ SUCCESS [18:11 min]
[INFO] Apache Fluo MapReduce Utilities .................... SUCCESS [ 48.874 s]
[INFO] Apache Fluo Distribution ........................... SUCCESS [ 23.514 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 21:10 min
[INFO] Finished at: 2017-10-27T22:19:20-03:00
[INFO] Final Memory: 70M/428M
[INFO] ------------------------------------------------------------------------
Any ideas on what's happening, @keith-turner
<https://github.com/keith-turner>?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#962 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/ABNcA6YlPSCNmxxh_aShal301kvHR6Hfks5swoLBgaJpZM4QH4e4>
.
|
keith-turner
pushed a commit
that referenced
this pull request
Oct 30, 2017
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Closes #949. Using TimestampSkipping iterator on the RollbackCheckIterator to jump to later timestamps.