-
Notifications
You must be signed in to change notification settings - Fork 123
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
GORA-443 Upgrade HBase to 1.2.0 #86
Conversation
import org.apache.hadoop.hbase.client.Result; | ||
import org.apache.hadoop.hbase.client.ResultScanner; | ||
import org.apache.hadoop.hbase.client.Scan; | ||
import org.apache.hadoop.hbase.client.*; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can you import individual classes?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For sure, I will do it for the final patch. There is still some clean up to be done ;)
what I'd like to fix is this weird behaviour of BufferedMutator and our tests.
I notice that we never |
After using admin, .close() method should be called. |
for (ConcurrentLinkedQueue<Mutation> buffer : bPool) { | ||
for (Mutation m: buffer) { | ||
bufMutator.mutate(m); | ||
bufMutator.flush(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This can be lifted outside the for loop.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@tedyu are you referring specifically to bufMutator.flush();
?
The code would therefore be
public void flushCommits() throws IOException {
+ BufferedMutator bufMutator = connection.getBufferedMutator(this.tableName);
+ for (ConcurrentLinkedQueue<Mutation> buffer : bPool) {
+ for (Mutation m: buffer) {
+ bufMutator.mutate(m);
+ }
+ }
+ bufMutator.flush();
+ bufMutator.close();
+ }
Can you please clarify?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes. This would do.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
OK thanks. I've pushed a PR to @renato2099 which I hope he will add to this one. You can see the PR at renato2099#1
@renato2099 can you please merge and push the combined PR to this one?
@renato2099 can you please merge and push renato2099#1 |
Work in progress for updating gora to 1.2.2. I also looked at @lewismc 's work ;)
All of tests pass when run individually, but when run as a whole some of them fail. I think this is because of the BufferedMutator and taking longer than it should. The usage of the BufferedMutator is like setting autoFlush=False which then makes sense to actually use our Flush method.
The weird thing is that if we use the regular Table and do the operations synchronously all tests pass always. This would be like setting autoFlush=True. The problem with using this option is that Gora wouldn't be batching any operations and just passing them down.
Maybe someone has an idea on how to check for BufferedMutator to be done (?)