GORA-443 Upgrade HBase to 1.2.0 #86
Conversation
| import org.apache.hadoop.hbase.client.Result; | ||
| import org.apache.hadoop.hbase.client.ResultScanner; | ||
| import org.apache.hadoop.hbase.client.Scan; | ||
| import org.apache.hadoop.hbase.client.*; |
There was a problem hiding this comment.
Can you import individual classes?
There was a problem hiding this comment.
For sure, I will do it for the final patch. There is still some clean up to be done ;)
what I'd like to fix is this weird behaviour of BufferedMutator and our tests.
|
I notice that we never |
|
After using admin, .close() method should be called. |
| for (ConcurrentLinkedQueue<Mutation> buffer : bPool) { | ||
| for (Mutation m: buffer) { | ||
| bufMutator.mutate(m); | ||
| bufMutator.flush(); |
There was a problem hiding this comment.
This can be lifted outside the for loop.
There was a problem hiding this comment.
@tedyu are you referring specifically to bufMutator.flush();?
The code would therefore be
public void flushCommits() throws IOException {
+ BufferedMutator bufMutator = connection.getBufferedMutator(this.tableName);
+ for (ConcurrentLinkedQueue<Mutation> buffer : bPool) {
+ for (Mutation m: buffer) {
+ bufMutator.mutate(m);
+ }
+ }
+ bufMutator.flush();
+ bufMutator.close();
+ }
Can you please clarify?
There was a problem hiding this comment.
OK thanks. I've pushed a PR to @renato2099 which I hope he will add to this one. You can see the PR at renato2099#1
@renato2099 can you please merge and push the combined PR to this one?
|
@renato2099 can you please merge and push renato2099#1 |
Work in progress for updating gora to 1.2.2. I also looked at @lewismc 's work ;)
All of tests pass when run individually, but when run as a whole some of them fail. I think this is because of the BufferedMutator and taking longer than it should. The usage of the BufferedMutator is like setting autoFlush=False which then makes sense to actually use our Flush method.
The weird thing is that if we use the regular Table and do the operations synchronously all tests pass always. This would be like setting autoFlush=True. The problem with using this option is that Gora wouldn't be batching any operations and just passing them down.
Maybe someone has an idea on how to check for BufferedMutator to be done (?)