[CAMEL-9145] update hbase version to 1.1.1 and hadoop2 to 2.7.1 #634

woj-i · 2015-10-09T14:28:34Z

Before the upgrade I was not able to run tests properly, because of an error during setup of HadoopMinicluster. In that case all tests pass, but are not started. I have a linux environment.

After the update I was able to run tests. There are 18 positive results and 2 negative in my environment. I cannot check how many negative tests were on the previous versions, because of the error mentioned at the beginning.

davsclaus · 2015-10-16T14:45:17Z

This branch has conflicts that must be resolved

Also do not use tabs for xml but spaces

and why is the osgi range for this starting from 2 and not from 1 as before?
org.apache.hadoop.conf;version="[2,3)",

woj-i · 2015-10-16T15:13:24Z

oh, I haven't noticed the conflict. I'll try to resolve it.
I changed the Hadoop version, because API for Hadoop2 is not compatible with Hadoop1.

Conflicts: parent/pom.xml

woj-i · 2015-10-16T15:43:58Z

I've made a merge and replaced all tabs with spaces.

I am not an expert of OSGi, so @davsclaus waiting for your opinion about it.

davsclaus · 2015-10-18T07:02:38Z

There is also a camel-hbase in the features.xml file. I wonder if that needs any adjustments? You can test it that works by running CamelHbaseTest in the tests/camel-itest-karaf directory.

davsclaus · 2015-10-19T05:54:27Z

The unit tests of camel-hbase fails, do they work for you?

Running org.apache.camel.component.hbase.CamelHBaseFilterTest
2015-10-19 07:54:40.106 java[26361:1903] Unable to load realm info from SCDynamicStore
Formatting using clusterid: testClusterID
Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 25.732 sec <<< FAILURE! - in org.apache.camel.component.hbase.CamelHBaseFilterTest
testPutMultiRowsAndScanWithFilters(org.apache.camel.component.hbase.CamelHBaseFilterTest)  Time elapsed: 15.796 sec  <<< FAILURE!
java.lang.AssertionError: null
    at org.junit.Assert.fail(Assert.java:86)
    at org.junit.Assert.assertTrue(Assert.java:41)
    at org.junit.Assert.assertTrue(Assert.java:52)
    at org.apache.camel.component.hbase.CamelHBaseFilterTest.testPutMultiRowsAndScanWithFilters(CamelHBaseFilterTest.java:58)

Running org.apache.camel.component.hbase.HBaseConsumerTest
2015-10-19 07:55:06.414 java[26383:1903] Unable to load realm info from SCDynamicStore
Formatting using clusterid: testClusterID
2015-10-19 07:55:44,852 [ hbase://person] WARN  HBaseConsumer                  - Consumer Consumer[hbase://person] failed polling endpoint: Endpoint[hbase://person]. Will try again at next poll. Caused by: [org.apache.hadoop.hbase.TableNotFoundException - Table 'person' was not found, got: hbase:namespace.]
org.apache.hadoop.hbase.TableNotFoundException: Table 'person' was not found, got: hbase:namespace.
    at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1274)
    at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1155)
    at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.relocateRegion(ConnectionManager.java:1126)
    at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:298)
    at org.apache.hadoop.hbase.client.ScannerCallable.prepare(ScannerCallable.java:149)
    at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.prepare(ScannerCallableWithReplicas.java:381)
    at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:124)
    at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:64)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:744)
Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 41.617 sec <<< FAILURE! - in org.apache.camel.component.hbase.HBaseConsumerTest
testPutMultiRowsAndConsume(org.apache.camel.component.hbase.HBaseConsumerTest)  Time elapsed: 31.779 sec  <<< FAILURE!
java.lang.AssertionError: mock://result Received message count. Expected: <3> but was: <0>
    at org.apache.camel.component.mock.MockEndpoint.fail(MockEndpoint.java:1433)
    at org.apache.camel.component.mock.MockEndpoint.assertEquals(MockEndpoint.java:1415)
    at org.apache.camel.component.mock.MockEndpoint.doAssertIsSatisfied(MockEndpoint.java:403)
    at org.apache.camel.component.mock.MockEndpoint.assertIsSatisfied(MockEndpoint.java:382)
    at org.apache.camel.component.mock.MockEndpoint.assertIsSatisfied(MockEndpoint.java:370)
    at org.apache.camel.component.hbase.HBaseConsumerTest.testPutMultiRowsAndConsume(HBaseConsumerTest.java:46)

Running org.apache.camel.component.hbase.HBaseConvertionsTest
2015-10-19 07:55:48.599 java[26405:1903] Unable to load realm info from SCDynamicStore
Formatting using clusterid: testClusterID
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 24.806 sec - in org.apache.camel.component.hbase.HBaseConvertionsTest
Running org.apache.camel.component.hbase.HBaseProducerTest
2015-10-19 07:56:14.055 java[26424:1903] Unable to load realm info from SCDynamicStore
Formatting using clusterid: testClusterID
...

davsclaus · 2015-10-19T06:01:41Z

I have upgraded camel-hdfs2 to the newer hadoop2 version. There was also a change needed for osgi users.

davsclaus · 2015-10-19T06:03:35Z

And btw looks like there is hbase 1.1.2 out

woj-i · 2015-10-19T11:39:55Z

Hi @davsclaus !
These unit-tests also don't work for me. In the previous version I wasn't able to run these tests and I think of I would be able to run it it would also not pass. Could you try to run tests on the old version?

I'll check karaf tests and also look at camel-hdfs2 project.

Right, HBase 1.1.2 is out, but currenly not attached to any distribution of Hadoop platform (neither Hortonworks nor Cloudera)

davsclaus · 2015-10-20T06:14:57Z

The tests works on the old version

woj-i · 2015-10-20T11:21:13Z

do you type just mvn test in camel-hbase directory? In my case such tests pass because minicluster cannot be started.

davsclaus · 2015-10-20T11:45:55Z

Ah sorry yeah those tests are skipped locally, would need a hbase cluster. I guess you need to setup that manually.

Though the test should pass running locally build or skip the tests in a maven setting in the pom.xml, and have a profile for running integration tests. There is some other components doing that, such as camel-twitter that has both unit tests and integration with a profile.

woj-i · 2015-10-21T09:24:13Z

Hi!
I am very curious why this test don't pass. I think there is a problem with hbase-consumer. I started to debugging it.

So far I spotted a inconsistency in documentation. There are two parameters of hbase-consumer in documentation http://camel.apache.org/hbase.html no longer available in the component: initialDelay, delay. When I tried to use it as a parameter of a hbase-consumer I had an error ("unrecognized parameter" or something).

Still debugging.

woj-i · 2015-10-21T12:11:17Z

In case of HBaseConsumerTest.testPutMultiRowsAndConsume
I can see puts are done without exception (and are in a table).
There is a problem with scan parameters in HBaseConsumer. Besides, there are some good practice to include in the scan, such as cache, etc.
@davsclaus, should I create another issue and pull request or could I work on this PR?

davsclaus · 2015-10-21T12:16:33Z

You are welcome to what you feel the best. If an update + this change is needed together then one PR is fine. We would like to upgrade hbase so eventually we would like it all sorted.

woj-i · 2015-10-21T14:11:52Z

In CamelHBaseFilterTest.createRouteBuilder I see a definition of scan operation with a param:
filters=#myFilters
There is a empty filter created, which cause the value of filter is weird: 0 elements and "MUST_PASS_ALL" rule. It is the reasons why the test fails (the fiter filters everything). I tried to replace this filter with SingleColumnValueFilter, but filters in camel-hbase are completely customized and this standard filter not fit into camel-hbase.
I'll delete this parameter. The only one filter remains is maxResults=2.

IMO filters in camel-hbase need to be rethink.

woj-i · 2015-10-21T14:18:57Z

I've fixed tests (there was also a bug in HBaseConsumer).

The last, but not least is osgi. I tried to run camel-itest-karaf, but have errors for all tests :(

woj-i · 2015-10-21T14:29:46Z

one of fails in camel-itest-karaf:
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 5.583 sec <<< FAILURE! - in org.apache.camel.itest.karaf.CamelAhcTest
test(org.apache.camel.itest.karaf.CamelAhcTest) Time elapsed: 5.436 sec <<< ERROR!
java.lang.ClassNotFoundException: org.apache.camel.spi.RouteContext not found by PAXEXAM-PROBE-787bd7bd-b995-414e-bb8c-1cc0664f233c [73]
at org.apache.felix.framework.BundleWiringImpl.findClassOrResourceByDelegation(BundleWiringImpl.java:1556)
at org.apache.felix.framework.BundleWiringImpl.access$400(BundleWiringImpl.java:77)
at org.apache.felix.framework.BundleWiringImpl$BundleClassLoader.loadClass(BundleWiringImpl.java:1993)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
at java.lang.Class.privateGetPublicMethods(Class.java:2902)
at java.lang.Class.getMethods(Class.java:1615)
at org.ops4j.pax.exam.invoker.junit.internal.JUnitProbeInvoker.findAndInvoke(JUnitProbeInvoker.java:94)
at org.ops4j.pax.exam.invoker.junit.internal.JUnitProbeInvoker.call(JUnitProbeInvoker.java:73)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.ops4j.pax.exam.rbc.internal.RemoteBundleContextImpl.remoteCall(RemoteBundleContextImpl.java:80)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:323)
at sun.rmi.transport.Transport$1.run(Transport.java:200)
at sun.rmi.transport.Transport$1.run(Transport.java:197)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Transport.java:196)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:568)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:826)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$79(TCPTransport.java:683)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:682)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

davsclaus · 2015-10-21T14:38:08Z

Works for me with

mvn test -Dtest=CamelAhcTest

Make sure you have built all the source code, I usually do in the morning

mvn clean install -Dtest=false

woj-i · 2015-10-21T15:54:07Z

thanks for commands.
I cannot install camel project :( I have a problem with GAE module:
Failed to execute goal on project camel-gae: Could not resolve dependencies for project org.apache.camel:camel-gae:bundle:2.17-SNAPSHOT: The following artifacts could not be resolved: com.google.gdata:gdata-contacts-3.0:jar:1.41.5, com.google.gdata:gdata-calendar-2.0:jar:1.41.5, com.google.gdata:gdata-client-1.0:jar:1.41.5, com.google.gdata:gdata-core-1.0:jar:1.41.5: Could not find artifact com.google.gdata:gdata-contacts-3.0:jar:1.41.5 in central (https://repo.maven.apache.org/maven2)

I used also conjars repo, the same problem.

davsclaus · 2015-10-22T05:34:14Z

I just updated camel-gae to use dependencies that are downloadable from mvn central. Can you try to pull latest code from master and see if you can build Camel ?

woj-i · 2015-10-22T08:24:35Z

@davsclaus , thanks. Now camel-gae is ok, but I have the same problem with camel-example-gauth. Could you repair it too?

Failed to execute goal on project camel-example-gauth: Could not resolve dependencies for project org.apache.camel:camel-example-gauth:war:2.17-SNAPSHOT: The following artifacts could not be resolved: com.google.gdata:gdata-calendar-2.0:jar:1.41.5, com.google.gdata:gdata-client-1.0:jar:1.41.5, com.google.gdata:gdata-core-1.0:jar:1.41.5: Failure to find com.google.gdata:gdata-calendar-2.0:jar:1.41.5 in https://repo.maven.apache.org/maven2 was cached in the local repository, resolution will not be reattempted until the update interval of central has elapsed or updates are forced -> [Help 1]

davsclaus · 2015-10-22T08:37:11Z

ok can you try again

woj-i · 2015-10-22T09:15:53Z

Step forward. :) Another problem with dependencies:
Failed to execute goal on project camel-example-guice-jms: Could not resolve dependencies for project org.apache.camel:camel-example-guice-jms:bundle:2.17-SNAPSHOT: The following artifacts could not be resolved: org.springframework:org.springframework.aop:jar:2.5.6.SEC01, org.springframework:org.springframework.beans:jar:2.5.6.SEC01, org.springframework:org.springframework.context:jar:2.5.6.SEC01, org.springframework:org.springframework.core:jar:2.5.6.SEC01: Could not find artifact org.springframework:org.springframework.aop:jar:2.5.6.SEC01 in central (https://repo.maven.apache.org/maven2)

davsclaus · 2015-10-22T10:01:13Z

its osgi pulling in old JARs that are not in central. I really wonder why spring is removed from central? Anyway the example has been fixed to not do osgi as that was a bad example anyway.

So please try again

woj-i · 2015-10-22T11:19:02Z

Thank you, it works now.
I run test for camel-hbase: mvn clean test -Dtest=CamelHBaseTest
Tests pass with the proposed version [2,3), as well as [1,2). As I wrote before, I recommend to use [2,3), because of Hadoop API change.
@davsclaus, I think it's ready to merge.

davsclaus · 2015-10-23T05:23:46Z

The test org.apache.camel.component.hbase.HBaseProducerTest seems to hang, when i run mvn clean install in the camel-hbase directory.

davsclaus · 2015-10-23T05:27:35Z

Ah it passed but took a fair long time, about 5 minutes

woj-i · 2015-10-23T05:42:06Z

It's because all test cases starts a new Hadoop mini cluster.
23 paź 2015 07:27 "Claus Ibsen" notifications@github.com napisał(a):

Ah it passed but took a fair long time, about 5 minutes

—
Reply to this email directly or view it on GitHub
#634 (comment).

davsclaus · 2015-10-23T05:46:48Z

Just pushed the stuff. Thanks. Can you close this PR

woj-i · 2015-10-23T07:28:29Z

Thank you too!

Fixed CS for camel-core and camel-catalog

[CAMEL-9145] update hbase version to 1.1.1 and hadoop2 to 2.7.1

7849983

woj-i added 2 commits October 16, 2015 17:36

Merge branch 'master' into CAMEL-9145

91ca9f1

Conflicts: parent/pom.xml

replace tabs with spaces

2551b61

fix tests

accafbd

woj-i closed this Oct 23, 2015

woj-i deleted the CAMEL-9145 branch October 23, 2015 07:29

tdiesler pushed a commit to tdiesler/camel that referenced this pull request Feb 20, 2019

Merge pull request apache#634 from jboss-fuse/fixed-cs

9391e48

Fixed CS for camel-core and camel-catalog

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[CAMEL-9145] update hbase version to 1.1.1 and hadoop2 to 2.7.1 #634

[CAMEL-9145] update hbase version to 1.1.1 and hadoop2 to 2.7.1 #634

woj-i commented Oct 9, 2015

davsclaus commented Oct 16, 2015

woj-i commented Oct 16, 2015

woj-i commented Oct 16, 2015

davsclaus commented Oct 18, 2015

davsclaus commented Oct 19, 2015

davsclaus commented Oct 19, 2015

davsclaus commented Oct 19, 2015

woj-i commented Oct 19, 2015

davsclaus commented Oct 20, 2015

woj-i commented Oct 20, 2015

davsclaus commented Oct 20, 2015

woj-i commented Oct 21, 2015

woj-i commented Oct 21, 2015

davsclaus commented Oct 21, 2015

woj-i commented Oct 21, 2015

woj-i commented Oct 21, 2015

woj-i commented Oct 21, 2015

davsclaus commented Oct 21, 2015

woj-i commented Oct 21, 2015

davsclaus commented Oct 22, 2015

woj-i commented Oct 22, 2015

davsclaus commented Oct 22, 2015

woj-i commented Oct 22, 2015

davsclaus commented Oct 22, 2015

woj-i commented Oct 22, 2015

davsclaus commented Oct 23, 2015

davsclaus commented Oct 23, 2015

woj-i commented Oct 23, 2015

davsclaus commented Oct 23, 2015

woj-i commented Oct 23, 2015

[CAMEL-9145] update hbase version to 1.1.1 and hadoop2 to 2.7.1 #634

[CAMEL-9145] update hbase version to 1.1.1 and hadoop2 to 2.7.1 #634

Conversation

woj-i commented Oct 9, 2015

davsclaus commented Oct 16, 2015

woj-i commented Oct 16, 2015

woj-i commented Oct 16, 2015

davsclaus commented Oct 18, 2015

davsclaus commented Oct 19, 2015

davsclaus commented Oct 19, 2015

davsclaus commented Oct 19, 2015

woj-i commented Oct 19, 2015

davsclaus commented Oct 20, 2015

woj-i commented Oct 20, 2015

davsclaus commented Oct 20, 2015

woj-i commented Oct 21, 2015

woj-i commented Oct 21, 2015

davsclaus commented Oct 21, 2015

woj-i commented Oct 21, 2015

woj-i commented Oct 21, 2015

woj-i commented Oct 21, 2015

davsclaus commented Oct 21, 2015

woj-i commented Oct 21, 2015

davsclaus commented Oct 22, 2015

woj-i commented Oct 22, 2015

davsclaus commented Oct 22, 2015

woj-i commented Oct 22, 2015

davsclaus commented Oct 22, 2015

woj-i commented Oct 22, 2015

davsclaus commented Oct 23, 2015

davsclaus commented Oct 23, 2015

woj-i commented Oct 23, 2015

davsclaus commented Oct 23, 2015

woj-i commented Oct 23, 2015