TRAFODION-2067 support cdh5.7 & hbase1.2 #610

mashengchen · 2016-07-21T03:10:23Z

support cdh5.7 & hbase1.2. add hbase-trx-cdh5.7.jar & hbase-trx-apache1.2.jar
HBASE_DISTRO add CDH5.7 APACHE1.2
remove cdh & apache & cdh version in trafodion-dtm-.jar & trafodion-sql-.jar & trafodion-utility-.jar, they will be trafodion-dtm-[apache][cdh][hdp]-$trafodion_version.jar & trafodion-sql-[apache][cdh][hdp]-$trafodion_version.jar & trafodion-utility-$trafodion_version.jar . becacuse we can't determine which version user will install. and we only have one version of each jar. but not hbase-trx.jar, so hbase-trx*.jar have version, because they have multi version. eg: for cdh they have hbase-trx-cdh5_4-2.1.0.jar & hbase-trx-cdh5_5-2.1.0.jar & hbase-trx-cdh5_7-2.1.0.jar
HBASE1.* ->APACHE1.* in .tmpl files.
do some optimize in traf_config_check

update head

Traf-Jenkins · 2016-07-21T03:15:18Z

Check Test Started: https://jenkins.esgyn.com/job/Check-PR-master/946/

Traf-Jenkins · 2016-07-21T05:30:02Z

Test Passed. https://jenkins.esgyn.com/job/Check-PR-master/946/

moscowgentalman · 2016-07-21T08:09:43Z

Looks good for me.

Traf-Jenkins · 2016-07-21T09:03:12Z

New Check Test Started: https://jenkins.esgyn.com/job/Check-PR-master/949/

Traf-Jenkins · 2016-07-21T12:05:10Z

Test Passed. https://jenkins.esgyn.com/job/Check-PR-master/949/

DaveBirdsall · 2016-07-25T20:27:53Z

Looks good to me.

amandamoran · 2016-07-26T19:16:59Z

I tested these changes on CDH 5.7 on a 2 node cluster.

[trafodion@addnode-4 ~]$ sqlci
Apache Trafodion Conversational Interface 2.1.0
Copyright (c) 2015-2016 Apache Software Foundation

create table toby (f1 int, f2 int);
--- SQL operation complete.
get tables;

Tables in Schema TRAFODION.SEABASE

TOBY
--- SQL operation complete.
exit
End of MXCI Session

[trafodion@addnode-4 ~]$ echo $CDH_VERSION
5.7

There are some more install changes that need to be made to support this, but I can make those in another pull request. (For example CDH_5_7_SUPPORT is not being added to /etc/trafodion/trafodion_config and the HBase Version during trafodion_install).

07/28: Here are the additional install changes
#625

sbroeder · 2016-07-27T17:03:37Z

I'm able to use maven to build the coprocessor/transactional/generated/TrxRegionProtos.java file for the pom.xml.apache* files, but I can't for the pom.xml.cdh* files. Can you tell me which version of protoc you are using and what command you use to build these files?

amandamoran · 2016-07-28T20:07:29Z

CDH 5.8 came out today. I tested these changes on a single node CDH 5.8 system and it worked great!

[trafodion@cdh-58 ~]$ sqcheck

*** Checking Trafodion Environment ***

Checking if processes are up.
Checking attempt: 1; user specified max: 2. Execution time in seconds: 0.

The Trafodion environment is up!

Process Configured Actual Down

DTM 2 2
RMS 4 4
DcsMaster 1 1
DcsServer 1 1
mxosrvr 8 8

RestServer 1 1

[trafodion@cdh-58 ~]$ sqlci
Apache Trafodion Conversational Interface 2.1.0
Copyright (c) 2015-2016 Apache Software Foundation

create table toby (f1 int, f2 int);

--- SQL operation complete.

insert into toby values (8,10);

--- 1 row(s) inserted.

get tables;

TOBY

--- SQL operation complete.

select * from toby;

F1 F2

      8           10

--- 1 row(s) selected.

exit

End of MXCI Session

mashengchen · 2016-07-29T04:39:21Z

@sbroeder
pls use 2.5
and i used to use protoc to gen TrxRegionProtos.java first, after gen the file .you can do make under /hbase-trx.

moscowgentalman · 2016-08-01T23:17:45Z

Is this ready for merge?

prashanth-vasudev · 2016-08-02T16:45:26Z

.../src/main/java/org/apache/hadoop/hbase/coprocessor/transactional/TrxRegionEndpoint.java.tmpl

@@ -4241,8 +4257,12 @@ CoprocessorService, Coprocessor {
                 e.add(kv);
                 try {
                    final WALKey wk = new WALKey(this.regionInfo.getEncodedNameAsBytes(), this.regionInfo.getTable(), EnvironmentEdgeManager.currentTime());
+#ifdef CDH5.7 APACHE1.2
+                    txid = this.tHLog.append(this.m_Region.getTableDesc(), this.regionInfo, wk, e, false);


The sequenceID gets missing here during logging. Please confirm this does not have side effects during recovery.

i do a test and find out there is no side effect

before cdh5.7

2016-08-04 05:44:09,318 TRACE [B.defaultRpcServer.handler=17,queue=2,port=35946] transactional.TrxRegionEndpoint: TrxRegionEndpoint coprocessor: commit - txId 3, Write commit HLOG seq 274
----do sqstop && swsopall and do sqstart && swstartall
2016-08-04 06:44:48,680 TRACE [B.defaultRpcServer.handler=29,queue=2,port=43771] transactional.TrxRegionEndpoint: TrxRegionEndpoint coprocessor: commit - txId 3, Write commit HLOG seq 214

cdh 5.7

2016-08-04 06:18:52,101 TRACE [PriorityRpcServer.handler=5,queue=1,port=58401] transactional.TrxRegionEndpoint: TrxRegionEndpoint coprocessor: commit - txId 3, Write commit HLOG seq 227

prashanth-vasudev · 2016-08-02T17:27:27Z

Reviewed the changes. Except for few comments added, rest looks good. Please address those comments and run full regressions before the merge. Thanks.

Traf-Jenkins · 2016-08-04T06:57:15Z

Check Test Started: https://jenkins.esgyn.com/job/Check-PR-master/1013/

Traf-Jenkins · 2016-08-04T07:36:02Z

Test Failed. https://jenkins.esgyn.com/job/Check-PR-master/1013/

mashengchen · 2016-08-09T01:30:50Z

jenkins ,retest

Traf-Jenkins · 2016-08-09T01:33:11Z

New Check Test Started: https://jenkins.esgyn.com/job/Check-PR-master/1032/

Traf-Jenkins · 2016-08-09T02:09:05Z

Test Failed. https://jenkins.esgyn.com/job/Check-PR-master/1032/

mashengchen · 2016-08-09T07:14:29Z

jenkins, retest

Traf-Jenkins · 2016-08-09T07:15:16Z

New Check Test Started: https://jenkins.esgyn.com/job/Check-PR-master/1035/

Traf-Jenkins · 2016-08-09T07:56:06Z

Test Failed. https://jenkins.esgyn.com/job/Check-PR-master/1035/

moscowgentalman · 2016-08-09T15:36:12Z

jenkins, retest

Traf-Jenkins · 2016-08-09T15:39:14Z

New Check Test Started: https://jenkins.esgyn.com/job/Check-PR-master/1037/

Traf-Jenkins · 2016-08-09T16:17:59Z

Test Failed. https://jenkins.esgyn.com/job/Check-PR-master/1037/

arvind-narain · 2016-08-11T08:40:01Z

The build error seems to be the following. Maybe this is will go once rebase is done and conflicts are resolved ? Change in installer/.gitignore for DISCLAIMER went in 15 days back.

Error: Build created 1 untracked file(s)
Update .gitignore files
install/installer/DISCLAIMER

mashengchen added 3 commits July 20, 2016 11:25

Merge pull request #5 from apache/master

a6b9b2d

update head

TRAFODION-2067 support CDH5.7

ad9258c

change some bug

f523af4

add install_features & change a bug

72b4f43

prashanth-vasudev reviewed Aug 2, 2016
View reviewed changes

remove unnecessary log

04d7f0b

mashengchen closed this Aug 11, 2016

mashengchen mentioned this pull request Aug 12, 2016

TRAFODION-2067 support cdh5.7 & hbase1.2 #655

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

TRAFODION-2067 support cdh5.7 & hbase1.2 #610

TRAFODION-2067 support cdh5.7 & hbase1.2 #610

mashengchen commented Jul 21, 2016 •

edited

Traf-Jenkins commented Jul 21, 2016

Traf-Jenkins commented Jul 21, 2016

moscowgentalman commented Jul 21, 2016

Traf-Jenkins commented Jul 21, 2016

Traf-Jenkins commented Jul 21, 2016

DaveBirdsall commented Jul 25, 2016

amandamoran commented Jul 26, 2016 •

edited

Tables in Schema TRAFODION.SEABASE

sbroeder commented Jul 27, 2016

amandamoran commented Jul 28, 2016 •

edited

mashengchen commented Jul 29, 2016

moscowgentalman commented Aug 1, 2016

prashanth-vasudev Aug 2, 2016

mashengchen Aug 4, 2016

prashanth-vasudev commented Aug 2, 2016

Traf-Jenkins commented Aug 4, 2016

Traf-Jenkins commented Aug 4, 2016

mashengchen commented Aug 9, 2016

Traf-Jenkins commented Aug 9, 2016

Traf-Jenkins commented Aug 9, 2016

mashengchen commented Aug 9, 2016

Traf-Jenkins commented Aug 9, 2016

Traf-Jenkins commented Aug 9, 2016

moscowgentalman commented Aug 9, 2016

Traf-Jenkins commented Aug 9, 2016

Traf-Jenkins commented Aug 9, 2016

arvind-narain commented Aug 11, 2016 •

edited

TRAFODION-2067 support cdh5.7 & hbase1.2 #610

TRAFODION-2067 support cdh5.7 & hbase1.2 #610

Conversation

mashengchen commented Jul 21, 2016 • edited

Traf-Jenkins commented Jul 21, 2016

Traf-Jenkins commented Jul 21, 2016

moscowgentalman commented Jul 21, 2016

Traf-Jenkins commented Jul 21, 2016

Traf-Jenkins commented Jul 21, 2016

DaveBirdsall commented Jul 25, 2016

amandamoran commented Jul 26, 2016 • edited

Tables in Schema TRAFODION.SEABASE

sbroeder commented Jul 27, 2016

amandamoran commented Jul 28, 2016 • edited

CDH 5.8 came out today. I tested these changes on a single node CDH 5.8 system and it worked great!

mashengchen commented Jul 29, 2016

moscowgentalman commented Aug 1, 2016

prashanth-vasudev Aug 2, 2016

Choose a reason for hiding this comment

mashengchen Aug 4, 2016

Choose a reason for hiding this comment

before cdh5.7

cdh 5.7

prashanth-vasudev commented Aug 2, 2016

Traf-Jenkins commented Aug 4, 2016

Traf-Jenkins commented Aug 4, 2016

mashengchen commented Aug 9, 2016

Traf-Jenkins commented Aug 9, 2016

Traf-Jenkins commented Aug 9, 2016

mashengchen commented Aug 9, 2016

Traf-Jenkins commented Aug 9, 2016

Traf-Jenkins commented Aug 9, 2016

moscowgentalman commented Aug 9, 2016

Traf-Jenkins commented Aug 9, 2016

Traf-Jenkins commented Aug 9, 2016

arvind-narain commented Aug 11, 2016 • edited

mashengchen commented Jul 21, 2016 •

edited

amandamoran commented Jul 26, 2016 •

edited

amandamoran commented Jul 28, 2016 •

edited

arvind-narain commented Aug 11, 2016 •

edited