Attempting to create the History cache for Clearcase repositories throwing SQL Errors (Bugzilla #18166)

[vladak]

status NEW severity major in component scm for ---
Reported in version unspecified on platform ANY/Generic
Assigned to: Trond Norbye

On 2011-04-15 18:25:50 +0000, Dana wrote:

Using grok - January 2011 update 0.10
Redhat: 2.6.18-238.1.1.el5
Clearcase: 7.1.2.1 (Using a snapshot view )
Subversion: 1.6.15

The index throws the following SQL error when creating history for a repository. The history itself is being retrieval correctly from Clearcase..

I added a HistoryEntry.dump() right before the on line 733 in the JDBCHistoryCache class, before handleSQLException(sqle, i);

Subversion works correctly..

------> log excerpt...

Apr 15, 2011 12:24:15 PM org.opensolaris.opengrok.history.Repository getHistory
WARNING: Incremental history retrieval is not implemented for ClearCaseRepository.
Apr 15, 2011 12:24:15 PM org.opensolaris.opengrok.history.Repository getHistory
WARNING: Falling back to slower full history retrieval.
Apr 15, 2011 12:24:15 PM org.opensolaris.opengrok.util.Executor exec
FINE: Executing command [/opt/IBM/RationalSDLC/clearcase/bin/cleartool, lshistory, -dir, -fmt, %e
%Nd
%Fu (%u)
%Vn
%Nc
.
, ] in directory /indexstore/rps/cc/vobs/CGSCommon

Apr 15, 2011 12:24:34 PM org.opensolaris.opengrok.util.Executor exec
FINE: Executing command [/opt/IBM/RationalSDLC/clearcase/bin/cleartool, lshistory, -dir, -fmt, %e
%Nd
%Fu (%u)
%Vn
%Nc
.
, ] in directory /indexstore/rps/cc/vobs/CGSCommon

Apr 15, 2011 12:24:34 PM org.opensolaris.opengrok.history.HistoryEntry dump
FINE: HistoryEntry : revision = /main/0
Apr 15, 2011 12:24:34 PM org.opensolaris.opengrok.history.HistoryEntry dump
FINE: HistoryEntry : date = 4/27/07 2:15 PM
Apr 15, 2011 12:24:34 PM org.opensolaris.opengrok.history.HistoryEntry dump
FINE: HistoryEntry : author = CLEARCASE.ADMIN (ccadmin)
Apr 15, 2011 12:24:34 PM org.opensolaris.opengrok.history.HistoryEntry dump
FINE: HistoryEntry : active = True
Apr 15, 2011 12:24:34 PM org.opensolaris.opengrok.history.HistoryEntry dump
FINE: HistoryEntry : message =
Apr 15, 2011 12:24:34 PM org.opensolaris.opengrok.history.HistoryGuru createCache
WARNING: An error occured while creating cache for /indexstore/rps/cc/vobs/CGSCommon (ClearCaseRepository)
org.opensolaris.opengrok.history.HistoryException: java.sql.SQLIntegrityConstraintViolationException: The statement was aborted because it would have c
aused a duplicate key value in a unique or primary key constraint or unique index identified by 'CHANGESETS_REPO_REV_UNIQUE' defined on 'CHANGESETS'.
at org.opensolaris.opengrok.history.JDBCHistoryCache.store(JDBCHistoryCache.java:607)
at org.opensolaris.opengrok.history.Repository.createCache(Repository.java:224)
at org.opensolaris.opengrok.history.HistoryGuru.createCache(HistoryGuru.java:434)
at org.opensolaris.opengrok.history.HistoryGuru.ensureHistoryCacheExists(HistoryGuru.java:554)
at org.opensolaris.opengrok.index.IndexDatabase.update(IndexDatabase.java:337)
at org.opensolaris.opengrok.index.IndexDatabase$1.run(IndexDatabase.java:157)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

Caused by: java.sql.SQLIntegrityConstraintViolationException: The statement was aborted because it would have caused a duplicate key value in a unique
or primary key constraint or unique index identified by 'CHANGESETS_REPO_REV_UNIQUE' defined on 'CHANGESETS'.
at org.apache.derby.client.am.SQLExceptionFactory40.getSQLException(Unknown Source)
at org.apache.derby.client.am.SqlException.getSQLException(Unknown Source)
at org.apache.derby.client.am.PreparedStatement.executeUpdate(Unknown Source)
at org.opensolaris.opengrok.history.JDBCHistoryCache.storeHistory(JDBCHistoryCache.java:702)
at org.opensolaris.opengrok.history.JDBCHistoryCache.store(JDBCHistoryCache.java:602)
... 11 more
Caused by: org.apache.derby.client.am.SqlException: The statement was aborted because it would have caused a duplicate key value in a unique or primary
key constraint or unique index identified by 'CHANGESETS_REPO_REV_UNIQUE' defined on 'CHANGESETS'.
at org.apache.derby.client.am.Statement.completeExecute(Unknown Source)
at org.apache.derby.client.net.NetStatementReply.parseEXCSQLSTTreply(Unknown Source)
at org.apache.derby.client.net.NetStatementReply.readExecute(Unknown Source)
at org.apache.derby.client.net.StatementReply.readExecute(Unknown Source)
at org.apache.derby.client.net.NetPreparedStatement.readExecute_(Unknown Source)
at org.apache.derby.client.am.PreparedStatement.readExecute(Unknown Source)
at org.apache.derby.client.am.PreparedStatement.flowExecute(Unknown Source)
at org.apache.derby.client.am.PreparedStatement.executeUpdateX(Unknown Source)
... 14 more

Output from the history command above...

$ cleartool lshistory -dir -fmt '%e\n%Nd\n%Fu (%u)\n%Vn\n%Nc\n.\n' .
create directory version
20081203.104450
Lee (L306834)
/main/2
Uncataloged directory element "ris.cgs.dbp.1_0".
.
create directory version
20070507.161140
Lee (l306834)
/main/1
Added directory element "ris.cgs.dbp.1_0".
.
create directory version
20070427.141519
CLEARCASE.ADMIN (ccadmin)
/main/0

.
create branch
20070427.141519
CLEARCASE.ADMIN (ccadmin)
/main

.
create directory element
20070427.141519
CLEARCASE.ADMIN (ccadmin)

Predefined directory element for root of versioned object base.
.

On 2011-04-22 16:50:14 +0000, Dana wrote:

I did a little more research and starting from a clean cachedb, and running the indexer with a slightly modified org.opensolaris.opengrok.jdbc.InsertQuery.
I added a DebuggablePreparedStatement to capture the actual SQL.

It seems that during the storeHistory(conn,history,repository) for a clearcase repository it never gets passed adding the Changeset inserts.. No fileschange/dirchanges are ever added during the parsing of the history for a given directory or repository.. Using SquirrelSQL I have looked in the derby database and I can only find references to a clearcase repository in REPOSITORIES, CHANGESETS, AUTHORS.