Fixing search for existing executable testReadWritePayloads #18542

mrodozov · 2017-05-02T15:31:55Z

Investigating unit test fail
https://cmssdt.cern.ch/SDT/cgi-bin/buildlogs/slc6_amd64_gcc530/CMSSW_9_1_X_2017-04-28-1100/unitTestLogs/CondCore/CondDB
which fails due to missing executable
/bin/sh: /tmp/cmsbuild/workspace/ib-run-qa/CMSSW_9_1_X_2017-04-28-1100/test/slc6_amd64_gcc530/testReadWritePayloads: No such file or directory
we added some logic to search all possible locations for and not only the local directory in
CondCore/CondDB/test/condTestRegression.py
It's only a partial fix since there are additional problems.
Unit test failing started after this commit
#18495

cmsbuild · 2017-05-02T15:32:13Z

A new Pull Request was created by @mrodozov for master.

It involves the following packages:

CondCore/CondDB

@ggovi, @cmsbuild, @davidlange6 can you please review it and eventually sign? Thanks.
@ghellwig this is something you requested to watch as well.
@davidlange6, @smuzaffar you are the release manager for this.

cms-bot commands are listed here

smuzaffar · 2017-05-02T16:00:49Z

CondCore/CondDB/test/condTestRegression.py

+def get_cmssw_file(file_name):
+
+    for i in ['CMSSW_BASE','CMSSW_RELEASE_BASE','CMSSW_FULL_RELEASE_BASE']:
+        print "Looking for :%s = %s " % (i,os.getenv(i))


@mrodozov , please remove this line. It was added just for debugging

smuzaffar · 2017-05-02T16:01:23Z

CondCore/CondDB/test/condTestRegression.py

@@ -39,6 +46,7 @@ def check_output(*popenargs, **kwargs):

 # nice one from:
 # https://www.daniweb.com/software-development/python/code/216610/timing-a-function-python
+


why this extra line?

smuzaffar · 2017-05-02T16:03:56Z

Thanks @mrodozov, please cleanup the unnecessary (extra empty lines etc.) changes.
Can you please also add the boost missing library message (as a comment here) which you were getting after the fix.

smuzaffar · 2017-05-02T16:10:15Z

CondCore/CondDB/test/condTestRegression.py

          # we run in the local environment, but need to make sure that we start "top-level" of the devel area
          # and we assume that the test was already built 
+          payloadScript = 'test/%s/testReadWritePayloads' % self.arch
          cmd = 'export SCRAM_ARCH=%s; cd %s/src; eval `scram runtime -sh 2>/dev/null` ; ' % (os.environ['SCRAM_ARCH'],os.environ['CMSSW_BASE'], )


@ggovi , if CMSSW_BASE is set then this means cmsenv is already set, so in this case we do not need to source the env again .... rght?

@smuzaffar
true, but in fact, where it is done again? Not sure I find it...

smuzaffar · 2017-05-02T16:15:08Z

CondCore/CondDB/test/condTestRegression.py

          cmd =  'source %s/cmsset_default.sh; export SCRAM_ARCH=%s; cd %s/src ; eval `scram runtime -sh`; cd - ; ' %(cmsPath,arch,releaseDir)
          cmd += "echo 'CMSSW_BASE='$CMSSW_BASE; echo 'RELEASE_BASE='$RELEASE_BASE; echo 'PATH='$PATH; echo 'LD_LIBRARY_PATH='$LD_LIBRARY_PATH;"
-          cmd += '$LOCALRT/test/%s/testReadWritePayloads %s sqlite_file:///%s/%s ' % (arch, readOrWrite, self.dbDir, dbName)


@ggovi , here you are setting env for releaseDir (an old release build with old version of boost) and then running $LOCALRT/test/%s/testReadWritePayloads from local dev area ( which is linked against new boost version 1.63).

well I don't know exactly what is going on in this version, since I don't quite understand the function "get_cmssw_file(file_name)".
In the original version, printing $LOCALRT in any of the writing procedure, one obtains the right path, where the executable runs against the correct boost library.
Is it possible to see the log of the failure that triggered these changes?

Removing extra lines

cmsbuild · 2017-05-03T07:57:22Z

Pull request #18542 was updated. @ggovi, @cmsbuild, @davidlange6 can you please check and sign again.

mrodozov · 2017-05-03T08:00:29Z

After the executable path fix, the test is still failing with :
error while loading shared libraries: libboost_filesystem.so.1.63.0: cannot open shared object file: No such file or directory

smuzaffar · 2017-05-04T09:06:24Z

cmsbuild, please test

cmsbuild · 2017-05-04T09:06:53Z

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-any-integration/19560/console Started: 2017/05/04 11:07

cmsbuild · 2017-05-04T10:20:29Z

-1

Tested at: f6143aa

You can see the results of the tests here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-18542/19560/summary.html

I found follow errors while testing this PR

Failed tests: UnitTests

Unit Tests:

I found errors in the following unit tests:

---> test condTestRegression had ERRORS

cmsbuild · 2017-05-04T10:20:32Z

Comparison job queued.

cmsbuild · 2017-05-04T13:00:45Z

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-18542/19560/summary.html

Comparison Summary:

You potentially added 11 lines to the logs
Reco comparison results: 1702 differences found in the comparisons
DQMHistoTests: Total files compared: 24
DQMHistoTests: Total histograms compared: 1835819
DQMHistoTests: Total failures: 56570
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 1779069
DQMHistoTests: Total skipped: 180
DQMHistoTests: Total Missing objects: 0
Checked 98 log files, 14 edm output root files, 24 DQM output files

ggovi · 2017-05-04T13:36:26Z

@smuzaffar
I'm still lost about the problem we are trying to solve here. In what circumstance the test as it was delivered before these changes was failing? It was not failing in my private environment, nor in the PR validation...

Thanks,
cheers

G.

ggovi · 2017-05-04T13:41:35Z

Ok I've seen the log, It looks that the failing part is the writing/reading by the IB under test. The writing by the reference releases should be ok, it should be not altered.

ggovi · 2017-05-04T13:44:16Z

@smuzaffar @mrodozov
is there a way to reproduce the problem in a private environment?

ggovi · 2017-05-04T13:46:20Z

@mrodozov
I guess the run() function in the code should work as it was. What we may adapt is the runSelf() function.

smuzaffar · 2017-05-04T13:56:21Z

@ggovi , yes you can re-produce the error. By doign something like

scram p CMSSW_9_1_X_2017-05-04-1100
cd CMSSW_9_1_X_2017-05-04-1100
cmsenv
git cms-addpkg CondCore/CondDB
scram b -j 10
#delete testReadWritePayloads as we want this to be picked up from release area (As this is the case for IBs)
rm test/$SCRAM_ARCH/testReadWritePayloads
scram b unittests_condTestRegression

ggovi · 2017-05-04T14:47:37Z

Ok, thanks for the recipe and for the hint. I think I have a fix based on your code. I'll do a new PR.

ggovi · 2017-05-09T07:20:25Z

-1
superseded by #18579

Fixing search for existing executable testReadWritePayloads

39df811

cmsbuild added this to the Next CMSSW_9_1_X milestone May 2, 2017

cmsbuild added comparison-pending db-pending orp-pending pending-signatures tests-pending labels May 2, 2017

smuzaffar reviewed May 2, 2017

View reviewed changes

Update condTestRegression.py

f6143aa

Removing extra lines

cmsbuild added tests-started and removed tests-pending labels May 4, 2017

cmsbuild added tests-rejected and removed tests-started labels May 4, 2017

cmsbuild added comparison-available and removed comparison-pending labels May 4, 2017

smuzaffar modified the milestones: CMSSW_9_2_X, CMSSW_9_1_X May 4, 2017

ggovi mentioned this pull request May 5, 2017

Fixes for regression in the persistency of std::string payloads #18579

Merged

cmsbuild added db-rejected and removed db-pending labels May 9, 2017

davidlange6 closed this May 9, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fixing search for existing executable testReadWritePayloads #18542

Fixing search for existing executable testReadWritePayloads #18542

mrodozov commented May 2, 2017 •

edited

cmsbuild commented May 2, 2017

smuzaffar May 2, 2017

smuzaffar May 2, 2017

smuzaffar commented May 2, 2017

smuzaffar May 2, 2017

ggovi May 4, 2017

smuzaffar May 2, 2017

ggovi May 4, 2017

cmsbuild commented May 3, 2017

mrodozov commented May 3, 2017

smuzaffar commented May 4, 2017

cmsbuild commented May 4, 2017 •

edited

cmsbuild commented May 4, 2017

cmsbuild commented May 4, 2017

cmsbuild commented May 4, 2017

ggovi commented May 4, 2017

ggovi commented May 4, 2017

ggovi commented May 4, 2017

ggovi commented May 4, 2017

smuzaffar commented May 4, 2017

ggovi commented May 4, 2017

ggovi commented May 9, 2017

		@@ -39,6 +46,7 @@ def check_output(popenargs, *kwargs):

		# nice one from:
		# https://www.daniweb.com/software-development/python/code/216610/timing-a-function-python

Fixing search for existing executable testReadWritePayloads #18542

Fixing search for existing executable testReadWritePayloads #18542

Conversation

mrodozov commented May 2, 2017 • edited

cmsbuild commented May 2, 2017

smuzaffar May 2, 2017

Choose a reason for hiding this comment

smuzaffar May 2, 2017

Choose a reason for hiding this comment

smuzaffar commented May 2, 2017

smuzaffar May 2, 2017

Choose a reason for hiding this comment

ggovi May 4, 2017

Choose a reason for hiding this comment

smuzaffar May 2, 2017

Choose a reason for hiding this comment

ggovi May 4, 2017

Choose a reason for hiding this comment

cmsbuild commented May 3, 2017

mrodozov commented May 3, 2017

smuzaffar commented May 4, 2017

cmsbuild commented May 4, 2017 • edited

cmsbuild commented May 4, 2017

cmsbuild commented May 4, 2017

cmsbuild commented May 4, 2017

ggovi commented May 4, 2017

ggovi commented May 4, 2017

ggovi commented May 4, 2017

ggovi commented May 4, 2017

smuzaffar commented May 4, 2017

ggovi commented May 4, 2017

ggovi commented May 9, 2017

mrodozov commented May 2, 2017 •

edited

cmsbuild commented May 4, 2017 •

edited