Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

New minor version of fasthadd, 2.1, and move llvm sqlite from BuildRequires and Requires for py2-dxr #2147

Merged
merged 2 commits into from Feb 17, 2016

Conversation

diguida
Copy link

@diguida diguida commented Feb 15, 2016

Define new minor version of fasthadd, namely 2.1:

  • the new minor version 2.1 is defined so that it is explicitly built on top of the new ROOT 6.0.6, featuring changes in the histogram serialisation;
  • the code base has been moved to the latest 80X pre-release, but no changes in the source file is done.

No changes in performance observed or expected.
The new version is required for ease the deployment in the online environment.
@deguio @vanbesien @dmitrijus this is something you might want to watch as well.

Local test failed, but it seems unrelated with this change: fasthadd RPM was built:

....
Starting to process package external+fasthadd+2.1
Checking if fasthadd is cached.
Package external+fasthadd+2.1 not found in repository. Queuing for build.
Dependencies for external+fasthadd+2.1: ['install-external+gcc+4.9.3', 'install-lcg+root+6.06.00-cms2', 'install-external+protobuf+2.4.1', 'fetch-external+fasthadd+2.1']
....
Fetching sources for fasthadd
Done fetching https://raw.githubusercontent.com/cms-sw/cmssw/38304a4dbdfc1a5e7cc860afbb1c31dd09a9ff1e/DQMServices/Components/bin/fastHadd.cc
Done fetching https://raw.githubusercontent.com/cms-sw/cmssw/38304a4dbdfc1a5e7cc860afbb1c31dd09a9ff1e/DQMServices/Core/src/ROOTFilePB.proto
....
Creating directory /build/diguida/ext/CMSSW_8_0_X/20160215_1421/a/BUILD/slc6_amd64_gcc493/external/fasthadd/2.1 if not existing.
Building fasthadd. Log can be found in /build/diguida/ext/CMSSW_8_0_X/20160215_1421/a/BUILD/slc6_amd64_gcc493/external/fasthadd/2.1/log.
TMPDIR=/build/diguida/ext/CMSSW_8_0_X/20160215_1421/a/tmp  rpmbuild  --buildroot /build/diguida/ext/CMSSW_8_0_X/20160215_1421/a/tmp/BUILDROOT/e98e7322aec04cd4ef8cd98649921a6f -ba --define '_topdir /build/diguida/ext/CMSSW_8_0_X/20160215_1421/a' --define "compiling_processes 24"  --define "cmscompilerv  493" --define "cmsos slc6_amd64" /build/diguida/ext/CMSSW_8_0_X/20160215_1421/a/SPECS/external/fasthadd/2.1/spec >/build/diguida/ext/CMSSW_8_0_X/20160215_1421/a/BUILD/slc6_amd64_gcc493/external/fasthadd/2.1/log 2>&1
Build successful.
Trying to install the rpm package external+fasthadd+2.1 just built.
Done
....
* The action "build-cms+cmssw-tool-conf+32.0-cms2" was not completed successfully because The following dependencies could not complete:
install-external+py2-dxr-toolfile+1.0-ikhhed
* The action "final-job" was not completed successfully because The following dependencies could not complete:
build-cms+cmssw-tool-conf+32.0-cms2
install-cms+cmssw-tool-conf+32.0-cms2
install-external+py2-dxr-toolfile+1.0-ikhhed
* The action "install-external+py2-dxr-toolfile+1.0-ikhhed" was not completed successfully because Traceback (most recent call last):
File "/build/diguida/ext/CMSSW_8_0_X/20160215_1421/PKGTOOLS/scheduler.py", line 199, in doSerial
result = commandSpec[0](*commandSpec[1:])
File "PKGTOOLS/cmsBuild", line 2782, in installApt
RpmInstallFailed: Failed to install package py2-dxr-toolfile. Reason:
Reading Package Lists...
Building Dependency Tree...
The following extra packages will be installed:
external+py2-dxr+1.0-ikhhed
The following NEW packages will be installed:
external+py2-dxr+1.0-ikhhed external+py2-dxr-toolfile+1.0-ikhhed
0 upgraded, 2 newly installed, 0 removed and 0 not upgraded.
Need to get 0B/868kB of archives.
After unpacking 2596kB of additional disk space will be used.
Executing RPM (/build/diguida/ext/CMSSW_8_0_X/20160215_1421/a/slc6_amd64_gcc493/external/apt/0.5.16/bin/rpm-wrapper -Uvh -r /build/diguida/ext/CMSSW_8_0_X/20160215_1421/a --force --prefix /build/diguida
/ext/CMSSW_8_0_X/20160215_1421/a --ignoreos --ignorearch --force --prefix /build/diguida/ext/CMSSW_8_0_X/20160215_1421/a --ignoreos --ignorearch --oldpackage)...
error: Failed dependencies:
#looks like something wrong with LLVM/clang, I do not list the .so files here
E: Sub-process /build/diguida/ext/CMSSW_8_0_X/20160215_1421/a/slc6_amd64_gcc493/external/apt/0.5.16/bin/rpm-wrapper returned an error code (2)

* The action "install-cms+cmssw-tool-conf+32.0-cms2" was not completed successfully because The following dependencies could not complete:
build-cms+cmssw-tool-conf+32.0-cms2

This was traced to be an issue with py2-dxr spec file. LLVM and SQLite are run-time dependencies. In addition to that this causes installation failures. APT-RPM fails if LLVM and SQLite is not installed before DXR. This is fixed in IB/CMSSW_8_0_X/gcc530 branch by 1ac4cd1. This is cherry-picked here.

Local tests are now ok, indeed.

* the new minor version 2.1 is defined so that it is explicitly built on top of the new ROOT 6.0.6, featuring changes in the histogram serialisation;
* the code base has been moved to the latest 80X pre-release, but no changes in the source file is done.
No changes in performance observed or expected.
@cmsbuild
Copy link
Contributor

A new Pull Request was created by @diguida (Salvatore Di Guida) for branch IB/CMSSW_8_0_X/stable.

@cmsbuild, @smuzaffar, @Degano, @iahmad-khan, @davidlange6 can you please review it and eventually sign? Thanks.
You can sign-off by replying to this message having '+1' in the first line of your reply.
You can reject by replying to this message having '-1' in the first line of your reply.

@diguida
Copy link
Author

diguida commented Feb 15, 2016

please test

@cmsbuild
Copy link
Contributor

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/cmsdist-test-pr/104/console

@davidlt
Copy link
Contributor

davidlt commented Feb 15, 2016

The issue is in py2-dxr, I think, I fixed that at least in GCC 5.3.0 branch. The problem is that LLVM is marked as BuildRequires while it is actually needed at run-time (i.e. it must be Requires). Thus in depends on how lucky you get, i.e. things work as long as LLVM is installed first.

@diguida
Copy link
Author

diguida commented Feb 15, 2016

@davidlt that's correct:

* The action "install-external+py2-dxr-toolfile+1.0-ikhhed" was not completed successfully because Traceback (most recent call last):
File "/build/diguida/ext/CMSSW_8_0_X/20160215_1421/PKGTOOLS/scheduler.py", line 199, in doSerial
result = commandSpec[0](*commandSpec[1:])
File "PKGTOOLS/cmsBuild", line 2782, in installApt
RpmInstallFailed: Failed to install package py2-dxr-toolfile. Reason:
Reading Package Lists...
Building Dependency Tree...
The following extra packages will be installed:
external+py2-dxr+1.0-ikhhed
The following NEW packages will be installed:
external+py2-dxr+1.0-ikhhed external+py2-dxr-toolfile+1.0-ikhhed
0 upgraded, 2 newly installed, 0 removed and 0 not upgraded.
Need to get 0B/868kB of archives.
After unpacking 2596kB of additional disk space will be used.
Executing RPM (/build/diguida/ext/CMSSW_8_0_X/20160215_1421/a/slc6_amd64_gcc493/external/apt/0.5.16/bin/rpm-wrapper -Uvh -r /build/diguida/ext/CMSSW_8_0_X/20160215_1421/a --force --prefix /build/diguida
/ext/CMSSW_8_0_X/20160215_1421/a --ignoreos --ignorearch --force --prefix /build/diguida/ext/CMSSW_8_0_X/20160215_1421/a --ignoreos --ignorearch --oldpackage)...
error: Failed dependencies:
#LLVM/clang so files

I can cherry-pick your commit and re-run the test. Can you point it to me?

@davidlt
Copy link
Contributor

davidlt commented Feb 15, 2016

That should be it 1ac4cd1#diff-085ccbfcce396237ceea010830b8c75c

LLVM and SQLite are run-time dependencies. In addition to that this
causes installation failures. APT-RPM fails if LLVM and SQLite is not
installed before DXR.

Signed-off-by: David Abdurachmanov <David.Abdurachmanov@cern.ch>
@cmsbuild
Copy link
Contributor

@diguida
Copy link
Author

diguida commented Feb 16, 2016

@davidlt I confirm that by cherry-picking 1ac4cd1#diff-085ccbfcce396237ceea010830b8c75c the local build was ok. In particular:

  • py2-dxr:
Starting to process package external+py2-dxr+1.0-cms2
Checking if py2-dxr is cached.
Package external+py2-dxr+1.0-cms2 not found in repository. Queuing for build.
Dependencies for external+py2-dxr+1.0-cms2: ['install-external+gcc+4.9.3', 'install-external+zlib+1.2.8', 'install-external+sqlite+3.8.11.1', 'install-external+python+2.7.6-kpegke', 'install-external+py2-setuptools+2.1-kpegke', 'install-external+py2-futures+2.2.0-kpegke', 'install-external+py2-jinja+2.7.2-kpegke', 'install-external+py2-markupsafe+0.23-kpegke', 'install-external+py2-ordereddict+1.1-kpegke', 'install-external+py2-parsimonious+0.6.1-kpegke', 'install-external+py2-pysqlite+2.8.1', 'install-external+llvm+3.7.1-ikhhed', 'fetch-external+py2-dxr+1.0-cms2']
Fetching sources for py2-dxr
Done fetching git+https://github.com/mozilla/dxr.git?obj=master/6ea764102a&export=dxr-6ea764102a&module=dxr-6ea764102a&output=/dxr-6ea764102a.tgz
Done fetching git+https://github.com/jonasfj/trilite.git?obj=master/e64a2a1&export=trilite-e64a2a1&module=trilite-e64a2a1&output=/trilite-e64a2a1.tgz
Done fetching /build/diguida/ext/CMSSW_8_0_X/20160215_1421/CMSDIST/py2-dxr.patch
Done fetching /build/diguida/ext/CMSSW_8_0_X/20160215_1421/CMSDIST/py2-dxr-fix-clang-linker-flags.patch
Done fetching /build/diguida/ext/CMSSW_8_0_X/20160215_1421/CMSDIST/py2-dxr-clang36.patch
Done fetching /build/diguida/ext/CMSSW_8_0_X/20160215_1421/CMSDIST/py2-dxr-sqlite38.patch
Done fetching /build/diguida/ext/CMSSW_8_0_X/20160215_1421/CMSDIST/py2-dxr-clang37.patch
Creating directory /build/diguida/ext/CMSSW_8_0_X/20160215_1421/a/BUILD/slc6_amd64_gcc493/external/py2-dxr/1.0-cms2 if not existing.
Building py2-dxr. Log can be found in /build/diguida/ext/CMSSW_8_0_X/20160215_1421/a/BUILD/slc6_amd64_gcc493/external/py2-dxr/1.0-cms2/log.
TMPDIR=/build/diguida/ext/CMSSW_8_0_X/20160215_1421/a/tmp  rpmbuild  --buildroot /build/diguida/ext/CMSSW_8_0_X/20160215_1421/a/tmp/BUILDROOT/c78b11cae7fa71e6f79947f7ff8af668 -ba --define '_topdir /build/diguida/ext/CMSSW_8_0_X/20160215_1421/a' --define "compiling_processes 24"  --define "cmscompilerv  493" --define "cmsos slc6_amd64" /build/diguida/ext/CMSSW_8_0_X/20160215_1421/a/SPECS/external/py2-dxr/1.0-cms2/spec >/build/diguida/ext/CMSSW_8_0_X/20160215_1421/a/BUILD/slc6_amd64_gcc493/external/py2-dxr/1.0-cms2/log 2>&1
Build successful.
Trying to install the rpm package external+py2-dxr+1.0-cms2 just built.
Done
  • py2-dxr-toolfile
Starting to process package external+py2-dxr-toolfile+1.0-cms2
Checking if py2-dxr-toolfile is cached.
Package external+py2-dxr-toolfile+1.0-cms2 not found in repository. Queuing for build.
Dependencies for external+py2-dxr-toolfile+1.0-cms2: ['install-external+py2-futures+2.2.0-kpegke', 'install-external+py2-jinja+2.7.2-kpegke', 'install-external+py2-markupsafe+0.23-kpegke', 'install-external+py2-ordereddict+1.1-kpegke', 'install-external+py2-parsimonious+0.6.1-kpegke', 'install-external+py2-dxr+1.0-cms2', 'fetch-external+py2-dxr-toolfile+1.0-cms2']
Fetching sources for py2-dxr-toolfile
No sources to be downloaded found for packages py2-dxr-toolfile.
Creating directory /build/diguida/ext/CMSSW_8_0_X/20160215_1421/a/BUILD/slc6_amd64_gcc493/external/py2-dxr-toolfile/1.0-cms2 if not existing.
Building py2-dxr-toolfile. Log can be found in /build/diguida/ext/CMSSW_8_0_X/20160215_1421/a/BUILD/slc6_amd64_gcc493/external/py2-dxr-toolfile/1.0-cms2/log.
TMPDIR=/build/diguida/ext/CMSSW_8_0_X/20160215_1421/a/tmp  rpmbuild  --buildroot /build/diguida/ext/CMSSW_8_0_X/20160215_1421/a/tmp/BUILDROOT/7fccf7c4891cbef0481fd6010c70a56c -ba --define '_topdir /build/diguida/ext/CMSSW_8_0_X/20160215_1421/a' --define "compiling_processes 24"  --define "cmscompilerv  493" --define "cmsos slc6_amd64" /build/diguida/ext/CMSSW_8_0_X/20160215_1421/a/SPECS/external/py2-dxr-toolfile/1.0-cms2/spec >/build/diguida/ext/CMSSW_8_0_X/20160215_1421/a/BUILD/slc6_amd64_gcc493/external/py2-dxr-toolfile/1.0-cms2/log 2>&1
Build successful.
Trying to install the rpm package external+py2-dxr-toolfile+1.0-cms2 just built.
Done
  • cmssw-tool-conf:
Creating directory /build/diguida/ext/CMSSW_8_0_X/20160215_1421/a/BUILD/slc6_amd64_gcc493/cms/cmssw-tool-conf/32.0-cms2 if not existing.
Building cmssw-tool-conf. Log can be found in /build/diguida/ext/CMSSW_8_0_X/20160215_1421/a/BUILD/slc6_amd64_gcc493/cms/cmssw-tool-conf/32.0-cms2/log.
TMPDIR=/build/diguida/ext/CMSSW_8_0_X/20160215_1421/a/tmp  rpmbuild  --buildroot /build/diguida/ext/CMSSW_8_0_X/20160215_1421/a/tmp/BUILDROOT/147217ac7713c6398a52a0aff8650632 -ba --define '_topdir /build/diguida/ext/CMSSW_8_0_X/20160215_1421/a' --define "compiling_processes 24"  --define "cmscompilerv  493" --define "cmsos slc6_amd64" /build/diguida/ext/CMSSW_8_0_X/20160215_1421/a/SPECS/cms/cmssw-tool-conf/32.0-cms2/spec >/build/diguida/ext/CMSSW_8_0_X/20160215_1421/a/BUILD/slc6_amd64_gcc493/cms/cmssw-tool-conf/32.0-cms2/log 2>&1
Build successful.
Nothing else to be done, exiting.
Trying to install the rpm package cms+cmssw-tool-conf+32.0-cms2 just built.
Done
Requested to quit.
Requested to quit.
Requested to quit.
Requested to quit.
3192.26user 4132.25system 2:05:44elapsed 494%CPU (0avgtext+0avgdata 2740432maxresident)k
813416inputs+94008320outputs (3214major+584732551minor)pagefaults 0swaps

@smuzaffar
Copy link
Contributor

@Degano , can you please cherry-pick 1ac4cd1#diff-085ccbfcce396237ceea010830b8c75c for IB/CMSSW_8_0_X/stable?

@diguida
Copy link
Author

diguida commented Feb 16, 2016

@smuzaffar I can push it here

@diguida
Copy link
Author

diguida commented Feb 16, 2016

@smuzaffar @Degano my local branch fasthadd_2.1 contains already the cherry-pick of 1ac4cd1
Do you want me to push it here?

@smuzaffar
Copy link
Contributor

@diguida , yes please. thanks

@diguida
Copy link
Author

diguida commented Feb 16, 2016

done

@cmsbuild
Copy link
Contributor

Pull request #2147 was updated.

@diguida
Copy link
Author

diguida commented Feb 16, 2016

please test

@cmsbuild
Copy link
Contributor

@diguida diguida changed the title New minor version of fasthadd, 2.1 New minor version of fasthadd, 2.1, and move llvm sqlite from BuildRequires and Requires for py2-dxr Feb 16, 2016
@diguida
Copy link
Author

diguida commented Feb 16, 2016

It seems that the test failed due to a test setup failure, as the jenkins log reports:

+ export CMSDIST_COMMIT=501ddeb65a6f8eac413700dad764f18a204d0f5e
+ CMSDIST_COMMIT=501ddeb65a6f8eac413700dad764f18a204d0f5e
+ cd /build/cmsbuild/jenkins-workarea/workspace/cmsdist-test-pr
+ /build/cmsbuild/jenkins-workarea/workspace/cmsdist-test-pr/cms-bot/report-pull-request-results TESTS_RUNNING --repo cms-sw/cmsdist --pr 2147 -c 501ddeb65a6f8eac413700dad764f18a204d0f5e --pr-job-id 105
Traceback (most recent call last):
  File "/build/cmsbuild/jenkins-workarea/workspace/cmsdist-test-pr/cms-bot/report-pull-request-results", line 6, in <module>
    import requests
ImportError: No module named requests
Build step 'Execute shell' marked build as failure
Notifying upstream projects of job completion
Loading slave statistic
Slave statistic loaded
Finished: FAILURE

@diguida
Copy link
Author

diguida commented Feb 16, 2016

please test
let's retry

@cmsbuild
Copy link
Contributor

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/cmsdist-test-pr/107/console

@cmsbuild
Copy link
Contributor

@diguida
Copy link
Author

diguida commented Feb 17, 2016

@smuzaffar can we have this merged and available in next IB? We need it to appear in cmsdist RPM repo in order to deploy it online.

@smuzaffar
Copy link
Contributor

We are building 8.0.0 today. @davidlange6 , are you OK to get it in? Nothing explicitly depends on it in cmssw. It is just a dependency on cmssw-tool-conf (with out any scram tool file) so that it can be build and upload at the time of release build.

@davidlange6
Copy link
Contributor

yes, its fine (but would be more appropriate to raise in the release meeting next time

On Feb 17, 2016, at 9:42 AM, Malik Shahzad Muzaffar notifications@github.com wrote:

We are building 8.0.0 today. @davidlange6 , are you OK to get it in? Nothing explicitly depends on it in cmssw. It is just a dependency on cmssw-tool-conf (with out any scram tool file) so that it can be build and upload at the time of release build.


Reply to this email directly or view it on GitHub.

smuzaffar added a commit that referenced this pull request Feb 17, 2016
New minor version of fasthadd, 2.1, and move llvm sqlite from BuildRequires and Requires for py2-dxr
@smuzaffar smuzaffar merged commit 369653f into cms-sw:IB/CMSSW_8_0_X/stable Feb 17, 2016
@diguida
Copy link
Author

diguida commented Feb 17, 2016

@davidlange6 thanks. Sorry about that: I should have requested it at the ORP meeting, or at least reported in the agenda (the orp-pendig label is there).

@diguida
Copy link
Author

diguida commented Feb 17, 2016

BTW I have just added it in the yesterday's agenda, for completeness.

@davidlange6
Copy link
Contributor

Thanks (but that definitely not my point!)

On Feb 17, 2016, at 10:48 AM, Salvatore Di Guida notifications@github.com wrote:

BTW I have just added it in the yesterday's agenda, for completeness.


Reply to this email directly or view it on GitHub.

@diguida
Copy link
Author

diguida commented Feb 17, 2016

@davidlange6's point - all requests for changes in externals should be reported in the ORP agenda and discussed and the ORP meeting - taken!
Thanks

@deguio
Copy link
Contributor

deguio commented Feb 18, 2016

my fault. I forgot to mention this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

6 participants