Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

XRootD: Update to version 5.4.0 and build with scitokens-cpp #7594

Merged
merged 6 commits into from Feb 24, 2022

Conversation

iarspider
Copy link
Contributor

No description provided.

@cmsbuild
Copy link
Contributor

cmsbuild commented Feb 1, 2022

A new Pull Request was created by @iarspider for branch IB/CMSSW_12_3_X/master.

@cmsbuild, @smuzaffar, @iarspider can you please review it and eventually sign? Thanks.
@perrotta, @dpiparo, @qliphy you are the release manager for this.
cms-bot commands are listed here

@iarspider
Copy link
Contributor Author

please test for alc7_amd64_gcc11

@iarspider
Copy link
Contributor Author

please test for slc7_amd64_gcc11

@iarspider
Copy link
Contributor Author

please test for slc7_aarch64_gcc11

@iarspider
Copy link
Contributor Author

please test for slc7_ppc64le_gcc11

@smuzaffar
Copy link
Contributor

thanks @iarspider . Have you checked if scitokens-cpp generates any shared library? If yes then we need a scram tool file for it too and update xroot toolfile to have dependency on that.

@davidlange6
Copy link
Contributor

hi - has there been any discussion of what (if any) new requirements having client-side tokens libraries will have on the need to update old release cycles in the future? Eg, will old external stacks become more problematic with this evolution?

@smuzaffar
Copy link
Contributor

smuzaffar commented Feb 1, 2022

No @davidlange6 I do not know any such discussion. @bockjoo do you know ? Note that currently xrootd 5.3 is only available in CMSSW 12.3.X, and there was no request to back port it to old release cycles.

@davidlange6
Copy link
Contributor

davidlange6 commented Feb 1, 2022 via email

@bockjoo
Copy link

bockjoo commented Feb 1, 2022

@smuzaffar I don't know any such discussion.
By the way, just heads-up:
Although succeeding, xrdcp in CMSSW_12_3_0_pre4 behaves differently from the one in XRootD 5.4 with the token issued by https://cms-auth.web.cern.ch/
and I can open an AAA file with CMSSW_12_3_0_pre4 but failed to read the file with error '[ERROR] Socket timeout' (errno=0, code=103).

@iarspider
Copy link
Contributor Author

@smuzaffar should I wait until tests are done and then push the toolfile? I don't want to abort the builds

@cmsbuild
Copy link
Contributor

cmsbuild commented Feb 1, 2022

-1

Failed Tests: UnitTests RelVals AddOn
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-dee40b/22119/summary.html
COMMIT: 64b9920
CMSSW: CMSSW_12_3_X_2022-01-31-1100/slc7_amd64_gcc11
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/7594/22119/install.sh to create a dev area with all the needed externals and cmssw changes.

Unit Tests

I found errors in the following unit tests:

---> test TestDQMOnlineClient-csc_dqm_sourceclient had ERRORS
---> test TestDQMOnlineClient-ecal_dqm_sourceclient had ERRORS
---> test testAlignmentOfflineValidation had ERRORS
---> test TestDQMOnlineClient-pixel_dqm_sourceclient had ERRORS
and more ...

RelVals

  • 5.15.1_TTbar+TTbarFS+HARVESTFS/step1_TTbar+TTbarFS+HARVESTFS.log
  • 135.4135.4_ZEE_13+ZEEFS_13+HARVESTUP15FS+MINIAODMCUP15FS/step1_ZEE_13+ZEEFS_13+HARVESTUP15FS+MINIAODMCUP15FS.log
  • 136.731136.731_RunSinglePh2016B+RunSinglePh2016B+HLTDR2_2016+RECODR2_2016reHLT_skimSinglePh_HIPM+HARVESTDR2_skimSinglePh/step2_RunSinglePh2016B+RunSinglePh2016B+HLTDR2_2016+RECODR2_2016reHLT_skimSinglePh_HIPM+HARVESTDR2_skimSinglePh.log
Expand to see more relval errors ...

AddOn Tests

  • fastsimcmsDriver.py TTbar_8TeV_TuneCUETP8M1_cfi --conditions auto:run1_mc --fast -n 100 --eventcontent AODSIM,DQM --relval 100000,1000 -s GEN,SIM,RECOBEFMIX,DIGI:pdigi_valid,L1,DIGI2RAW,L1Reco,RECO,VALIDATION --customise=HLTrigger/Configuration/CustomConfigs.L1THLT --datatier GEN-SIM-DIGI-RECO,DQMIO --beamspot Realistic8TeVCollision : FAILED - time: date Tue Feb 1 17:47:02 2022-date Tue Feb 1 17:46:53 2022 s - exit: 256
  • fastsim1cmsDriver.py TTbar_13TeV_TuneCUETP8M1_cfi --conditions auto:run2_mc_l1stage1 --fast -n 100 --eventcontent AODSIM,DQM --relval 100000,1000 -s GEN,SIM,RECOBEFMIX,DIGI:pdigi_valid,L1,DIGI2RAW,L1Reco,RECO,VALIDATION --customise=HLTrigger/Configuration/CustomConfigs.L1THLT --datatier GEN-SIM-DIGI-RECO,DQMIO --beamspot NominalCollision2015 --era Run2_25ns : FAILED - time: date Tue Feb 1 17:47:02 2022-date Tue Feb 1 17:46:57 2022 s - exit: 256
  • fastsim2cmsDriver.py TTbar_13TeV_TuneCUETP8M1_cfi --conditions auto:run2_mc --fast -n 100 --eventcontent AODSIM,DQM --relval 100000,1000 -s GEN,SIM,RECOBEFMIX,DIGI:pdigi_valid,L1,DIGI2RAW,L1Reco,RECO,VALIDATION --customise=HLTrigger/Configuration/CustomConfigs.L1THLT --datatier GEN-SIM-DIGI-RECO,DQMIO --beamspot NominalCollision2015 --era Run2_2016 : FAILED - time: date Tue Feb 1 17:47:03 2022-date Tue Feb 1 17:46:59 2022 s - exit: 256
Expand to see more addon errors ...

@cmsbuild
Copy link
Contributor

cmsbuild commented Feb 1, 2022

Pull request #7594 was updated.

@iarspider
Copy link
Contributor Author

please test

@cmsbuild
Copy link
Contributor

cmsbuild commented Feb 2, 2022

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-dee40b/22133/summary.html
COMMIT: 84c06cb
CMSSW: CMSSW_12_3_X_2022-02-01-1100/slc7_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/7594/22133/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

  • No significant changes to the logs found
  • Reco comparison results: 16478 differences found in the comparisons
  • DQMHistoTests: Total files compared: 43
  • DQMHistoTests: Total histograms compared: 3449612
  • DQMHistoTests: Total failures: 137311
  • DQMHistoTests: Total nulls: 1
  • DQMHistoTests: Total successes: 3312278
  • DQMHistoTests: Total skipped: 22
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: -0.641 KiB( 42 files compared)
  • DQMHistoSizes: changed ( 11834.0 ): -0.645 KiB SiStrip/MechanicalView
  • DQMHistoSizes: changed ( 312.0 ): 0.004 KiB MessageLogger/Warnings
  • Checked 181 log files, 42 edm output root files, 43 DQM output files
  • TriggerResults: found differences in 6 / 42 workflows

@cmsbuild
Copy link
Contributor

Pull request #7594 was updated.

@smuzaffar smuzaffar changed the title XRootD: build with scitokens-cpp XRootD: Update to version 5.4.0 and build with scitokens-cpp Feb 23, 2022
@smuzaffar
Copy link
Contributor

please test

@cmsbuild
Copy link
Contributor

-1

Failed Tests: UnitTests RelVals
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-dee40b/22612/summary.html
COMMIT: 3a8f5cc
CMSSW: CMSSW_12_3_X_2022-02-22-2300/slc7_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/7594/22612/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-dee40b/22612/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-dee40b/22612/git-merge-result

Unit Tests

I found errors in the following unit tests:

---> test test_edmPickEvents had ERRORS

RelVals

----- Begin Fatal Exception 23-Feb-2022 17:08:05 CET-----------------------
An exception of category 'FatalRootError' occurred while
   [0] Constructing the EventProcessor
   [1] Constructing service of type InitRootHandlers
   Additional Info:
      [a] Fatal Root Error: @SUB=TInterpreter::TCling::RegisterModule
Problems in compiling forward declarations for module DataFormatsStdDictionaries_x3r: '
#line 1 "DataFormatsStdDictionaries_x3r dictionary forward declarations' payload"
#pragma clang diagnostic ignored "-Wkeyword-compat"
#pragma clang diagnostic ignored "-Wignored-attributes"
#pragma clang diagnostic ignored "-Wreturn-type-c-linkage"
extern int __Cling_AutoLoading_Map;
namespace std{template <typename _Tp> struct __attribute__((annotate("$clingAutoload$bits/unique_ptr.h")))  __attribute__((annotate("$clingAutoload$memory")))  default_delete;
}
namespace std{template <typename _Tp> class __attribute__((annotate("$clingAutoload$bits/allocator.h")))  __attribute__((annotate("$clingAutoload$string")))  allocator;
}
namespace std{template <typename _T1, typename _T2> struct __attribute__((annotate("$clingAutoload$bits/stl_iterator.h")))  __attribute__((annotate("$clingAutoload$string")))  pair;
}
namespace std{template <class _CharT> struct __attribute__((annotate("$clingAutoload$bits/char_traits.h")))  __attribute__((annotate("$clingAutoload$string")))  char_traits;
}
namespace std{template <typename _Tp = void> struct __attribute__((annotate("$clingAutoload$bits/stl_function.h")))  __attribute__((annotate("$clingAutoload$string")))  less;
}
'

----- End Fatal Exception -------------------------------------------------
----- Begin Fatal Exception 23-Feb-2022 17:08:05 CET-----------------------
An exception of category 'FatalRootError' occurred while
   [0] Constructing the EventProcessor
   [1] Constructing service of type InitRootHandlers
   Additional Info:
      [a] Fatal Root Error: @SUB=TInterpreter::TCling::RegisterModule
Problems in compiling forward declarations for module DataFormatsStdDictionaries_x3r: '
#line 1 "DataFormatsStdDictionaries_x3r dictionary forward declarations' payload"
#pragma clang diagnostic ignored "-Wkeyword-compat"
#pragma clang diagnostic ignored "-Wignored-attributes"
#pragma clang diagnostic ignored "-Wreturn-type-c-linkage"
extern int __Cling_AutoLoading_Map;
namespace std{template <typename _Tp> struct __attribute__((annotate("$clingAutoload$bits/unique_ptr.h")))  __attribute__((annotate("$clingAutoload$memory")))  default_delete;
}
namespace std{template <typename _Tp> class __attribute__((annotate("$clingAutoload$bits/allocator.h")))  __attribute__((annotate("$clingAutoload$string")))  allocator;
}
namespace std{template <typename _T1, typename _T2> struct __attribute__((annotate("$clingAutoload$bits/stl_iterator.h")))  __attribute__((annotate("$clingAutoload$string")))  pair;
}
namespace std{template <class _CharT> struct __attribute__((annotate("$clingAutoload$bits/char_traits.h")))  __attribute__((annotate("$clingAutoload$string")))  char_traits;
}
namespace std{template <typename _Tp = void> struct __attribute__((annotate("$clingAutoload$bits/stl_function.h")))  __attribute__((annotate("$clingAutoload$string")))  less;
}
'

----- End Fatal Exception -------------------------------------------------
----- Begin Fatal Exception 23-Feb-2022 17:08:05 CET-----------------------
An exception of category 'FatalRootError' occurred while
   [0] Constructing the EventProcessor
   [1] Constructing service of type InitRootHandlers
   Additional Info:
      [a] Fatal Root Error: @SUB=TInterpreter::TCling::RegisterModule
Problems in compiling forward declarations for module DataFormatsStdDictionaries_x3r: '
#line 1 "DataFormatsStdDictionaries_x3r dictionary forward declarations' payload"
#pragma clang diagnostic ignored "-Wkeyword-compat"
#pragma clang diagnostic ignored "-Wignored-attributes"
#pragma clang diagnostic ignored "-Wreturn-type-c-linkage"
extern int __Cling_AutoLoading_Map;
namespace std{template <typename _Tp> struct __attribute__((annotate("$clingAutoload$bits/unique_ptr.h")))  __attribute__((annotate("$clingAutoload$memory")))  default_delete;
}
namespace std{template <typename _Tp> class __attribute__((annotate("$clingAutoload$bits/allocator.h")))  __attribute__((annotate("$clingAutoload$string")))  allocator;
}
namespace std{template <typename _T1, typename _T2> struct __attribute__((annotate("$clingAutoload$bits/stl_iterator.h")))  __attribute__((annotate("$clingAutoload$string")))  pair;
}
namespace std{template <class _CharT> struct __attribute__((annotate("$clingAutoload$bits/char_traits.h")))  __attribute__((annotate("$clingAutoload$string")))  char_traits;
}
namespace std{template <typename _Tp = void> struct __attribute__((annotate("$clingAutoload$bits/stl_function.h")))  __attribute__((annotate("$clingAutoload$string")))  less;
}
'

----- End Fatal Exception -------------------------------------------------
Expand to see more relval errors ...

@smuzaffar
Copy link
Contributor

please test
lets get the latest IB

@cmsbuild
Copy link
Contributor

-1

Failed Tests: UnitTests
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-dee40b/22625/summary.html
COMMIT: 3a8f5cc
CMSSW: CMSSW_12_3_X_2022-02-23-1100/slc7_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/7594/22625/install.sh to create a dev area with all the needed externals and cmssw changes.

Unit Tests

I found errors in the following unit tests:

---> test test_edmPickEvents had ERRORS

Comparison Summary

Summary:

  • No significant changes to the logs found
  • Reco comparison results: 7 differences found in the comparisons
  • DQMHistoTests: Total files compared: 49
  • DQMHistoTests: Total histograms compared: 4001143
  • DQMHistoTests: Total failures: 11
  • DQMHistoTests: Total nulls: 1
  • DQMHistoTests: Total successes: 4001109
  • DQMHistoTests: Total skipped: 22
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: -0.004 KiB( 48 files compared)
  • DQMHistoSizes: changed ( 312.0 ): -0.004 KiB MessageLogger/Warnings
  • Checked 204 log files, 45 edm output root files, 49 DQM output files
  • TriggerResults: no differences found

@smuzaffar
Copy link
Contributor

please test for slc7_aarch64_gcc11

@smuzaffar
Copy link
Contributor

please test for slc7_ppc64le_gcc11

@smuzaffar
Copy link
Contributor

please test for cs8_amd64_gcc11

@cmsbuild
Copy link
Contributor

-1

Failed Tests: UnitTests
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-dee40b/22637/summary.html
COMMIT: 3a8f5cc
CMSSW: CMSSW_12_3_X_2022-02-23-2300/slc7_ppc64le_gcc11
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/7594/22637/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-dee40b/22637/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-dee40b/22637/git-merge-result

Unit Tests

I found errors in the following unit tests:

---> test testFWCoreUtilities had ERRORS
---> test DRNTest had ERRORS
---> test test_edmPickEvents had ERRORS
---> test materialBudgetTrackerPlots had ERRORS
and more ...

@cmsbuild
Copy link
Contributor

-1

Failed Tests: UnitTests RelVals
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-dee40b/22639/summary.html
COMMIT: 3a8f5cc
CMSSW: CMSSW_12_3_X_2022-02-23-2300/slc7_aarch64_gcc11
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/7594/22639/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-dee40b/22639/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-dee40b/22639/git-merge-result

Unit Tests

I found errors in the following unit tests:

---> test TestFWCoreServicesDriver had ERRORS
---> test testFWCoreUtilities had ERRORS
---> test test_edmPickEvents had ERRORS
---> test DRNTest had ERRORS
and more ...

RelVals

----- Begin Fatal Exception 24-Feb-2022 18:57:58 CET-----------------------
An exception of category 'Vertex' occurred while
   [0] Processing  Event run: 194533 lumi: 329 event: 462355458 stream: 0
   [1] Running path 'dqmofflineOnPAT_1_step'
   [2] Prefetching for module SingleTopTChannelLeptonDQM_miniAOD/'singleTopElectronMediumDQM_miniAOD'
   [3] Prefetching for module PATMuonSlimmer/'slimmedMuons'
   [4] Prefetching for module PATMuonSelector/'selectedPatMuons'
   [5] Prefetching for module PATMuonProducer/'patMuons'
   [6] Prefetching for module MuonProducer/'muons'
   [7] Prefetching for module PFProducer/'particleFlowTmp'
   [8] Prefetching for module PFBlockProducer/'particleFlowBlock'
   [9] Prefetching for module PFElecTkProducer/'pfTrackElec'
   [10] Prefetching for module PFConversionProducer/'pfConversions'
   [11] Calling method for module ConversionProducer/'allConversions'
Exception Message:
Refitted track not found in list
----- End Fatal Exception -------------------------------------------------
----- Begin Fatal Exception 24-Feb-2022 19:11:47 CET-----------------------
An exception of category 'Vertex' occurred while
   [0] Processing  Event run: 326479 lumi: 7 event: 1579493 stream: 0
   [1] Running path 'dqmoffline_8_step'
   [2] Prefetching for module SMPDQM/'SMPDQM'
   [3] Prefetching for module MuonProducer/'muons'
   [4] Prefetching for module PFProducer/'particleFlowTmp'
   [5] Prefetching for module PFBlockProducer/'particleFlowBlock'
   [6] Prefetching for module PFElecTkProducer/'pfTrackElec'
   [7] Prefetching for module PFConversionProducer/'pfConversions'
   [8] Calling method for module ConversionProducer/'allConversions'
Exception Message:
Refitted track not found in list
----- End Fatal Exception -------------------------------------------------
----- Begin Fatal Exception 24-Feb-2022 19:30:13 CET-----------------------
An exception of category 'Vertex' occurred while
   [0] Processing  Event run: 319450 lumi: 76 event: 106007323 stream: 0
   [1] Running path 'dqmoffline_10_step'
   [2] Prefetching for module SMPDQM/'SMPDQM'
   [3] Prefetching for module MuonProducer/'muons'
   [4] Prefetching for module PFProducer/'particleFlowTmp'
   [5] Prefetching for module PFBlockProducer/'particleFlowBlock'
   [6] Prefetching for module PFElecTkProducer/'pfTrackElec'
   [7] Prefetching for module PFConversionProducer/'pfConversions'
   [8] Calling method for module ConversionProducer/'allConversions'
Exception Message:
Refitted track not found in list
----- End Fatal Exception -------------------------------------------------

@smuzaffar smuzaffar changed the base branch from IB/CMSSW_12_3_X/master to IB/CMSSW_12_3_X/devel February 24, 2022 19:25
@smuzaffar
Copy link
Contributor

lets merge this for DEVEL IBs for full tests

@smuzaffar smuzaffar merged commit 6d7d10d into IB/CMSSW_12_3_X/devel Feb 24, 2022
@smuzaffar smuzaffar deleted the enable-libXrdSecztn branch February 24, 2022 19:26
@cmsbuild
Copy link
Contributor

-1

Failed Tests: UnitTests
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-dee40b/22638/summary.html
COMMIT: 3a8f5cc
CMSSW: CMSSW_12_3_X_2022-02-23-2300/cs8_amd64_gcc11
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/7594/22638/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-dee40b/22638/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-dee40b/22638/git-merge-result

Unit Tests

I found errors in the following unit tests:

---> test test_edmPickEvents had ERRORS

Comparison Summary

Summary:

  • No significant changes to the logs found
  • Reco comparison results: 69191 differences found in the comparisons
  • DQMHistoTests: Total files compared: 49
  • DQMHistoTests: Total histograms compared: 4001143
  • DQMHistoTests: Total failures: 483442
  • DQMHistoTests: Total nulls: 373
  • DQMHistoTests: Total successes: 3517306
  • DQMHistoTests: Total skipped: 22
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.358 KiB( 48 files compared)
  • DQMHistoSizes: changed ( 10224.0 ): -0.288 KiB SiStrip/MechanicalView
  • DQMHistoSizes: changed ( 11834.0 ): 2.899 KiB SiStrip/MechanicalView
  • DQMHistoSizes: changed ( 250202.181 ): -0.111 KiB SiStrip/MechanicalView
  • DQMHistoSizes: changed ( 25202.0 ): -0.129 KiB SiStrip/MechanicalView
  • DQMHistoSizes: changed ( 312.0 ): -0.004 KiB MessageLogger/Warnings
  • DQMHistoSizes: changed ( 7.3 ): -2.009 KiB SiStrip/MechanicalView
  • Checked 204 log files, 45 edm output root files, 49 DQM output files
  • TriggerResults: found differences in 14 / 48 workflows

@cmsbuild
Copy link
Contributor

backport done
Successfully backported PR #7594 as #7652 for branch IB/CMSSW_12_3_X/master

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants