DAQ file broker client (10_2_X) #23947

smorovic · 2018-07-23T12:14:34Z

Backport of #23944 (10_3_X)
Client implementation for new DAQ File Broker protocol

…ile service which will distribute input files

Signed-off-by: Srecko Morovic <srecko.morovic@cern.ch>

catch file not exists and check for run directory on BU handle NORUN fix READY with no file current LS handle JSON file move by server to subdirectory with prefix

support for socket keep alive (experimental)

fix undefined variable value when no raw file use shared lock for server contact and promote it to exclusive if EoLS file(s) have to be written

…ase of deletion of the run directory

access to copy JSON file from ramdisk. This allows better diagnosis of problems (such as recently seen bad file descriptor problem in some tests). This is enabled only for File Broker, while NFS locking code remains unchanged (as it is going to be phased out).

keep alive by default close socket on error NB: in tests it turns out that read_until reads the whole response. this is however not always guaranteed and updates will be needed to read based on HTTP Content-Length

cmsbuild · 2018-07-23T12:14:56Z

A new Pull Request was created by @smorovic (Srecko Morovic) for CMSSW_10_2_X.

It involves the following packages:

EventFilter/Utilities

@perrotta, @mommsen, @cmsbuild, @emeschi, @slava77 can you please review it and eventually sign? Thanks.
@Martin-Grunewald this is something you requested to watch as well.
@davidlange6, @slava77, @fabiocos you are the release manager for this.

cms-bot commands are listed here

Backported from DAQ file broker client (10_3_X) #23944

mommsen · 2018-07-23T14:23:05Z

please test

cmsbuild · 2018-07-23T14:23:25Z

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-any-integration/29342/console Started: 2018/07/23 16:24

cmsbuild · 2018-07-23T17:07:11Z

+1
Tested at: 09f835c
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-23947/29342/summary.html

cmsbuild · 2018-07-23T17:07:13Z

Comparison job queued.

mommsen · 2018-07-23T17:16:11Z

+1

cmsbuild · 2018-07-23T18:28:35Z

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-23947/29342/summary.html

Comparison Summary:

No significant changes to the logs found
Reco comparison results: 2 differences found in the comparisons
DQMHistoTests: Total files compared: 31
DQMHistoTests: Total histograms compared: 2892290
DQMHistoTests: Total failures: 2
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 2892098
DQMHistoTests: Total skipped: 190
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 30 files compared)
Checked 128 log files, 14 edm output root files, 31 DQM output files

slava77 · 2018-07-24T18:10:11Z

+1

for #23947 09f835c

the backport is correct (comparing to DAQ file broker client (10_3_X) #23944 4c7f2cd : only some of the removed code in EventFilter/Utilities/src/FedRawDataInputSource.cc differs by const& in two catch arguments). The code changes are not really related to reco and do not affect the standard workflows tested offline.
jenkins tests pass

cmsbuild · 2018-07-24T18:10:29Z

This pull request is fully signed and it will be integrated in one of the next CMSSW_10_2_X IBs (tests are also fine) and once validation in the development release cycle CMSSW_10_3_X is complete. This pull request will now be reviewed by the release team before it's merged. @davidlange6, @slava77, @smuzaffar, @fabiocos (and backports should be raised in the release meeting by the corresponding L2)

slava77 · 2018-07-24T18:11:40Z

backport of #23944

smorovic · 2018-07-24T18:49:41Z

Hi @slava77,
code which got rmoved from FedRawDataInputSource.cc ended in EvFDaqDirector.cc with modifications. There I converted all exceptions to const&.

kpedro88 · 2018-07-25T19:00:11Z

+1

smorovic added 20 commits July 23, 2018 14:07

reorganizing source and service to support both old locking and new f…

ded3225

…ile service which will distribute input files

2nd part of modifications to support new FFF file service interface

0ce25b3

handle max LS (cloud stopping) and updates to the API

28d786f

fixes to file locking and changes to the file service protocol

742ae6e

back off mechanism to reduce number of open sockets in BU service

602af5a

allow stop LS 0

0f0e8bd

Signed-off-by: Srecko Morovic <srecko.morovic@cern.ch>

logging for stop LS

7ade6d4

pid added to HTTP attributes (for possible use in renaming)

26b08bf

catch file not exists and check for run directory on BU handle NORUN fix READY with no file current LS handle JSON file move by server to subdirectory with prefix

support for optional unlocking

76b9f15

support for socket keep alive (experimental)

rename fileService to fileBroker

969847d

fix undefined variable value when no raw file use shared lock for server contact and promote it to exclusive if EoLS file(s) have to be written

write EoLS files before index files from next LS are written locally

406703b

remove shared lock implementation which doesn't work correctly

958ca03

removing unused lock file function

aa41c87

proper error recovery (avoiding assertions, segmentation faults) in c…

7e85cf1

…ase of deletion of the run directory

Use HTTP 1.1 as it will be needed for keep alive

7ef6400

fix for keep-alive

bac7414

2nd round of keep-alive fixes

1ee9ac4

keep alive by default close socket on error NB: in tests it turns out that read_until reads the whole response. this is however not always guaranteed and updates will be needed to read based on HTTP Content-Length

catching const& exceptions

c807ce6

remove double const

09f835c

cmsbuild added this to the CMSSW_10_2_X milestone Jul 23, 2018

cmsbuild added comparison-pending daq-pending orp-pending pending-signatures reconstruction-pending tests-pending labels Jul 23, 2018

cmsbuild added tests-started and removed tests-pending labels Jul 23, 2018

cmsbuild added tests-approved and removed tests-started labels Jul 23, 2018

cmsbuild added daq-approved and removed daq-pending labels Jul 23, 2018

cmsbuild added comparison-available and removed comparison-pending labels Jul 23, 2018

cmsbuild added fully-signed reconstruction-approved and removed pending-signatures reconstruction-pending labels Jul 24, 2018

cmsbuild added the backport label Jul 24, 2018

slava77 mentioned this pull request Jul 24, 2018

DAQ file broker client (10_1_X) #23948

Merged

cmsbuild added backport-ok and removed backport labels Jul 24, 2018

cmsbuild added orp-approved and removed orp-pending labels Jul 25, 2018

cmsbuild merged commit aa73fa4 into cms-sw:CMSSW_10_2_X Jul 25, 2018

smorovic deleted the newFileServer-10_2_X branch August 23, 2018 10:18

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DAQ file broker client (10_2_X) #23947

DAQ file broker client (10_2_X) #23947

smorovic commented Jul 23, 2018 •

edited

cmsbuild commented Jul 23, 2018 •

edited

mommsen commented Jul 23, 2018

cmsbuild commented Jul 23, 2018 •

edited

cmsbuild commented Jul 23, 2018

cmsbuild commented Jul 23, 2018

mommsen commented Jul 23, 2018

cmsbuild commented Jul 23, 2018

slava77 commented Jul 24, 2018

cmsbuild commented Jul 24, 2018

slava77 commented Jul 24, 2018

smorovic commented Jul 24, 2018

kpedro88 commented Jul 25, 2018

DAQ file broker client (10_2_X) #23947

DAQ file broker client (10_2_X) #23947

Conversation

smorovic commented Jul 23, 2018 • edited

cmsbuild commented Jul 23, 2018 • edited

mommsen commented Jul 23, 2018

cmsbuild commented Jul 23, 2018 • edited

cmsbuild commented Jul 23, 2018

cmsbuild commented Jul 23, 2018

mommsen commented Jul 23, 2018

cmsbuild commented Jul 23, 2018

slava77 commented Jul 24, 2018

cmsbuild commented Jul 24, 2018

slava77 commented Jul 24, 2018

smorovic commented Jul 24, 2018

kpedro88 commented Jul 25, 2018

smorovic commented Jul 23, 2018 •

edited

cmsbuild commented Jul 23, 2018 •

edited

cmsbuild commented Jul 23, 2018 •

edited