Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Online file-locking improvements (75X) #9385

Merged
merged 9 commits into from Jun 2, 2015
Merged

Online file-locking improvements (75X) #9385

merged 9 commits into from Jun 2, 2015

Conversation

smorovic
Copy link
Contributor

@smorovic smorovic commented Jun 1, 2015

  • removed unnecessary file operations on fu.lock file modification
  • configurable fu lock poll interval, reduce default value to 2ms, as it is measured to perform better with NFS over 1-Gbit interfaces in old FUs
  • LOCK_EX can be LOCK_SH when taking local lock and acquiring new index(raw) file. It only needs to remain exclusive when writing EoLS file marker in FU.
  • writing lock acquire waiting times statistics in fast monitoring service json (for later injection into elastic)
  • print warning if there is mismatch between BU (file) and TCDS record lumisection or orbit number
  • explicit report of exception from input source to the fast monitoring service so that output can be suppressed (was not being detected with adler32 exception thrown)

@smorovic
*removing first lock file flush which is not needed because flush with a
3f7dbc1 
new content is done later
@smorovic
*LOCK_EX for locking in supervisor thread can be replaced with LOCK_S…
0174ae3 
…H,becauses processes

don't need to mutually synchronize with ordering of index json file copying locally, only with EoLS (done in main thread with other lock instance) which remains exclusive
@smorovic
explicit disabling of json output before throwing checksum exception
079500a 
from nextEvent in the input source
@smorovic
print mismatch between file LS, TCDS LS and orbit number LS
80b010f
@smorovic
*lock file update improvements
00e4bc7 
*configurable sleep time between lock attempts
*monitoring of average lock acquire attempting time
@smorovic
*use warning for lumisection mismatch messages
6e03139
@smorovic
using accumulated lock time and count so that average can be calculated
3fec8c9 
correctly at any aggregation levels
@smorovic
*truncating before update of the lock file, reintroducing fsync
8a38371 
*compilation fixes
@smorovic smorovic mentioned this pull request Jun 1, 2015
Merged
@cmsbuild cmsbuild added this to the Next CMSSW_7_5_X milestone Jun 1, 2015
@cmsbuild
Copy link
Contributor

cmsbuild commented Jun 1, 2015

A new Pull Request was created by @smorovic (Srecko Morovic) for CMSSW_7_5_X.

Online file-locking improvements (75X)

It involves the following packages:

EventFilter/Utilities

@mommsen, @cmsbuild, @emeschi can you please review it and eventually sign? Thanks.
@Martin-Grunewald this is something you requested to watch as well.
You can sign-off by replying to this message having '+1' in the first line of your reply.
You can reject by replying to this message having '-1' in the first line of your reply.
If you are a L2 or a release manager you can ask for tests by saying 'please test' in the first line of a comment.
@Degano you are the release manager for this.
You can merge this pull request by typing 'merge' in the first line of your comment.

@mommsen
Copy link
Contributor

mommsen commented Jun 2, 2015

+1

@cmsbuild
Copy link
Contributor

cmsbuild commented Jun 2, 2015

This pull request is fully signed and it will be integrated in one of the next CMSSW_7_5_X IBs unless changes or unless it breaks tests. This pull request requires discussion in the ORP meeting before it's merged. @davidlange6, @Degano, @smuzaffar

@cmsbuild
Copy link
Contributor

cmsbuild commented Jun 2, 2015

+1
Tested at: 34b38b6
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-9385/5224/summary.html

@cmsbuild
Copy link
Contributor

cmsbuild commented Jun 2, 2015

This pull request is fully signed and it will be integrated in one of the next CMSSW_7_5_X IBs unless changes (tests are also fine). This pull request requires discussion in the ORP meeting before it's merged. @davidlange6, @Degano, @smuzaffar

@davidlange6
Copy link
Contributor

+1

cmsbuild added a commit that referenced this pull request Jun 2, 2015
@cmsbuild
Merge pull request #9385 from smorovic/filelocking-75X
e2c5480 
Online file-locking improvements (75X)
@cmsbuild cmsbuild merged commit e2c5480 into cms-sw:CMSSW_7_5_X Jun 2, 2015
@smorovic smorovic deleted the filelocking-75X branch November 13, 2015 10:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
comparison-pending daq-approved fully-signed tests-approved
Projects
None yet
Milestone
CMSSW_7_5_X
Development

Successfully merging this pull request may close these issues.

None yet
4 participants
@smorovic @cmsbuild @mommsen @davidlange6