Make the online DQM fileinputsource more intelligent #16973
Conversation
To allow reprocessing StreamExpress runs. One dataset (as returned from DAS) can have many runs, so we allow selecting a specific run. The dataset always has all events and will be fully processed, which can be too much for the online DQM, therefore we allow selecting a subset of LS.
|
A new Pull Request was created by @schneiml (Marcel Schneider) for CMSSW_9_0_X. It involves the following packages: DQM/Integration @cmsbuild, @dmitrijus, @vanbesien, @davidlange6 can you please review it and eventually sign? Thanks. cms-bot commands are listed here #13028 |
|
+1 |
cmsbuild
added
dqm-approved
fully-signed
and removed
dqm-pending
pending-signatures
labels
|
The tests are being triggered in jenkins. |
|
This pull request is fully signed and it will be integrated in one of the next CMSSW_9_0_X IBs after it passes the integration tests. This pull request requires discussion in the ORP meeting before it's merged. @slava77, @davidlange6, @smuzaffar |
|
Regardless dataset_cfi.py, would it not make sense to query these from DAS directly? |
|
Yes it would, but I'd have to figure out how to do this. Any hints where to steal the relevant code? |
|
Comparison job queued. |
|
Progress note here: I found the snippet that queries DAS for the runTheMatrix sequences and it looks like that can be used here. Now I just have to come up with a working DAS query that terminates in finite time and test it once I have time and DAS is responding. |
|
hold |
|
Pull request has been put on hold by @dmitrijus |
cmsbuild
added
comparison-pending
dqm-pending
tests-pending
and removed
dqm-approved
tests-approved
labels
|
Pull request #16973 was updated. @cmsbuild, @dmitrijus, @vanbesien, @davidlange6 can you please check and sign again. |
|
@dmitrijus I added the DAS query support. Main issue is that it can take a while to get results, but that is pretty much expected. Therefore I allowed having the Works with the default options, might need some tuning for older runs... Bonus: |
|
super! |
|
+1 |
|
The tests are being triggered in jenkins. |
|
unhold |
|
This pull request is fully signed and it will be integrated in one of the next CMSSW_9_0_X IBs after it passes the integration tests. This pull request requires discussion in the ORP meeting before it's merged. @davidlange6, @smuzaffar |
|
Comparison job queued. |
|
+1 |
This allows reprocessing StreamExpress runs, as it was needed to analyse some Pilot Blade timing scan data after the online shutdown. It could also be useful in the future and for general Phase1 online DQM development.
One dataset (as returned from DAS) can have many runs, so we allow selecting a specific run. The
dataset always has all events and will be fully processed, which can be too much for the online DQM, therefore we allow selecting a subset of LS (like in the lookarea).
Remaining problem: The input files as reported from DAS appear in random order, so it can happen that many files have to be opened to find a small LS range, and the data will appear randomly in the DQM live view.
This configuration includes the dataset with the last 2016 pPb runs (again, as they were interesting for the Pilot Blade team and culd not be analysed using the online system). It is by default enabled in the
pixelandpixel_uponline clients after #16875 .