Bad sizes in Popularity

[joseflix]

We have detected that sometimes in the monit_prod_cmssw_pop_* the files have associated a bad File Size. For example [1]:

The actual file size according to DAS [2] is 4191351315 (4.2GB). Sometimes there are accesses that show less bytes, always the same value. Which could be the reason for this?

This particular file is at PIC Tier-1 storage, and it has the correct value:

[root@dtn01 ~]# ls

-ltr /pnfs/pic.es/data/cms/disk/store/hidata/HIRun2018A/HIMinimumBias16/MINIAOD/PbPb18_MiniAODv1-v1/30000/413a6fdc-4752-41ee-b53e-f221f854ad5e.root
-rw-r--r-- 1 cms001 cms 4191351315 Apr 5 2021 /pnfs/pic.es/data/cms/disk/store/hidata/HIRun2018A/HIMinimumBias16/MINIAOD/PbPb18_MiniAODv1-v1/30000/413a6fdc-4752-41ee-b53e-f221f854ad5e.root

But, we see this happens for many other files...

Valentin pointed to me that we should open a ticket here. This is part of the CMSSW report on popularity, so this might need a deeper look.

[1] https://monit-kibana.cern.ch/kibana/app/kibana#/discover?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now%2Fy,to:now%2Fy))&_a=(columns:!(data.file_lfn,data.file_size,data.read_bytes,data.client_host),filters:!(('$state':(store:appState),meta:(alias:!n,disabled:!f,index:'79cbcb40-e78f-11ea-966a-e1c0a7950cea',key:data.file_lfn,negate:!f,params:(query:%2Fstore%2Fhidata%2FHIRun2018A%2FHIMinimumBias16%2FMINIAOD%2FPbPb18_MiniAODv1-v1%2F30000%2F413a6fdc-4752-41ee-b53e-f221f854ad5e.root),type:phrase,value:%2Fstore%2Fhidata%2FHIRun2018A%2FHIMinimumBias16%2FMINIAOD%2FPbPb18_MiniAODv1-v1%2F30000%2F413a6fdc-4752-41ee-b53e-f221f854ad5e.root),query:(match:(data.file_lfn:(query:%2Fstore%2Fhidata%2FHIRun2018A%2FHIMinimumBias16%2FMINIAOD%2FPbPb18_MiniAODv1-v1%2F30000%2F413a6fdc-4752-41ee-b53e-f221f854ad5e.root,type:phrase))))),index:'79cbcb40-e78f-11ea-966a-e1c0a7950cea',interval:auto,query:(language:kuery,query:''),sort:!(metadata.timestamp,desc))

[2] https://cmsweb.cern.ch/das/request?view=list&limit=50&instance=prod%2Fglobal&input=%2Fstore%2Fhidata%2FHIRun2018A%2FHIMinimumBias16%2FMINIAOD%2FPbPb18_MiniAODv1-v1%2F30000%2F413a6fdc-4752-41ee-b53e-f221f854ad5e.root

[cmsbuild]

A new Issue was created by @joseflix Josep Flix, PhD.

@Dr15Jones, @perrotta, @dpiparo, @makortel, @smuzaffar, @qliphy can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

[vkuznet]

I think the issue is related to info sent by CMSSW jobs via UDP to our collector. I suggest that somebody will check the information provided by jobs.

[makortel]

assign core

[cmsbuild]

New categories assigned: core

@Dr15Jones,@smuzaffar,@makortel you have been requested to review this Pull request/Issue and eventually sign? Thanks

[makortel]

Thanks @vkuznet for the clarification on where the information originates from.

While addressing #34873 we realized that the file size reported in the UDP packages was set in a non-thread-safe way (#34873 (comment)) that could cause problems like this.

The overhaul of StatisticsSenderService in #35505 should have fixed this issue as well. That PR was merged in 12_1_0_pre5. Would it be useful in earlier (active) release cycles as well?

[davidlange6]

Presumably these jobs are using older releases given that they are run 2 data. I agree it would be worth back porting this. (Not that backporitng it will solve all incorrect results, but would help)

…

On Dec 3, 2021, at 7:21 PM, Matti Kortelainen ***@***.***> wrote: Thanks @vkuznet for the clarification on where the information originates from. While addressing #34873 we realized that the file size reported in the UDP packages was set in a non-thread-safe way (#34873 (comment)) that could cause problems like this. The overhaul of StatisticsSenderService in #35505 should have fixed this issue as well. That PR was merged in 12_1_0_pre5. Would it be useful in earlier (active) release cycles as well? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android.

[makortel]

@perrotta asked in #36355 (comment)

just to get an idea of the plans: would these backports in old but still active cycles require a new (patch) release? Or they should be merged "just if a new release has to be built independently"? (Or maybe they don't even need a new release: sorry, I don't know the details about how those popularity monitorings work...)

New releases will be needed for these changes to take effect. The changes themselves wouldn't necessarily warrant new (patch) releases, but perhaps we should in order to get the updated code to be run by WM?

[makortel]

Backports down to 9_4_X have been done.

[makortel]

The backport PRs have been merged

[makortel]

+1

[cmsbuild]

This issue is fully signed and ready to be closed.

[joseflix]

Hi there, Sorry, but is this solved already? I still see values which are not consistent in the popularity kibana views, for example the snapshot I attach.

In Popularity we don't have the CMSSW version used:

So, I am not sure if this is because those jobs are using a CMSSW where the fix back porting was not applied?

[joseflix]

@makortel @qliphy

[davidlange6]

Old releases will not get this fix. So production/analyst work that has not moved forward to the fixed releases (done for all release cycles 2016+ iiuc) will still give bad values.

…

On Mar 10, 2022, at 10:55 AM, Josep Flix, PhD ***@***.***> wrote: Hi there, Sorry, but is this solved already? I still see values which are not consistent in the popularity kibana views, for example the snapshot I attach. In Popularity we don't have the CMSSW version used: So, I am not sure if this is because those jobs are using a CMSSW where the fix back porting was not applied? — Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android. You are receiving this because you commented.

[joseflix]

Hi @davidlange6 , thanks for the answer. How shall I know which release was using the jobs I report here? Maybe we should add the cmssw version on the popu records?

[davidlange6]

Yea, I don't see anything in https://github.com/cms-sw/cmssw/blob/master/Utilities/StorageFactory/src/StatisticsSenderService.cc From which this can be derived unless there is something useful in the app_info field.

…

On Mar 10, 2022, at 11:41 AM, Josep Flix, PhD ***@***.***> wrote: Hi @davidlange6 , thanks for the answer. How shall I know which release was using the jobs I report here? Maybe we should add the cmssw version on the popu records? — Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android. You are receiving this because you were mentioned.

[makortel]

For a limited number of cases DAS/DBS should tell the release a file/dataset was produced, so at least for the example case you should be able to check that. For longer term, adding CMSSW release field to the popularity information looks like straightforward to add.

[vkuznet]

In DAS getting release info as simple as:

./dasgoclient -query="release dataset=/ZMM/Summer11-DESIGN42_V11_428_SLHC1-v1/GEN-SIM"
CMSSW_4_2_8_SLHC1

In DBS it is also trivial

https://cmsweb.cern.ch/dbs/prod/global/DBSReader/releaseversions?dataset=/ZMM/Summer11-DESIGN42_V11_428_SLHC1-v1/GEN-SIM

and it yields:

[
{"release_version":"CMSSW_4_2_8_SLHC1"}
,{"release_version":"CMSSW_4_2_8_SLHC1"}
,{"release_version":"CMSSW_4_2_8_SLHC1"}
]

(I need to check why we have three identical entries, most likely loose constrain on SQL query JOINS).

[davidlange6]

You want the release used by the cmsRun job not the release used to create the input data set, no? (But right, if you have the output dataset you could look it up that way)

…

On Mar 10, 2022, at 4:52 PM, Matti Kortelainen ***@***.***> wrote: For a limited number of cases DAS/DBS should tell the release a file/dataset was produced, so at least for the example case you should be able to check that. For longer term, adding CMSSW release field to the popularity information looks like straightforward to add. — Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android. You are receiving this because you were mentioned.

[joseflix]

Yes, I want the release used by cmsRun to open those files. We don't have any clue on the output generate files in popularity DB, if I am not mistaken, only input files...

[makortel]

You want the release used by the cmsRun job not the release used to create the input data set, no?

Ah, good point. Ok, we can add the CMSSW version in the popularity data. Unfortunately, as David mentioned above, that will be available only in new releases (of old release cycles).

[makortel]

#37220 adds CMSSW version to the UDP packets.