CMSSW metrics for FWJR #11663

vkuznet · 2023-07-17T17:26:21Z

Fixes #11538
Superseeds #11575

Status

ready

Description

Implement CMSSW metrics for WMCore FWJR report, and perform data-type casting of all existing metrics, i.e. int, float, boolean metrics values if they are represented in string form are converted to int, float, boolean data-type values, respectively, e.g. "1" string value is converted to 1 integer one, or "1.1" string value is converted to 1.1 float value. In addition, extra spaces surrounding strings are discarded, e.g. " foo " is converted to "foo".

This PR also provides proper way to handle XrdSiteStatistics metrics which are presented via the following XML form:

    <PerformanceReport>
      <PerformanceModule Metric="cern.ch"  Module="XrdSiteStatistics" >
        <Metric Name="read-numOperations" Value="2"/>
        ...
      </PerformanceModule>
    </PerformanceReport>

and transformed into the following representation in JSON:

{...,
        "XrdSiteStatistics": {
            "readv-numOperations": [
              {
                "site": "cern.ch",
                "value": 2
              }, ... ],
            ....
           }
}

This PR also provides a fresh CMSSWMergeReport2.xml file from one of the recent CMSSW jobs and made unit testing against it.

TODO: it is required that we should make a decision about preservation of current performance metrics since it may affect backward compatibility. I think we should preserve them for time being until new docs with full set of CMSSW metrics will be propagated to WMArchive/MONIT, and then we'll adjust existing dashboard to switch from current performance metrics (cpu, storage, memory) to a full set of CMSSW ones. Therefore, further work on WMArchive/MONIT side may be required.

Is it backward compatible (if not, which system it affects?)

YES, but when we merge it we may drop existing performance cpu, storage, memory metrics as they will be covered by full set of CMSSW one provided in this PR.

Related PRs

#11696

This PR is superset of #11575 which may be dropped if we'll decide to preserve all CMSSW metrics

External dependencies / deployment changes

cmsdmwmbot · 2023-07-17T17:37:49Z

Jenkins results:

Python3 Unit tests: failed
- 1 new failures
- 1 tests added
- 1 changes in unstable tests
Python3 Pylint check: failed
- 7 warnings and errors that must be fixed
- 3 warnings
- 158 comments to review
Pylint py3k check: failed
- 1 errors and warnings that should be fixed
Pycodestyle check: succeeded
- 36 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/14349/artifact/artifacts/PullRequestReport.html

cmsdmwmbot · 2023-07-17T17:57:33Z

Jenkins results:

Python3 Unit tests: failed
- 2 new failures
- 1 tests added
- 1 changes in unstable tests
Python3 Pylint check: failed
- 7 warnings and errors that must be fixed
- 3 warnings
- 160 comments to review
Pylint py3k check: failed
- 1 errors and warnings that should be fixed
Pycodestyle check: succeeded
- 36 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/14350/artifact/artifacts/PullRequestReport.html

cmsdmwmbot · 2023-07-18T13:39:02Z

Jenkins results:

Python3 Unit tests: failed
- 1 new failures
- 2 tests no longer failing
- 1 tests added
- 1 changes in unstable tests
Python3 Pylint check: failed
- 7 warnings and errors that must be fixed
- 3 warnings
- 160 comments to review
Pylint py3k check: succeeded
Pycodestyle check: succeeded
- 36 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/14351/artifact/artifacts/PullRequestReport.html

amaltaro

@vkuznet Valentin, these changes look good to me.

I am in favor of keeping the metrics summary for the next year or so, until we have data mostly consistent and can implement metric aggregation at a different layer, as you mentioned in the initial PR description.

Given that this PR changes (enforces) a better data schema in terms of data type, we might have to follow up with CMS Monit once it gets deployed to ensure that dashboards remain functional.

vkuznet · 2023-07-19T13:32:15Z

ok, then let's get feedback from @makortel about decision of which set of metrics to keep at the end. If Matti will agree to keep everything (this PR) then we can proceed with this PR and close #11575 one. I also checked unit test and the one which fail WorkQueueTest:testThrottling does not seem to be related to this PR.

amaltaro · 2023-07-19T20:33:50Z

Sounds good! If possible, could you please provide the FJR that would be generated by the agent? Just so we have a realistic example of how it will end up with all those performance sections.

vkuznet · 2023-07-25T17:58:15Z

I finally got FJR report from vocms0291 WMAgent where I run the following workflow

python3 inject-test-wfs.py -u "https://cmsweb-testbed.cern.ch" -m Integration -f TaskChain_MC.json -c DMWM_Test -r Agent0291_forceComplete -t testbed-vocms0291 -a Integ_Test -p forceComplete_VK_patch11663

It was submitted to cmsweb-testbed, where I obtained the report from one of sub jobs. The Report.0.pkl has been decoded in python and correctly shows cmssw section in FrameworkJobReport.cmsRun1.performance object.

@amaltaro , please review provided report and let me know if we need anything else.

vkuznet · 2023-07-26T18:49:40Z

The appropriate part for WMArchive integration can be found in this PR: dmwm/WMArchive#361

amaltaro

Other than the requested unit tests, these changes look good to me.

Based on the CMSW FJR and the Report file, I do want to point out the following:

The "performance.cmssw" contain all of the PerformanceSummary raw metrics provided by CMSSW
The "performance.cpu" section has the same information as "performance.cmssw.Timing", with the only difference that the latter has the new value type casting feature.
The "performance.memory" is a subset of the "performance.cmssw.ApplicationMemory", where the latter has value type casting.
The "performance.summaries" is an exact copy of "performance.cmssw.SystemCPU".
The "performance.storage" section does seem to be unique, because in the source code it is extracted from "Timing-*-numOperations" and some extra manipulation is performed.

In short, once we convert all of the potential monitoring tools to start consuming "performance.cmssw" metrics, we can then completely remove the following handlers/sections:

cpu, memory, summaries

amaltaro · 2023-08-01T10:56:09Z

src/python/WMCore/FwkJobReport/XMLParser.py

@@ -322,7 +328,6 @@ def perfRepHandler(targets):
                targets['PerformanceSummary'].send((perfRep.summaries,
                                                    subnode))

-
 @coroutine
 def perfSummaryHandler():


Can you please make a specific unit test for this function

makortel · 2023-08-01T21:22:42Z

ok, then let's get feedback from @makortel about decision of which set of metrics to keep at the end. If Matti will agree to keep everything (this PR) then we can proceed with this PR and close #11575 one.

I'm fine with keeping all the raw metrics (as I noted in #11538 (comment)).

Note my reply in #11575 (comment) that the xrootd per-site statistics are probably included currently in a sub-optimal way. Would it be possible to see an example job report JSON that gets passed forward to WMArchive?

amaltaro · 2023-08-02T02:26:35Z

@makortel Matti, Valentin didn't provide a WMArchive like document, but it will be completely based on this WMAgent job report that Valentin shared in his previous comment and very similar in terms of organization and metrics:

It was submitted to cmsweb-testbed, where I obtained the report from one of sub jobs. The Report.0.pkl has ...

If this is not enough, then I am afraid we need to wait for Valentin to get back from Vacation (around Aug 11).

makortel · 2023-08-02T14:30:02Z

Matti, Valentin didn't provide a WMArchive like document, but it will be completely based on this WMAgent job report that Valentin shared in his previous comment and very similar in terms of organization and metrics:

It was submitted to cmsweb-testbed, where I obtained the report from one of sub jobs. The Report.0.pkl has ...

If this is not enough, then I am afraid we need to wait for Valentin to get back from Vacation (around Aug 11).

Thanks, that format works too. Apparently that example doesn't contain the xrootd statistics. For the next round of example would it be possible to see both the XML and the JSON or WMAgent job report?

cmsdmwmbot · 2023-08-14T18:22:23Z

Jenkins results:

Python3 Unit tests: failed
- 2 new failures
- 2 tests added
- 2 changes in unstable tests
Python3 Pylint check: failed
- 8 warnings and errors that must be fixed
- 3 warnings
- 161 comments to review
Pylint py3k check: succeeded
Pycodestyle check: succeeded
- 40 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/14408/artifact/artifacts/PullRequestReport.html

vkuznet · 2023-08-14T18:38:13Z

I doubt reported failure are related to unit test I added since they are for Rucio and WQ. Therefore, I request a review again.

vkuznet · 2023-08-14T18:42:16Z

@makortel , to avoid arguing which XML file to use and which sections should be present please provide the one you would like to see and I can generate its WMCore JSON representation.

amaltaro · 2023-08-15T11:12:44Z

@vkuznet Valentin, to avoid confusion, can you please provide the following (either at the initial PR description or in a single comment):

CMSSW FJR (xml document)
WMAgent FJR (as far as I can see, it can be either a xml document or the dot-notation document)
JSON document passed over to WMArchive

vkuznet · 2023-08-15T13:33:27Z

@amaltaro , @makortel please provide proper CMSSW FJR (xml document) document to start with, and then I can provide pkl, WMCore FJR configuration format, and WMArchive JSON.

makortel · 2023-08-15T13:45:54Z

please provide proper CMSSW FJR (xml document) document to start with

I'll look into this some time soonish.

makortel · 2023-08-15T15:34:11Z

Attached is a framework job report XML produced with CMSSW_13_2_0_pre3 that should contain performance information from all components in CMSSW that currently produce such, and has information from several xrootd servers.
fwjr_performance.xml.txt
(edit: updated the file to list less accessed branches to make the file smaller, the exact set of branches is not relevant for this work)

vkuznet · 2023-08-15T18:18:11Z

Here are the links to WMAgent FJR in dot notation and JSON formats based on provided XML

All of them were produced using the following code (feel free to use this snippet with this PR to parse any other CMSSW XML file if you like):

#!/usr/bin/env python

# system modules
import json

# WMCore modules
from WMCore.FwkJobReport.Report import Report
from WMCore.FwkJobReport.XMLParser import xmlToJobReport

xmlfile = '/Users/vk/Downloads/fwjr_performance.xml'
report = Report('cmsRun1')
xmlToJobReport(report, xmlfile)
with open('report.txt', 'w') as ostream:
    ostream.write(report.__str__())
thunker = None
rdict = report.__to_json__(thunker)
with open('report.json', 'w') as ostream:
    ostream.write(json.dumps(rdict))

amaltaro

Valentin, I left one minor comment for your consideration.

In addition to that, we still need to implement the conversion from FJR to WMArchive document. Let us try to first complete the development and merge this one: #11692
then you can update this PR with the relevant changes.

src/python/WMCore/FwkJobReport/Report.py

cmsdmwmbot · 2023-08-23T15:51:37Z

Jenkins results:

Python3 Unit tests: succeeded
- 3 tests added
- 1 changes in unstable tests
Python3 Pylint check: failed
- 8 warnings and errors that must be fixed
- 3 warnings
- 161 comments to review
Pylint py3k check: succeeded
Pycodestyle check: succeeded
- 41 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/14438/artifact/artifacts/PullRequestReport.html

vkuznet · 2023-09-27T14:33:51Z

This PR is read for review and should be viewed together with #11696

todor-ivanov

this one looks good to me

amaltaro · 2023-09-27T19:46:07Z

@vkuznet Valentin, before I review it, can you please rebase it?

vkuznet · 2023-09-27T20:00:00Z

ok, now it is rebased.

cmsdmwmbot · 2023-09-27T20:09:56Z

Jenkins results:

Python3 Unit tests: succeeded
- 3 tests added
- 1 changes in unstable tests
Python3 Pylint check: failed
- 8 warnings and errors that must be fixed
- 3 warnings
- 162 comments to review
Pylint py3k check: succeeded
Pycodestyle check: succeeded
- 42 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/14527/artifact/artifacts/PullRequestReport.html

amaltaro

@vkuznet Valentin, these changes are looking good to me. Nonetheless, I left a couple of comments along the code.

I need to emphasize though that with this PR, you are casting every single performance metric in addition to adding the new cmssw metrics. Please make it extra clear in the pull request description.

src/python/WMCore/FwkJobReport/XMLParser.py

amaltaro · 2023-09-27T20:41:11Z

Please also document in the pull request description on our decision for the XrdSite metrics, such that it is easy to find it.

cmsdmwmbot · 2023-09-28T13:00:29Z

Jenkins results:

Python3 Unit tests: succeeded
- 4 tests added
- 1 changes in unstable tests
Python3 Pylint check: failed
- 11 warnings and errors that must be fixed
- 3 warnings
- 162 comments to review
Pylint py3k check: succeeded
Pycodestyle check: succeeded
- 44 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/14530/artifact/artifacts/PullRequestReport.html

vkuznet · 2023-09-28T13:12:45Z

Description is updated. Please review again.

amaltaro · 2023-09-28T21:47:57Z

@vkuznet can you please squash these commits accordingly?

cmsdmwmbot · 2023-09-29T11:37:23Z

Jenkins results:

Python3 Unit tests: succeeded
- 2 tests no longer failing
- 4 tests added
- 1 changes in unstable tests
Python3 Pylint check: failed
- 11 warnings and errors that must be fixed
- 3 warnings
- 162 comments to review
Pylint py3k check: succeeded
Pycodestyle check: succeeded
- 44 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/14541/artifact/artifacts/PullRequestReport.html

vkuznet · 2023-09-29T11:56:47Z

Alan, all commits are squashed now. Please proceed with review and merge.

amaltaro

Thanks Valentin!

vkuznet added PR: Do not merge yet PR: Work in progress labels Jul 17, 2023

vkuznet self-assigned this Jul 17, 2023

vkuznet requested a review from amaltaro July 17, 2023 17:26

vkuznet force-pushed the fix-issue-11538-v2 branch from 3b1b0a2 to 0611b4d Compare July 17, 2023 17:50

vkuznet force-pushed the fix-issue-11538-v2 branch from 0611b4d to f912f89 Compare July 18, 2023 13:27

amaltaro reviewed Jul 19, 2023

View reviewed changes

vkuznet mentioned this pull request Jul 19, 2023

Provide in the FWJR the CPU and wallclock time for CMSSW subprocesses #11665

Merged

amaltaro reviewed Aug 1, 2023

View reviewed changes

amaltaro mentioned this pull request Aug 1, 2023

Implement IO and Site metrics for FWJR #11575

Closed

vkuznet requested a review from amaltaro August 14, 2023 18:38

makortel mentioned this pull request Aug 15, 2023

Communicate with WM about CMSSW I/O metric reporting cms-sw/framework-team#527

Closed

vkuznet mentioned this pull request Aug 22, 2023

Add WMCMSSWSubprocess and WMTiming metrics to WMArchvie document #11692

Merged

amaltaro requested changes Aug 23, 2023

View reviewed changes

src/python/WMCore/FwkJobReport/Report.py Outdated Show resolved Hide resolved

vkuznet force-pushed the fix-issue-11538-v2 branch from 4586ec5 to d47c36b Compare August 23, 2023 15:42

vkuznet mentioned this pull request Aug 28, 2023

Add CMSSW performance metrics to WMArchive document #11696

Merged

vkuznet removed PR: Do not merge yet PR: Work in progress labels Sep 27, 2023

vkuznet requested review from todor-ivanov and amaltaro September 27, 2023 14:34

todor-ivanov approved these changes Sep 27, 2023

View reviewed changes

vkuznet added a commit to vkuznet/WMCore that referenced this pull request Sep 27, 2023

revert cmssw since it is covered in dmwm#11663

ea3dbf2

vkuznet force-pushed the fix-issue-11538-v2 branch from d47c36b to ca515ab Compare September 27, 2023 19:59

amaltaro requested changes Sep 27, 2023

View reviewed changes

src/python/WMCore/FwkJobReport/XMLParser.py Show resolved Hide resolved

src/python/WMCore/FwkJobReport/XMLParser.py Outdated Show resolved Hide resolved

vkuznet requested a review from amaltaro September 28, 2023 13:12

vkuznet added 2 commits September 29, 2023 07:16

Add CMSSW metrics to FJR

6f485cd

Unit tests for CMSSW metrics

8bf67bc

vkuznet force-pushed the fix-issue-11538-v2 branch from 36ee6d2 to 8bf67bc Compare September 29, 2023 11:16

vkuznet mentioned this pull request Sep 29, 2023

Review and improve CMSSW I/O metric reporting #11538

Closed

amaltaro approved these changes Sep 29, 2023

View reviewed changes

amaltaro merged commit 2a5117f into dmwm:master Sep 29, 2023
3 of 4 checks passed

CMSSW metrics for FWJR #11663

CMSSW metrics for FWJR #11663

Conversation

vkuznet commented Jul 17, 2023 • edited Loading

Status

Description

Is it backward compatible (if not, which system it affects?)

Related PRs

External dependencies / deployment changes

cmsdmwmbot commented Jul 17, 2023

cmsdmwmbot commented Jul 17, 2023

cmsdmwmbot commented Jul 18, 2023

amaltaro left a comment

Choose a reason for hiding this comment

vkuznet commented Jul 19, 2023

amaltaro commented Jul 19, 2023

vkuznet commented Jul 25, 2023

vkuznet commented Jul 26, 2023

amaltaro left a comment

Choose a reason for hiding this comment

amaltaro Aug 1, 2023

Choose a reason for hiding this comment

makortel commented Aug 1, 2023

amaltaro commented Aug 2, 2023

makortel commented Aug 2, 2023

cmsdmwmbot commented Aug 14, 2023

vkuznet commented Aug 14, 2023

vkuznet commented Aug 14, 2023

amaltaro commented Aug 15, 2023

vkuznet commented Aug 15, 2023

makortel commented Aug 15, 2023

makortel commented Aug 15, 2023 • edited Loading

vkuznet commented Aug 15, 2023

amaltaro left a comment

Choose a reason for hiding this comment

cmsdmwmbot commented Aug 23, 2023

vkuznet commented Sep 27, 2023

todor-ivanov left a comment

Choose a reason for hiding this comment

amaltaro commented Sep 27, 2023

vkuznet commented Sep 27, 2023

cmsdmwmbot commented Sep 27, 2023

amaltaro left a comment

Choose a reason for hiding this comment

amaltaro commented Sep 27, 2023

cmsdmwmbot commented Sep 28, 2023

vkuznet commented Sep 28, 2023

amaltaro commented Sep 28, 2023

cmsdmwmbot commented Sep 29, 2023

vkuznet commented Sep 29, 2023

amaltaro left a comment

Choose a reason for hiding this comment

vkuznet commented Jul 17, 2023 •

edited

Loading

makortel commented Aug 15, 2023 •

edited

Loading