ArchiverDataPoller failing to parse documents

[amaltaro]

Impact of the bug
WMAgent

Describe the bug
It's still unclear to me whether it's a bug or something new coming from a CMSSW release(?) Looking into the ArchiveDataReporter component in the FNAL submit6 agent, I see this error/traceback over and over [1]

How to reproduce it
No clue

Expected behavior
Investigate if there is anything wrong with those FJR and either add a protection in WMAgent, or actually fix whatever problem that needs to be fixed with ArchiveDataReporter and how those FJR documents are converted to WMArchive ones.

Additional context and error message
[1]

2022-05-27 14:28:43,750:139988735846144:INFO:ArchiveDataPoller:Found 1000 not archived documents from FWRJ db to upload to WMArchive.
2022-05-27 14:28:44,315:139988735846144:ERROR:ArchiveDataPoller:Error occurred, will retry later:
2022-05-27 14:28:44,315:139988735846144:ERROR:ArchiveDataPoller:more than one input value ['source', 'ce']
2022-05-27 14:28:44,315:139988735846144:ERROR:ArchiveDataPoller:Trace back: 
Traceback (most recent call last):
  File "/data/srv/wmagent/v2.0.2.patch1/sw/slc7_amd64_gcc630/cms/wmagentpy3/2.0.2.patch1/lib/python3.8/site-packages/WMComponent/ArchiveDataReporter/ArchiveDataPoller.py", line 53, in algorithm
    doc = createArchiverDoc(job)
  File "/data/srv/wmagent/v2.0.2.patch1/sw/slc7_amd64_gcc630/cms/wmagentpy3/2.0.2.patch1/lib/python3.8/site-packages/WMCore/Services/WMArchive/DataMap.py", line 394, in createArchiverDoc
    newfwjr = convertToArchiverFormat(fwjr)
  File "/data/srv/wmagent/v2.0.2.patch1/sw/slc7_amd64_gcc630/cms/wmagentpy3/2.0.2.patch1/lib/python3.8/site-packages/WMCore/Services/WMArchive/DataMap.py", line 320, in convertToArchiverFormat
    newFWJR["steps"] = convertSteps(newFWJR["steps"])
  File "/data/srv/wmagent/v2.0.2.patch1/sw/slc7_amd64_gcc630/cms/wmagentpy3/2.0.2.patch1/lib/python3.8/site-packages/WMCore/Services/WMArchive/DataMap.py", line 309, in convertSteps
    stepItem.update(convertStepValue(value))
  File "/data/srv/wmagent/v2.0.2.patch1/sw/slc7_amd64_gcc630/cms/wmagentpy3/2.0.2.patch1/lib/python3.8/site-packages/WMCore/Services/WMArchive/DataMap.py", line 276, in convertStepValue
    raise Exception("more than one input value %s" % list(stepValue['input']))
Exception: more than one input value ['source', 'ce']

[amaltaro]

@todor-ivanov I think we discussed about this issue last week and we agreed that you would be looking into it.
I see however that it's still unassigned and that you have quite a few items already on your plate, so I'd suggest to hand it over to Kenyi (@khurtado), especially because it's now affecting vocms0283 as well (if not all the agents) and we should target having a bug fix for this by the end of the week (to go in the new agents).

[khurtado]

Update: This is a problem when parsing some XML files and it seems to be a bug while creating the XML file from CMSSW itself.

So we end up with one section without a module label (which should be source)

input': {'': [{'module_label':'',....], 'source':['module_label':'source', ]

and we get an exception because we are only expecting `source` at that point.

Double checking regarding a possible core bug with the XML file in:
cms-sw/cmssw#38554 (comment)

[khurtado]

@amaltaro So, it was confirmed the CMSSW release used when we have this problem: 12_2_3_patch1 does not have the bugfix: cms-sw/cmssw#38019

that they incorporated starting CMSSW_12_2_4_patch1.

In summary, we are not dealing with a bug in WMCore and there is a fix in CMSSW in later 12.x releases.

So, how should we proceed? Do we request this PR to be added to 12_2_3_patch1? Or do we just acknowledge and ignore the issue?

[amaltaro]

Fixed by #11198