why is transferor still considering workflows

[sharad1126]

Impact of the bug
transferor is considering workflows in considered

Describe the bug
After the mitigation of input data placement to WMCore MS #519 , unified should just directly assign workflows. Why is transferor still considering workflows?

How to reproduce it
check these two logs in transferor:
/eos/cms/store/unified/www/logs/transferor/
2020-04-17_15:10:38.log
2020-04-17_16:10:29.log

Expected behavior
transferor should not be checking the datasets and blocks for secondaries or inputs any more. This could help to reduce stress on dbs and phedex as well.

Additional context and error message
@amaltaro @todor-ivanov @vlimant @z4027163

[sharad1126]

I saw two workflows now and both of them are stepchain conversion ones. @amaltaro do you think we missed out something for the stepchain conversion regarding the input data placement to change status?

https://cms-unified.web.cern.ch/cms-unified/showlog/?search=critical&module=transferor&limit=50&size=1000

task_HIN-HINPbPbAutumn18pLHE-00009
task_HIN-HINPbPbAutumn18pLHE-00013

[amaltaro]

Linking this log as well:
https://cms-unified.web.cern.ch/cms-unified/showlog/?search=task_JME-RunIISummer19UL17DIGI-00010

Sharad, I had a quick look at the taskchain to stepchain conversion and I couldn't find to which Unified state those get set to.

[sharad1126]

@amaltaro my understanding is that when the workflow conversion happens to step-chain, the workflow is set to status considered instead of staged and that's when transferor picks it up instead of assignor picking it up.

https://github.com/CMSCompOps/WmAgentScripts/blob/master/Unified/injector.py#L115

[sharad1126]

example:
https://dmytro.web.cern.ch/dmytro/cmsprodmon/workflows.php?prep_id=task_HIN-HINPbPbAutumn18pLHE-00013

[amaltaro]

It must be it, but I failed to find where that status get set for StepChains...

[sharad1126]

I think we can just comment out the transferor module as injector and assignor will automatically work on these unless there is an error.

[sharad1126]

@amaltaro another log 2020-04-21_23:10:42.log shows it considered workflows in transferor modules and checked all the datasets for:

        cmsunified_task_HIG-RunIIFall18GS-00062__v1_T_200421_225933_12
        cmsunified_task_HIG-RunIIFall18GS-00063__v1_T_200421_225906_2379
        cmsunified_task_HIG-RunIIFall18GS-00059__v1_T_200421_225959_9545
        cmsunified_task_HIG-RunIIFall18GS-00046__v1_T_200421_225811_9662
        cmsunified_task_HIG-RunIIFall18GS-00061__v1_T_200421_225839_2014

All are stepchain converted workflows.

[z4027163]

@amaltaro @sharad1126 From my understanding, the step chain conversion happens in "injector" [1], where its status is set to "trouble" because of --clone option. Then its status is changed to "considered" at [2]. Those happen after the "staged" set up in injector (If I understand correctly, we are supposed to set wf status to staged in injector). So changing [2] to "staged" could solve the problem?

[1]

WmAgentScripts/Unified/injector.py

Line 130 in 3b8d1ec

os.system('./Unified/rejector.py --clone --to_step --comments \"Transform to StepChain\" %s'% wf)

[2]

WmAgentScripts/Unified/injector.py

Line 192 in 3b8d1ec

status = 'considered'

[sharad1126]

@amaltaro @z4027163 I think your comments makes it more clear. I will make a PR now.