deleting jobsummary from wmagent summary when request is announced #4227
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
sfoulkes
pushed a commit
that referenced
this pull request
Nov 13, 2012
deleting jobsummary from wmagent summary when request is announced
|
dballesteros7
added a commit
to dballesteros7/WMCore
that referenced
this pull request
Nov 13, 2012
The Tier0 plugin for the AnalyticsDataCollector components calculates the state of Tier-0 workflows based on subscription information found in WMBS and updates the request document in central WMStats with this status. Additionally this patch fixes some bugs with PluginInterface implementation patch and renames it from PlugIn to Plugin to conform with the same name as the BossAir package (PlugIn is a bit harder to remember). Fix the AnalyticsDataCollector unittest and removed it from the allowed failing tests. This requires dmwm#4227 to work, it also fixes the unit test breaks from it. Fixes dmwm#4264
Merged
ticoann
added a commit
to ticoann/WMCore
that referenced
this pull request
Feb 1, 2013
commit e3cae2bdf053530f84e3df73f822bc7c7405a9be
Author: Seangchan Ryu <sryu@fnal.gov>
Date: Wed Jan 30 15:26:38 2013 -0600
check components are alive using pid and also thread status
commit e6fac9d603eaa882c8830b28a7913b37d56269b3
Merge: 5d42249 5fc3e3d
Author: Stephen Foulkes <sfoulkes@fnal.gov>
Date: Wed Jan 30 11:42:14 2013 -0800
Merge pull request #4379 from zdenekmaxa/reqmgr-custodialsites-bugfix
ReqMgr CustodialSite bug fix.
commit 5fc3e3d6d2fbb61f9355dad356d9763644dcdfd2
Author: Zdenek Maxa <zdenek.maxa@hep.caltech.edu>
Date: Wed Jan 30 16:30:02 2013 +0100
ReqMgr CustodialSite bug fix.
CustodialSite is an argument not used for long. (Non)CustodialSites have
superseded this, however no one bothered to remove all dead functionality and
parameter definition. On cloned requests, the default value (None) of
CustodialSite request parameter appared and PhEDExInjector was failing on
that. This patch removes CustodialSite from ReqMgr only.
commit 5d4224997378fa1778e79d59acd7dc7f97594b93
Author: Stephen Foulkes <sfoulkes@fnal.gov>
Date: Tue Jan 29 16:23:38 2013 -0600
0.9.39
Signed-off-by: Stephen Foulkes <sfoulkes@fnal.gov>
commit be297dcd343990d82b1b1051af0b099090b06128
Merge: faf418a aeb4645
Author: Stephen Foulkes <sfoulkes@fnal.gov>
Date: Tue Jan 29 14:20:33 2013 -0800
Merge pull request #4371 from hufnagel/runcompletedqm
fix cmsrun config for endofrun dqm harvesting
commit faf418a34c3fa032eec3cc4c1c1df6f01a0351a4
Merge: 10dbe1c eb9c657
Author: Stephen Foulkes <sfoulkes@fnal.gov>
Date: Tue Jan 29 14:20:04 2013 -0800
Merge pull request #4377 from zdenekmaxa/reqmgr-clone-prioritybugfix2
ReqMgr cloned requests priority fix
commit eb9c65720253f722c3872427ad994aca9e07df89
Author: Zdenek Maxa <zdenek.maxa@hep.caltech.edu>
Date: Mon Jan 28 17:59:41 2013 +0100
ReqMgr cloned requests priority fix
When cloning a request which had changed priority, the newly cloned request
priority is on default values. Fixes goes via MakeRequest.createRequest() to
database DAOs Request/New.py
Also reqmgr.py checks the case.
Modified default value SoftTimeout on Ops request.
commit 10dbe1ca03c8a31c2c64642c082363c83875e407
Merge: d7a3d13 694b379
Author: Stephen Foulkes <sfoulkes@fnal.gov>
Date: Tue Jan 29 08:20:07 2013 -0800
Merge pull request #4374 from ticoann/tier0_state_warning
add status duration
commit aeb4645a5395a3e252247e2009192f7815c61197
Author: Dirk Hufnagel <Dirk.Hufnagel@cern.ch>
Date: Mon Jan 28 16:44:44 2013 +0100
fix cmsrun config for endofrun dqm harvesting
EndOfRun DQM harvest jobs need a special cmsRun configuration.
commit 694b3798b0a894c6c7172e2381f6a55faf069e59
Author: Seangchan Ryu <sryu@fnal.gov>
Date: Mon Jan 28 16:03:11 2013 -0600
add status duration
commit d7a3d138ba63507658109b582770de2c93876ff9
Author: Stephen Foulkes <sfoulkes@fnal.gov>
Date: Fri Jan 25 15:34:15 2013 -0600
0.9.38
Signed-off-by: Stephen Foulkes <sfoulkes@fnal.gov>
commit e186453679fbdd4a611ab6eb1b3b817739821291
Merge: 6fcb77f a61239f
Author: Stephen Foulkes <sfoulkes@fnal.gov>
Date: Fri Jan 25 13:30:28 2013 -0800
Merge pull request #4369 from dballesteros7/fix-WorkQueue-lumicount
Fix Block policy accounting of lumis
commit 6fcb77f2474b768a7ca6112e1f880cd67f9bf816
Merge: cfed687 9c46dd9
Author: Stephen Foulkes <sfoulkes@fnal.gov>
Date: Fri Jan 25 13:29:52 2013 -0800
Merge pull request #4370 from ticoann/add_queue_column
add queue column in the job number table view
commit 9c46dd9331509d3dcce572d2f67a2807b6dcd875
Author: Seangchan Ryu <sryu@fnal.gov>
Date: Fri Jan 25 15:09:24 2013 -0600
add queue column
commit cfed687fbc58c4c478a134b1ad2a3721541756ee
Author: Stephen Foulkes <sfoulkes@fnal.gov>
Date: Fri Jan 25 15:06:21 2013 -0600
0.9.37
Signed-off-by: Stephen Foulkes <sfoulkes@fnal.gov>
commit e090caf716f71ecc2853c0221dff0f12692b095d
Merge: a852056 354ae40
Author: Stephen Foulkes <sfoulkes@fnal.gov>
Date: Fri Jan 25 12:49:02 2013 -0800
Merge pull request #4359 from ticoann/remove_t0wmstats_dependency
remove t0wmstats dependencies - just use reqmon to create
commit a852056c49ca61808ffbf8f71e79af97e22d08fa
Merge: a58ac10 c52a14c
Author: Stephen Foulkes <sfoulkes@fnal.gov>
Date: Fri Jan 25 12:48:47 2013 -0800
Merge pull request #4360 from dballesteros7/fix-tier0-plugin-races
Fix another possible glitch in the Tier-0 plugin
commit a58ac10ab120b0d480c9024d187f937777b34d0c
Merge: 39db917 feb0723
Author: Stephen Foulkes <sfoulkes@fnal.gov>
Date: Fri Jan 25 12:47:52 2013 -0800
Merge pull request #4361 from dballesteros7/Fix-EventBased-Tails
Fix EventBased algorithm dropping zero event lumis
commit 39db917a0a128a3c6b32972a567131dd569521df
Merge: 2110ca5 928dd90
Author: Stephen Foulkes <sfoulkes@fnal.gov>
Date: Fri Jan 25 12:47:30 2013 -0800
Merge pull request #4362 from ticoann/replication_retry
retry replication when it fails
commit 2110ca5d9a2ca1afe58e60be30c8135ead5547f2
Merge: 8f36704 7622553
Author: Stephen Foulkes <sfoulkes@fnal.gov>
Date: Fri Jan 25 12:47:09 2013 -0800
Merge pull request #4365 from ticoann/t0_operator_permission
add t0-operator permission for wmstats
commit 8f36704550f719486c46cb04ea6c81a99de1d871
Merge: 8cf925b 606862c
Author: Stephen Foulkes <sfoulkes@fnal.gov>
Date: Fri Jan 25 12:46:20 2013 -0800
Merge pull request #4367 from ticoann/prep_id_fix
fix prep id search and make detail page not draggable
commit 8cf925ba2bc02ae7693b0530e6006ab5dbe84af0
Merge: 4262559 75e9737
Author: Stephen Foulkes <sfoulkes@fnal.gov>
Date: Fri Jan 25 12:43:58 2013 -0800
Merge pull request #4368 from sfoulkes/run_whitelist
Make sure the input run white list is passed to the job splitting
commit a61239ffe1ed859d39b485c96df110ecc8d09f44
Author: Diego Ballesteros <dballest@fnal.gov>
Date: Thu Jan 24 16:34:46 2013 -0600
Fix Block policy accounting of lumis
When a run white/blacklist is present, the accounting of lumis is screwed up
and it only reports a number of lumis equal to the number of runs accepted in a block.
Unfortunately, DBS doesn't return block level information on runs and lumis
when calling the listRuns function. So the solution is the following:
- Check first with listRuns and apply run black and whitelist.
- If there is a single run in the block, and the block passed then no need to recalculate lumis.
- However if the block has more than one run, and the list of blocks in the run was affected
by the run white/blacklist then recalculate the event/file/lumi counts by pulling full file
information from DBS. This is expensive but we expect the case to be rarely present.
commit 75e973759951ac44970cf967583445b72561fcbb
Author: Stephen Foulkes <sfoulkes@fnal.gov>
Date: Fri Jan 25 14:40:12 2013 -0600
Improve the code that passes run white and black list to the job splitting
parameters so that it doesn't break other things.
commit 4262559d92516a2a25794d0736246703e223eacf
Merge: f2080b3 99b0f34
Author: Stephen Foulkes <sfoulkes@fnal.gov>
Date: Fri Jan 25 12:29:03 2013 -0800
Merge pull request #4358 from sfoulkes/parallel-dbs3
Fix up the DBS3Uploader and modify it to support an upload only mode. K...
commit 99b0f34ee6d5e9e2a29d4ad5659cd32cb2d096e7
Author: Stephen Foulkes <sfoulkes@fnal.gov>
Date: Fri Jan 25 14:27:58 2013 -0600
Fix issues with Oracle and DBS3Upload.
commit 91683197d3925dc4745cb3107214ec5be3e94847
Author: Stephen Foulkes <sfoulkes@fnal.gov>
Date: Fri Jan 25 10:12:05 2013 -0600
Make sure the input run white list is passed to the job splitting
algorithm.
commit 606862c5155deeee491d84535aacdbe095d54219
Author: Seangchan Ryu <sryu@fnal.gov>
Date: Thu Jan 24 15:54:12 2013 -0600
fix prep id search and make detail page not draggable
commit 76225534924e3cc2aa9814330305f2f9c3802b6a
Author: Seangchan Ryu <sryu@fnal.gov>
Date: Thu Jan 24 13:22:26 2013 -0600
add t0-operator permission
commit 928dd90313d9da924b17f906ad1b439305d20b76
Author: Seangchan Ryu <sryu@fnal.gov>
Date: Thu Jan 24 11:20:35 2013 -0600
retry replication when it fails
commit feb0723145854127e2e214a5df1022e8448ac75d
Author: Diego Ballesteros <dballest@fnal.gov>
Date: Wed Jan 23 23:55:02 2013 -0600
Fix EventBased algorithm dropping zero event lumis
The current mask definition performed by the EventBased splitting
algorithm on non-mc input files can lead to missing lumis in the output.
This occurs when an input file has zero events in the last lumis, because
the cmsRun process will stop once it has reached the max events and won't process
those eventless lumis. To fix this, avoid passing the maxEvents parameter
when configuring the mask of the last job in a file, the skipEvents parameter
will suffice and the job will run over the remainder of the file.
Many assertions changed in the unit tests, since we change from having
always a max events to having it only on intermediate jobs in a file.
commit 354ae40a1179f8bc02af0a0a6a15a3717c1b1f14
Author: Seangchan Ryu <sryu@fnal.gov>
Date: Wed Jan 23 13:45:32 2013 -0600
remove t0wmstats dependencies - just use reqmon to create
commit e6988496fd2be7410c32eaa8a14a22055e1686b7
Author: Stephen Foulkes <sfoulkes@fnal.gov>
Date: Wed Jan 23 09:22:35 2013 -0600
Fix up the DBS3Uploader and modify it to support an upload only mode. Kill
off SQLite in DBS3Buffer and DBSBuffer.
commit f2080b3ca2b54c0af756462c113f693b2c0e15bb
Author: Stephen Foulkes <sfoulkes@fnal.gov>
Date: Wed Jan 23 09:09:27 2013 -0600
0.9.36
Signed-off-by: Stephen Foulkes <sfoulkes@fnal.gov>
commit aab5e088feebc1432c97739b6a5fef17a38ad116
Merge: bb56bf8 083857c
Author: Zdenek Maxa <zdenek.maxa@hep.caltech.edu>
Date: Wed Jan 23 06:02:18 2013 -0800
Merge pull request #4357 from zdenekmaxa/reqmgr-feb-opsadditions
Reqmgr feb opsadditions
commit 083857ccc0e43773fa7fe18e5127f5a4031f8753
Author: Zdenek Maxa <zdenek.maxa@hep.caltech.edu>
Date: Wed Jan 23 14:47:41 2013 +0100
ReqMgr - PhEDEx subscriptions,cloned request state
- PhEDEx subscriptions arguments added into JSON request templates
- cloned requests will not have new state but assignment-approved
commit bb56bf8c9953689dfcd878203ec7802dc9dd9207
Merge: 2be5654 95da3bd
Author: Zdenek Maxa <zdenek.maxa@hep.caltech.edu>
Date: Wed Jan 23 05:43:10 2013 -0800
Merge pull request #4355 from dballesteros7/flexible-ReReco
Allow keeping ReReco outputs unmerged
commit 95da3bd42a6e8b11567ecf39a6220b6781fa003b
Author: Diego Ballesteros <dballest@fnal.gov>
Date: Fri Jan 18 17:07:35 2013 -0600
Allow keeping ReReco outputs unmerged
Add the option to the ReReco spec to keep some output module unmerged
but still run skims on it if necessary, if no skim is defined for it
the output will be produced, stored in unmerged area and cleaned up
afterwards.
commit 2be5654eac4ba621245514ea959fc460507bb5cb
Author: Stephen Foulkes <sfoulkes@fnal.gov>
Date: Sun Jan 20 11:50:50 2013 -0600
0.9.35
Signed-off-by: Stephen Foulkes <sfoulkes@fnal.gov>
commit 64cfdc0b534724f13e2d295c0e9c79f3956a9653
Merge: bc67672 d66df52
Author: Stephen Foulkes <sfoulkes@fnal.gov>
Date: Fri Jan 18 14:27:54 2013 -0800
Merge pull request #4354 from dballesteros7/changePhedexSubscriber
Add an option to the Phedex subscriber
commit d66df52dee894e4cc046fff612c2f731eb09652d
Author: Diego Ballesteros <dballest@fnal.gov>
Date: Fri Jan 18 15:39:34 2013 -0600
Add an option to the Phedex subscriber
Add the option to make only replica subscriptions, this
for agents which don't really require move subscriptions
such as those only dealing with T0 or T1s
commit bc67672266a502688bab123fef72812e5b14a10e
Merge: 71f5e6c 3fb456f
Author: Stephen Foulkes <sfoulkes@fnal.gov>
Date: Fri Jan 18 13:37:12 2013 -0800
Merge pull request #4353 from ticoann/running_open_update
fix bug for updating data for running-open and running-closed status
commit 3fb456fdf7e6c87f9a79d0267f958806c4d5aae8
Author: Seangchan Ryu <sryu@fnal.gov>
Date: Fri Jan 18 11:28:06 2013 -0600
fix bug for updating data for running-open and running-closed status
commit 71f5e6cb53c572f132a4cd90856877b5f0d14362
Merge: f1e7c86 7120dec
Author: Stephen Foulkes <sfoulkes@fnal.gov>
Date: Fri Jan 18 07:19:16 2013 -0800
Merge pull request #4352 from ticoann/pull_request_feb2
wmstats updates for feb, 2013 deployment
commit f1e7c86c19c58ee629935eeeb4037515eb71cfc3
Merge: 41ca82a dd94f64
Author: Stephen Foulkes <sfoulkes@fnal.gov>
Date: Fri Jan 18 07:18:59 2013 -0800
Merge pull request #4350 from hufnagel/newdropboxupload
changes to alcaharvest runtime code to support new dropbox
commit 7120decca4277ebb93e5884e2b735dbf1ca3327c
Merge: bf8035e 32bf6c3
Author: Seangchan Ryu <sryu@fnal.gov>
Date: Thu Jan 17 11:41:04 2013 -0600
Merge branch 'report_couch_replication' into pull_request_feb2
commit 32bf6c3ce0e20b610cb56c394115beb57251ff5f
Author: Seangchan Ryu <sryu@fnal.gov>
Date: Wed Jan 16 16:31:55 2013 -0600
add warning for couch server status
commit 6f7cdadd06cdbec290746007757115d70754ce97
Author: Zdenek Maxa <zdenek.maxa@hep.caltech.edu>
Date: Thu Jan 17 09:34:43 2013 +0100
Merge branches 'upmaster' and 'reqmgr2-databasecleanup'
commit bf8035eec188dd081186f980650004564ded362e
Author: Seangchan Ryu <sryu@fnal.gov>
Date: Wed Jan 16 15:19:01 2013 -0600
Squashed commit of the following:
commit 154b26deff714575f1abbdac86d2de67441147cd
Author: Seangchan Ryu <sryu@fnal.gov>
Date: Wed Jan 16 15:16:26 2013 -0600
add job number views for request view
commit b6820727939f58f4c236b54573226b0904da8f47
Author: Seangchan Ryu <sryu@fnal.gov>
Date: Fri Jan 11 16:58:02 2013 -0600
cleanup view related to analysis use case
commit 87ae24252e6d050e3c477a7edd067904529db640
Author: Seangchan Ryu <sryu@fnal.gov>
Date: Fri Jan 11 16:56:54 2013 -0600
search in wmstats instead of workloadsummary
add prepid serarch
fix reduce error on performanceSummaryByTask
commit 7505adea29c5f676cbd44b4d57556fec816c0502
Author: Seangchan Ryu <sryu@fnal.gov>
Date: Fri Jan 11 10:41:16 2013 -0600
view switch
commit 41ca82a853325f94865f8129a89a11c7666db001
Merge: d525feb 14700df
Author: Zdenek Maxa <zdenek.maxa@hep.caltech.edu>
Date: Wed Jan 16 10:54:36 2013 -0800
Merge pull request #4351 from zdenekmaxa/reqmgr2-databasecleanup
ReqMgr2 - Oracle, CouchDB database cleanup
commit 14700df6ed6e1b299712cee53558fdc0cd61031c
Author: Zdenek Maxa <zdenek.maxa@hep.caltech.edu>
Date: Wed Jan 16 19:45:53 2013 +0100
ReqMgr2 - Oracle, CouchDB database cleanup
-utils for Oracle dump, Oracle vs. CouchDB consistency comparison
-CouchDB requests cleanup
-temporary logs and notes, all helpful before dropping Oracle for ReqMgr2
commit dd94f6484dc6c6f261179acaa81ef1af3c47401c
Author: Dirk Hufnagel <Dirk.Hufnagel@cern.ch>
Date: Fri Jan 11 11:04:28 2013 +0100
changes to alcaharvest runtime code to support new dropbox
The new dropbox expects a different metadata format for
uploaded payloads. TO produce that format the format of
the information returned in the analysis files section
in the framework job report has been changed. Make the
needed code changed to deal with that new format.
commit d525febdd8c525e8594512c61d38bd50fc5ae9be
Merge: 2b49c88 aa7a866
Author: Stephen Foulkes <sfoulkes@fnal.gov>
Date: Thu Jan 10 14:03:00 2013 -0800
Merge pull request #4348 from dballesteros7/fixHarvestDuplicate
Fix Harvest splitter creating duplicate jobs
commit 2b49c88fb3b5ac1f7d729894cd63d46f6d1339bd
Merge: 479c1b1 67c268f
Author: Stephen Foulkes <sfoulkes@fnal.gov>
Date: Thu Jan 10 14:02:39 2013 -0800
Merge pull request #4349 from dballesteros7/fixCERNStagein
Fix EOS StageIn at CERN
commit 67c268f7eb1246eed5aa6a735376e69cba39d2dc
Author: Diego Ballesteros <dballest@fnal.gov>
Date: Fri Dec 7 10:56:02 2012 -0600
Fix EOS StageIn at CERN
Plugin was using rfcp to stageIn files from EOS at CERN.
Check properly if remotePFN is EOS, instead of targetPFN
as in stageIn the targetPFN will be the local.
Also fix deletions, in stageIn case it was going to
delete the remotePFN instead of the faulty local
copy. Include deletions for failed CASTOR stage-outs.
Finally, only create directories on remote PFNs.
commit aa7a8669492a4aa5ef1721da8bff27743b46b2c6
Author: Diego Ballesteros <dballest@fnal.gov>
Date: Wed Jan 9 23:25:46 2013 -0600
Fix Harvest splitter creating duplicate jobs
This because of multiple SEs for a single site.
commit 479c1b1c9d1da197bb58cbcce4e9bf6d7903e376
Merge: 1c64de1 bb34e7c
Author: Stephen Foulkes <sfoulkes@fnal.gov>
Date: Wed Jan 9 11:37:38 2013 -0800
Merge pull request #4347 from dballesteros7/fixMergeDuplicate
Fix WMBSMergeBySize causing duplicates
commit bb34e7c8f4641191a048574236c188c233a9fed7
Author: Diego Ballesteros <dballest@fnal.gov>
Date: Tue Jan 8 13:00:58 2013 -0600
Fix WMBSMergeBySize causing duplicates
This occurs when a site has multiple SEs, fix
changes the discovery query to return a file only once
with a tuple of the possible SEs.
commit 1c64de1072504c524c57d5954c445395b8be663f
Author: Stephen Foulkes <sfoulkes@fnal.gov>
Date: Wed Jan 9 09:27:29 2013 -0600
0.9.34
Signed-off-by: Stephen Foulkes <sfoulkes@fnal.gov>
commit abf282481bd03728345066aa85ef5f1a3920941d
Merge: d076564 5cd9564
Author: Stephen Foulkes <sfoulkes@fnal.gov>
Date: Wed Jan 9 07:03:29 2013 -0800
Merge pull request #4339 from PerilousApricot/bugfix/zipimport
Fix sandbox creation
commit d0765642d0921d935d304a05c65754282c2377d7
Merge: a8da4ac 0596ed1
Author: Stephen Foulkes <sfoulkes@fnal.gov>
Date: Wed Jan 9 07:02:51 2013 -0800
Merge pull request #4340 from giffels/restapi-docstrings
Display docstrings in RESTAPI documentation
commit a8da4ac3d680d937b8c46ab2501b5f088edfcc46
Merge: 9f60a15 5d5e845
Author: Stephen Foulkes <sfoulkes@fnal.gov>
Date: Wed Jan 9 07:02:18 2013 -0800
Merge pull request #4342 from dballesteros7/fix-errorSite-hash
Fix issue with site parameter in error summary
commit 9f60a151e19eb29edd9da9adb562605ebc300ad1
Merge: 8f5e565 04abda2
Author: Stephen Foulkes <sfoulkes@fnal.gov>
Date: Wed Jan 9 07:01:47 2013 -0800
Merge pull request #4345 from ticoann/pull_feb_deploy_1
major wmstats change for feb deployment
commit 04abda2e80a0e20184b431fba7f9e4fee8cee17e
Author: Seangchan Ryu <sryu@fnal.gov>
Date: Tue Jan 8 14:46:20 2013 -0600
add running-open and running-closed status and update event, lumi function.
add workload summary rewrite rules.
commit fe4f6830d18440121d6ef98ab6e9165ca7086f72
Author: Seangchan Ryu <sryu@fnal.gov>
Date: Mon Jan 7 17:18:20 2013 -0600
add basic search page
commit a3298d0f2215ba3048f4679765f7e51704efc90d
Author: Seangchan Ryu <sryu@fnal.gov>
Date: Fri Jan 4 16:22:48 2013 -0600
Squashed commit of the following:
commit 37505121c613f5c03821d2814931c0e488e872a5
Merge: d9a63f4 8599684
Author: Seangchan Ryu <sryu@fnal.gov>
Date: Fri Jan 4 15:46:26 2013 -0600
Merge branch 'fix_agent_info_update' into rpm_test
Conflicts:
src/python/WMComponent/AnalyticsDataCollector/AnalyticsPoller.py
commit d9a63f49a03f5a9ab74fff0a475bdf52243e6ab2
Author: Seangchan Ryu <sryu@fnal.gov>
Date: Fri Jan 4 15:43:08 2013 -0600
add couch server alert
commit 85996843562d4a34a183742d9324f876e9dc2bf2
Author: Seangchan Ryu <sryu@fnal.gov>
Date: Fri Jan 4 11:21:32 2013 -0600
directly upload agent info to remote wmstats to prevent conflict when agent redeployed
commit 900174f6fdd8b022d364d01224a184dbcbc22109
Author: Seangchan Ryu <sryu@fnal.gov>
Date: Fri Jan 4 11:17:36 2013 -0600
refactor Controls view and fix UTC
commit 1b328477033283a056c5156a547292dcffbd4a89
Author: Seangchan Ryu <sryu@fnal.gov>
Date: Wed Jan 2 10:43:09 2013 -0600
fix average event and lumi calculation
commit 61a873f549cff765880118ff32fef79523de7c67
Author: Seangchan Ryu <sryu@fnal.gov>
Date: Mon Dec 31 11:54:34 2012 -0600
lumi and event progress.
need to find how to handle the same out put dataset names for different workflow.
commit 31e875136da6038a3d3ec98451ea0d9d15f1bc06
Author: Seangchan Ryu <sryu@fnal.gov>
Date: Fri Dec 28 13:59:07 2012 -0600
lumi based progress bug fix and refactor request struct
commit f292faf502aa888e8471112789425be3684b930a
Author: Seangchan Ryu <sryu@fnal.gov>
Date: Wed Dec 19 11:14:00 2012 -0600
hard code summary link
commit 95162066eb99fb546c527293bc4d7d85ec9b4617
Author: Seangchan Ryu <sryu@fnal.gov>
Date: Tue Dec 18 16:53:21 2012 -0600
add minified version
commit 2a8a021ac8aae219adb4defebee2ccb3433a9705
Author: Seangchan Ryu <sryu@fnal.gov>
Date: Tue Dec 18 16:52:14 2012 -0600
12/18 vocms236 test version
commit 63a96526b30aaabb2c8c63f099157e5476872d83
Author: Seangchan Ryu <sryu@fnal.gov>
Date: Fri Dec 14 17:07:47 2012 -0600
fix site missing error for display this is temporary patch to handle old wmagent interaction
commit 1904760f02e8f44b13664184e122cbacaa8f6a61
Author: Seangchan Ryu <sryu@fnal.gov>
Date: Fri Dec 14 16:20:36 2012 -0600
hard coded value
commit b7beba9d43f6708e6087b6d40ed8ed46a90ef1ab
Merge: efdfcaf 97ecf83
Author: Seangchan Ryu <sryu@fnal.gov>
Date: Fri Dec 14 14:01:46 2012 -0600
Merge branch '2012-12-4-merge' into 2012-12-14-merge
commit 97ecf837a1740637de65ab5ff83f3e57cd89d25f
Author: Seangchan Ryu <sryu@fnal.gov>
Date: Fri Dec 14 13:55:35 2012 -0600
add search page work in progress
commit ffd317715345ae06cf57141c22466769256712eb
Author: Seangchan Ryu <sryu@fnal.gov>
Date: Fri Dec 14 13:51:52 2012 -0600
add total lumi calcuation
commit 1c67f0b23458771de3dc81f0d7ffe183fdc577d7
Author: Seangchan Ryu <sryu@fnal.gov>
Date: Thu Dec 13 15:53:32 2012 -0600
fix typo
commit b0d3b5d534a0d2cf1d83bc0512563b97550defc9
Author: Seangchan Ryu <sryu@fnal.gov>
Date: Thu Dec 13 15:36:37 2012 -0600
enhance the loaing time
commit 03fb87b883ce27ac281435deb640aeb67733535d
Author: Seangchan Ryu <sryu@fnal.gov>
Date: Thu Dec 13 15:35:28 2012 -0600
add file merge status in the job report
commit dae5e248c071dc70c86741e589ff55d4aea335ca
Author: Seangchan Ryu <sryu@fnal.gov>
Date: Fri Dec 7 15:23:18 2012 -0600
remove none used view
commit a49ed0e6d30c8178f511651017f48e524569f2ef
Author: Seangchan Ryu <sryu@fnal.gov>
Date: Tue Dec 4 16:59:01 2012 -0600
Conflicts:
src/python/WMComponent/AnalyticsDataCollector/AnalyticsPoller.py
src/python/WMComponent/AnalyticsDataCollector/DataCollectAPI.py
src/python/WMCore/Services/WMStats/WMStatsWriter.py
commit c52a14c5d96d2dfc8decfb51f0920d730ad64b42
Author: Diego Ballesteros <dballest@fnal.gov>
Date: Wed Jan 2 13:38:39 2013 -0600
Fix another possible glitch in the Tier-0 plugin
normal-archived and aborted-archived are final status and therefore
no transition from them should be admitted.
commit 5d5e845ce30b50bd2d5b6fe4ef9d0d27cf29ec20
Author: Diego Ballesteros <dballest@fnal.gov>
Date: Wed Jan 2 08:18:22 2013 -0600
Fix issue with site parameter in error summary
It can be an empty dictionary which causes problems
when trying to create the summary.
Fix that by ensuring it's a string.
commit 0596ed162954f642a4117aa0118f5825f4ab899c
Author: Manuel Giffels <giffels@gmail.com>
Date: Wed Dec 19 17:16:27 2012 +0100
Display docstrings in RESTAPI documentation
commit 5cd9564b9ba3d6e29a9de81e018bd676436b2099
Author: Andrew Melo <andrew.melo@gmail.com>
Date: Sat Dec 15 17:08:21 2012 +0100
Fix sandbox creation
The old way of finding the base of the WMCore tree didn't work with
patched installs (the sandboxes were getting made from the base RPM, not
the patched location)
commit 8f5e5653ed6199a8e0b203a7d10ba19e479c2d58
Merge: 6022c3a 7a9a74c
Author: Stephen Foulkes <sfoulkes@fnal.gov>
Date: Fri Dec 14 12:26:08 2012 -0800
Merge pull request #4336 from ticoann/add_merged_flag_frwj
add merged flag for framework job report
commit 6022c3a268bd7eae49e3024485568b1e02dbae6d
Merge: be7bd5c 3813be9
Author: Stephen Foulkes <sfoulkes@fnal.gov>
Date: Fri Dec 14 12:25:51 2012 -0800
Merge pull request #4335 from dballesteros7/fixLastFix
Fix cleanup of timestamp cache on the JobSubmitter
commit be7bd5c0a2d5f6d6b503c6f6a0f1c8eac9e26be3
Merge: efdfcaf f3180de
Author: Stephen Foulkes <sfoulkes@fnal.gov>
Date: Fri Dec 14 12:25:29 2012 -0800
Merge pull request #4334 from dballesteros7/fixSmartLumiSplitter
Fix bug in smart lumi splitter
commit 7a9a74ceef7a09f96bf4c407ef0bf5ca9ec0c7be
Author: Seangchan Ryu <sryu@fnal.gov>
Date: Thu Dec 13 15:44:57 2012 -0600
add merged flag for framework job report
commit f3180ded237f2f9dddb1ee910176ec886c9cb765
Author: Diego Ballesteros <dballest@fnal.gov>
Date: Wed Dec 12 10:55:11 2012 -0600
Fix bug in smart lumi splitter
It failed if the conditions aligned and it was trying to process
a too-big single-lumi file.
Add unit-test that fails without the fix
commit efdfcafc39f5df2144f060b18a47e73dd51f575d
Merge: 1529eac 4ba9657
Author: Zdenek Maxa <zdenek.maxa@hep.caltech.edu>
Date: Tue Dec 11 13:03:54 2012 -0800
Merge pull request #4332 from zdenekmaxa/reqmgr-couchapps-reqmgr-cleanup
Cleaning up ReqMgr couchapp
commit 4ba96577766ddcb1a2c708593ef01d269bb2a4cc
Author: Zdenek Maxa <zdenek.maxa@hep.caltech.edu>
Date: Tue Dec 11 14:50:47 2012 -0600
Cleaning up ReqMgr couchapp
(what gets currently installed in reqmgr_workload_cache database)
- index.html not functional, could not work behind cmcsweb frontend
- most but 1 views removed and list function, not referenced from anywhere
- entire evenly library was checked in without any purpose
commit 3813be999bbf5a8fb4a4581403ee6021546e6862
Author: Diego Ballesteros <dballest@fnal.gov>
Date: Tue Dec 11 13:16:10 2012 -0600
Fix cleanup of timestamp cache on the JobSubmitter
Last patch has a bug when cleaning up the memory cache on
workflow timestamps, fix this here.
commit 1529eac85263a55977d8f25b6723d3976c52e86c
Merge: c616173 aa3bfc5
Author: Stephen Foulkes <sfoulkes@fnal.gov>
Date: Fri Dec 7 14:24:50 2012 -0800
Merge pull request #4232 from ticoann/remove_workloadsummary_remgr
remove workloadsummary installation from reqmgr and move to reqmon
commit c61617307c8b2ecf506fc0990d1a03178a794896
Merge: 501aac6 4a74365
Author: Stephen Foulkes <sfoulkes@fnal.gov>
Date: Fri Dec 7 14:24:32 2012 -0800
Merge pull request #4325 from dballesteros7/fixSubscriptionErrors
Avoid crashes on custodial subscription to non-MSS
commit 501aac6d0e04eb6e4d5e36eca7fe803657d40359
Merge: df5d482 f37a613
Author: Stephen Foulkes <sfoulkes@fnal.gov>
Date: Fri Dec 7 14:24:18 2012 -0800
Merge pull request #4326 from ticoann/cleanupcouch_fix
handle when there is no jobs in request
commit aa3bfc59e1e041525ed81f167ed8bb4c90fe2377
Merge: efbcaa7 df5d482
Author: Seangchan Ryu <sryu@fnal.gov>
Date: Fri Dec 7 15:29:39 2012 -0600
Merge branch 'rm_workladsummary' into remove_workloadsummary_remgr
Conflicts:
setup_dependencies.py
commit df5d4822703e9fcf3624939c926f931ceee0b794
Merge: 5b46f06 1e12bfa
Author: Stephen Foulkes <sfoulkes@fnal.gov>
Date: Thu Dec 6 13:22:47 2012 -0800
Merge pull request #4327 from dballesteros7/ensureJobOrderByTime
Ensure jobs are submitted, oldest workflow first
commit 5b46f06ddf87efc930605d0921af62e342403177
Merge: e030e4a b183120
Author: Stephen Foulkes <sfoulkes@fnal.gov>
Date: Thu Dec 6 13:22:22 2012 -0800
Merge pull request #4328 from dballesteros7/addTimestampToStatusView
Add update_time to result of requestByStatus view
commit b18312054395b2bd6d6913191a3f9750ea46e0ed
Author: Diego Ballesteros <dballest@fnal.gov>
Date: Wed Dec 5 17:19:29 2012 -0600
Add update_time to result of requestByStatus view
Add a WMStatsReader method that allows us to get
the request sorted by state with timestamp
of the state.
commit 1e12bfa8feae7bbdcbcf698f2f0e5bc3ff47a3e0
Author: Diego Ballesteros <dballest@fnal.gov>
Date: Wed Dec 5 14:51:13 2012 -0600
Ensure jobs are submitted, oldest workflow first
In Oracle we don't have guarantee of strictly increasing
indexes for the workflows and the JobSubmitter relies
on this to order the workflows and submit older workflows first.
Change the sort to use the timestamp in the last_update column
from the subscriptions.
commit e030e4aede1c7c1d52c6e91a6918025e1ff7ff52
Merge: 6cbc2e9 a45dfa8
Author: Stephen Foulkes <sfoulkes@fnal.gov>
Date: Wed Dec 5 12:46:59 2012 -0800
Merge pull request #4324 from zdenekmaxa/reqmgr-couchapps-configcache-cleanup
Cleaning up ReqMgr ConfigCache couchapp
commit f37a61335a89b24dc1be9ef0bc1f7b7bcf52db4a
Author: Seangchan Ryu <sryu@fnal.gov>
Date: Wed Dec 5 13:31:33 2012 -0600
handle when there is no jobs in request
commit 4a74365217e4138689cbdde26ce7579a1416e5d9
Author: Diego Ballesteros <dballest@fnal.gov>
Date: Wed Dec 5 12:54:50 2012 -0600
Avoid crashes on custodial subscription to non-MSS
The PhEDEx subscriber must check if a custodial
subscription will be made to a MSS node before doing it
otherwise PhEDEx will reject it and the poller will crash.
If a custodial subscription is made to a site without MSS
phedex node (no tape backend) then it defaults to a non-custodial
subscription.
commit a45dfa87e53a7f378f8a4c3af663be728b044b82
Author: Zdenek Maxa <zdenek.maxa@hep.caltech.edu>
Date: Tue Dec 4 18:27:13 2012 -0600
Cleaning up ReqMgr ConfigCache couchapp
commit 6cbc2e9fc053ac134a47696018e4cd5300a28aab
Author: Stephen Foulkes <sfoulkes@fnal.gov>
Date: Tue Dec 4 16:52:59 2012 -0600
0.9.33
Signed-off-by: Stephen Foulkes <sfoulkes@fnal.gov>
commit 201ef72c104725b73e2a2a3e14330bd06b27ac32
Merge: 8d35fbb badad7b
Author: Stephen Foulkes <sfoulkes@fnal.gov>
Date: Tue Dec 4 14:46:09 2012 -0800
Merge pull request #4308 from mmascher/accountant-location
Do the SE name => Site name conversion also for failed jobs
commit 8d35fbb1062374e75c7ea1d131276ad99917d91a
Merge: bb75562 f2dc186
Author: Stephen Foulkes <sfoulkes@fnal.gov>
Date: Tue Dec 4 14:45:45 2012 -0800
Merge pull request #4311 from ticoann/job_report_change
add job state transition information
commit bb755623f09d706ac42140264addb957c3cdf28a
Merge: 3fb41a2 ad6f726
Author: Zdenek Maxa <zdenek.maxa@hep.caltech.edu>
Date: Tue Dec 4 14:27:23 2012 -0800
Merge pull request #4319 from zdenekmaxa/reqmgr-remove-opsclipboard
Removing already dead ReqMgr/OpsClipboard
commit 3fb41a2070a04eb9e57f6dae97d6ad1da5a12173
Merge: 330e38f 2d58a70
Author: Stephen Foulkes <sfoulkes@fnal.gov>
Date: Tue Dec 4 13:08:14 2012 -0800
Merge pull request #4322 from dballesteros7/fixLSFTimeout
Fix timeout kills in LSF plugin
commit f2dc1862866c72d7c4c3295dfe783e0ea034bf89
Author: Seangchan Ryu <sryu@fnal.gov>
Date: Mon Nov 26 14:19:48 2012 -0600
add job state transition information
add agent info detail (handle thread is crashed)
restrict the state transition to prevent race condion
bug fix for state transition
commit 2d58a7017fba27692e203f5aece742d9929ba8dc
Author: Diego Ballesteros <dballest@fnal.gov>
Date: Tue Dec 4 11:28:00 2012 -0600
Fix timeout kills in LSF plugin
Add the Timeout state to the LSF plugin
state map, needed when a job is killed by timeout
commit ad6f7266063197ca8dc75660881464c950e8d6d2
Author: Zdenek Maxa <zdenek.maxa@hep.caltech.edu>
Date: Mon Dec 3 17:48:32 2012 -0600
Removing already dead ReqMgr/OpsClipboard
commit 330e38fb6f4eb98fd526cae734ecbeb8a8051346
Merge: 354a0aa 8985433
Author: Zdenek Maxa <zdenek.maxa@hep.caltech.edu>
Date: Mon Dec 3 09:37:30 2012 -0800
Merge pull request #4316 from zdenekmaxa/remove-remotemsg
Removing dead messaging feature RemoteMsg.
commit 354a0aa12cd6b71495ebde3eb8b36ef09f80f342
Merge: 54aa86f ce21f26
Author: Zdenek Maxa <zdenek.maxa@hep.caltech.edu>
Date: Mon Dec 3 09:34:43 2012 -0800
Merge pull request #4313 from zdenekmaxa/reqmgr-security-issue-permissions
Reqmgr - fix permission problem
commit 8985433f69b35eb651651fc6912f94de2e6e0021
Author: Zdenek Maxa <zdenek.maxa@hep.caltech.edu>
Date: Mon Dec 3 11:29:55 2012 -0600
Removing dead messaging feature RemoteMsg.
commit 54aa86f9a0e0736d980eae0517163e7dec3df12d
Merge: 4a70451 5dfd4a1
Author: Stephen Foulkes <sfoulkes@fnal.gov>
Date: Fri Nov 30 13:35:46 2012 -0800
Merge pull request #4312 from dballesteros7/changeLogCollectCollectionToList
Change LogCollect reporting from a reduce to list
commit ce21f26d0bd67712d0e619f6dde56842910d8c14
Author: Zdenek Maxa <zdenek.maxa@hep.caltech.edu>
Date: Fri Nov 30 14:21:42 2012 -0600
Reqmgr - fix permission problem
commit 5dfd4a1503cfe915b770e48c2faca127e903b94e
Author: Diego Ballesteros <dballest@fnal.gov>
Date: Fri Nov 30 14:03:48 2012 -0600
Change LogCollect reporting from a reduce to list
Growing arrays in a reduce can blow up couch, even if
bounded. Use a list function to collate the results,
make the appropiate changes
in the TaskArchiver and add a test for this in the
TaskArchiver unit test.
commit 4a7045138444c4274e76b3c0d0836b81431633cd
Author: Stephen Foulkes <sfoulkes@fnal.gov>
Date: Fri Nov 30 13:30:31 2012 -0600
0.9.32
Signed-off-by: Stephen Foulkes <sfoulkes@fnal.gov>
commit 9ac969a277b43b738fd44c5eb02e64588f948443
Merge: 8ef4aed 8e89393
Author: Stephen Foulkes <sfoulkes@fnal.gov>
Date: Fri Nov 30 11:24:00 2012 -0800
Merge pull request #4310 from dballesteros7/changeReduceToList
Change approach to getting tasks by dataset
commit 8e89393cf0ae9b0ea840760cb337b9081f70009b
Author: Diego Ballesteros <dballest@fnal.gov>
Date: Fri Nov 30 07:45:54 2012 -0600
Change approach to getting tasks by dataset
It was abusing of couchdb reduce by creating growing lists
(doesn't matter if bounded). Change the approach to using
a list with generates the desired mapping of tasks
that produced a dataset.
commit 8ef4aede97fafc69723ea3f931f82c0bfe0e1781
Merge: 5185905 3e18c4b
Author: Stephen Foulkes <sfoulkes@fnal.gov>
Date: Thu Nov 29 13:26:42 2012 -0800
Merge pull request #4309 from dballesteros7/fixReduceFunction
Fix Reduce function for outputByWorkflowName
commit 3e18c4bc70f4f40d287ba80f42c6a590e596fe1f
Author: Diego Ballesteros <dballest@fnal.gov>
Date: Thu Nov 29 14:45:39 2012 -0600
Fix Reduce function for outputByWorkflowName
It was generating a growing object in the output,
although it would only grow to size 2 at any given time.
Couchdb reduce_limit check throws an overflow
error on this function, changing the type to an array
fixes the problem. Added an object variable
that only lives in the reduce function to
keep track of duplicates.
Changed the TaskArchiver accordingly.
commit 51859054a71a8f83bc8fe8bb866cae9c0305a16f
Merge: 70e349d da1c927
Author: Stephen Foulkes <sfoulkes@fnal.gov>
Date: Wed Nov 28 09:22:33 2012 -0800
Merge pull request #4307 from dballesteros7/fixLogArchives
Include the new PSet pickle in the logArchs
commit badad7ba3135162227f2fac8dc9c376008dba144
Author: Marco Mascheroni <marco.mascheroni@cern.ch>
Date: Wed Nov 28 17:57:31 2012 +0100
Do the SE name => Site name conversion also for failed jobs
commit da1c9278b927393d3e28afb545fbfaf2e489da67
Author: Diego Ballesteros <dballest@fnal.gov>
Date: Wed Nov 28 00:19:29 2012 -0600
Include the new PSet pickle in the logArchs
Since we changed to having a pickle file
with the configuration, the logArchs only have
the .py file which is useless without the pickle.
commit 70e349d36597ad0f4d94ceebe534807a6c730197
Merge: 30afbb6 25f5a8e
Author: Stephen Foulkes <sfoulkes@fnal.gov>
Date: Tue Nov 27 07:13:22 2012 -0800
Merge pull request #4306 from giffels/dbs-lexicon-changes
Added basic http(s) url validation for DBS3
commit 25f5a8ec8ab5ad3cf475c865edb9175d028bdf5c
Author: Manuel Giffels <giffels@gmail.com>
Date: Tue Nov 27 13:51:29 2012 +0100
Added basic http(s) url validation for DBS3
commit 30afbb68d5ff4d08276b994f3e9e61731348c3f2
Author: Stephen Foulkes <sfoulkes@fnal.gov>
Date: Mon Nov 26 14:27:57 2012 -0600
0.9.31
Signed-off-by: Stephen Foulkes <sfoulkes@fnal.gov>
commit 6ffbbb163782a5e940cedf7726ade1c8b79e1937
Merge: fc96eec db50ae8
Author: Stephen Foulkes <sfoulkes@fnal.gov>
Date: Mon Nov 26 11:09:50 2012 -0800
Merge pull request #4304 from dballesteros7/fix-unittests-workloadsummary
Fix unit tests for new data structure
commit db50ae859bbec64d3a38be3182aadf495c8ac18d
Author: Diego Ballesteros <dballest@fnal.gov>
Date: Mon Nov 26 11:59:43 2012 -0600
Fix unit tests for new data structure
Test was incompatible with 2.6 due to the delta
parameter in unittest.TestCase.assertAlmostEqual
that was introduced only until 2.7. Replace it
with places instead.
commit fc96eec87859b81a6daa71ab91c1b59551980f9b
Merge: d019cb5 18e1366
Author: Stephen Foulkes <sfoulkes@fnal.gov>
Date: Mon Nov 26 09:02:34 2012 -0800
Merge pull request #4302 from sfoulkes/himc
Add /store/himc as an assignment option.
commit 18e1366c6e4adddd1c45ecfe925552a495ee9ec6
Author: Stephen Foulkes <sfoulkes@fnal.gov>
Date: Mon Nov 26 11:00:55 2012 -0600
Add /store/himc as an assignment option.
commit d019cb56c2240d5cd6ed69c4cd3fac7f53b32057
Merge: 82d8d8e 0c2d9ba
Author: Stephen Foulkes <sfoulkes@fnal.gov>
Date: Mon Nov 26 08:37:41 2012 -0800
Merge pull request #4222 from dballesteros7/implement-checksum-check-lcgImpl
Implement checksums in LCGImpl backend module
commit 82d8d8edf8a476546cbd3a081749be87bba404cb
Merge: 4631426 b8870c5
Author: Stephen Foulkes <sfoulkes@fnal.gov>
Date: Mon Nov 26 08:37:08 2012 -0800
Merge pull request #4285 from mmascher/check-lumi
Check that files contains at least one good lumi before adding them to W...
commit 4631426d017a4887349c8e94544009885064918c
Merge: 71c8eb4 7409d07
Author: Stephen Foulkes <sfoulkes@fnal.gov>
Date: Mon Nov 26 08:36:14 2012 -0800
Merge pull request #4246 from PerilousApricot/feature/subclassable-worker-sleep
Make component sleeping configurable
commit 71c8eb4e712d66a8a02ccb2828ecdbc0302178e6
Merge: 8e5644c 209e6bf
Author: Stephen Foulkes <sfoulkes@fnal.gov>
Date: Mon Nov 26 08:35:28 2012 -0800
Merge pull request #4283 from dballesteros7/bugfix-insert-thresholds
Minor bugfix for the resource control script
commit 8e5644cd94614b02041c54ee6c285b5c56215e9f
Merge: b5a6cba bc5b4e0
Author: Stephen Foulkes <sfoulkes@fnal.gov>
Date: Mon Nov 26 08:35:06 2012 -0800
Merge pull request #4284 from dballesteros7/workloadsummar-nullauth
Allow nullauth for WorkloadSummary in local couch
commit b5a6cba85cb840cc498a7fffc776e6e2432485c7
Merge: b35cab4 aaa4d35
Author: Stephen Foulkes <sfoulkes@fnal.gov>
Date: Mon Nov 26 08:34:19 2012 -0800
Merge pull request #4288 from dballesteros7/phedexSubscriptionFix
Remove restriction of autoapprove on custodial
commit b35cab426a8c6ed388fe3e878fdfbc07bc2b84e3
Merge: 843c0a9 ca0db2c
Author: Stephen Foulkes <sfoulkes@fnal.gov>
Date: Mon Nov 26 08:27:49 2012 -0800
Merge pull request #4298 from samircury/failed-jobs-histograms
Failed jobs histograms
commit 843c0a98efecfad19fc6f603a885fcc261c67981
Merge: aaa3ca1 88cf145
Author: Stephen Foulkes <sfoulkes@fnal.gov>
Date: Mon Nov 26 08:21:42 2012 -0800
Merge pull request #4295 from dballesteros7/dqmharvest-configcache
Allow ConfigCacheIDs for DQM harvesting
commit aaa3ca15942f373c3d8193479652fe0f219d4234
Merge: 70b56ea 2d116fb
Author: Stephen Foulkes <sfoulkes@fnal.gov>
Date: Mon Nov 26 08:21:04 2012 -0800
Merge pull request #4301 from dballesteros7/fix-update-jobs-dao
Fix status_time type for oracle DAO
commit 70b56ea54e7ede719f77f4951dbf614dae20b87a
Merge: 45d69ac bb8afc1
Author: Stephen Foulkes <sfoulkes@fnal.gov>
Date: Mon Nov 26 08:20:23 2012 -0800
Merge pull request #4299 from dballesteros7/exploit-xrootd
Allow requests to run at sites without the data
commit 45d69ace6842931e6f8cad41ef98c67cbdca5ddf
Merge: 06e7439 1a2f077
Author: Stephen Foulkes <sfoulkes@fnal.gov>
Date: Mon Nov 26 08:18:36 2012 -0800
Merge pull request #4300 from sfoulkes/hidata
Add /store/hidata as a possible LFN base for single request assignment
commit 2d116fb41d496c93931260c981878c55c512034c
Author: Diego Ballesteros <dballest@fnal.gov>
Date: Mon Nov 26 10:18:17 2012 -0600
Fix status_time type for oracle DAO
Oracle optimization require a constant type on the binds
when calling a query many times. Ensure this from the CondorPlugin
track method.
Fixes #4297
commit 1a2f0774073d05c19ef07975fa9096a7ca34630c
Author: Stephen Foulkes <sfoulkes@fnal.gov>
Date: Mon Nov 26 10:15:46 2012 -0600
Add /store/hidata as a possible LFN base for single request assignment
and add tests to the Lexicon tests verifying that works as well.
commit 06e74393872519414dcd6b1a5230fc153893d4ff
Merge: fed19d2 b8a6bd5
Author: Stephen Foulkes <sfoulkes@fnal.gov>
Date: Mon Nov 26 06:48:31 2012 -0800
Merge pull request #4164 from mmascher/poller-errors
When killing a job for timeout, check if a report exist before writing t...
commit ca0db2cd524cab9b094c5f989a05a4a833e39d41
Author: Samir <samir@cern.ch>
Date: Fri Nov 23 18:16:54 2012 +0100
* Adds histograms for failing jobs per task/step under the label of errorsHistogram
* Improves TaskArchiver test adding the feature where half of the jobs get the FWJR of a succeeding job, just to have variety in the job set and simulate prod situations where there are failed jobs, but those are not all jobs.
* Adds jobTime as a histogram key, so now we will actually generate histograms in the summary of this test, before we weren't
Fixes #4215
commit 7d6bc96c2be415dc1c306cba3ab8be5b15af28bc
Author: Diego Ballesteros <dballest@fnal.gov>
Date: Tue Nov 20 10:05:11 2012 -0600
Implement summary of failure per site
Include two type of histograms with the number of failed jobs
per site. One given per site at a workflow level, and other
per step and exit code.
This is included in the summary using the new Histogram data structure
that allows generic histograms to be built with a simple interface.
This strutcture takes care of statistic calculation and generation
of a dictionary suitable for couchdb, also is meant to provide
support for multiple agents which will require merging summaries.
Fixes #4213
commit 06ad15739c35db813bdfaacd9ab2b5bf87b4bb10
Author: Diego Ballesteros <dballest@fnal.gov>
Date: Fri Nov 16 13:26:47 2012 -0600
Include REST API for missing lumis in the summary
Add a show function which works as a REST API
to get the missing lumis for a completed workflow from
the workload summary.
URL is ugly at the moment:
/_design/WorkloadSummary/_show/lumiMissingByWorkflow/<workflowName>?outputDataset=<string>
The outputDataset parameter is optional and is used to filter the output. The format of the output
is:
{<outputDataset:string> : { <run:string> : [ [<startLumi:int>, <endLumi:init>], ...] , ... }, ...}
Note that for summaries produced by agents without the TaskArchiver changes in this patch, the outputDataset won't
be returned as a key, instead it will be the task where the lumis were lost.
Fixes #4216
commit bb8afc18ec02cf3ca91c1fc19a17eb3e0ca114e5
Author: Diego Ballesteros <dballest@fnal.gov>
Date: Thu Nov 22 03:45:26 2012 -0600
Allow requests to run at sites without the data
Include a flag at assignment time that
allows ops to decide if a request should run
in all CMS sites filtered by whitelist/blacklist
instead of where the data is. This assumes Ops knows
what is doing (sites have xrootd to get the data).
commit fed19d28cc69db9ee4ad8c65d0cfbc3e3368d8f5
Merge: 5361dc7 d97f8c3
Author: Andrew Melo <andrew.melo@gmail.com>
Date: Thu Nov 22 16:26:57 2012 -0800
Merge pull request #4291 from samircury/logarchives-listing-workflowsummary
Adds PFN of logArchives@CERN per task to the summary
commit 88cf145ffbf7b4f7b0887d071cc43bac7e4ecf80
Author: Diego Ballesteros <dballest@fnal.gov>
Date: Thu Nov 22 00:14:31 2012 -0600
Allow ConfigCacheIDs for DQM harvesting
Add the option in the ReqMgr and handle the specific tweaks
for harvesting
Fixes #4262
commit 5361dc7d38838446ccb8b9670cb985e37270b24d
Merge: 51aafc3 130df0d
Author: Zdenek Maxa <zdenek.maxa@hep.caltech.edu>
Date: Wed Nov 21 09:49:53 2012 -0800
Merge pull request #4293 from zdenekmaxa/reqmgr-ops-scrips-02-1
ConfigCacheUrl, extented reqmgr.py Ops script
commit 130df0de620305d163a43610021c7e259f8ff395
Author: Zdenek Maxa <zdenek.maxa@hep.caltech.edu>
Date: Mon Nov 19 23:17:05 2012 +0100
ConfigCacheUrl, extented reqmgr.py Ops script
* ConfigCacheUrl implemented for all worklflows types (allows to load
ConfigCache document from an arbitrary CouchDB server, not necessarily
the one coupled with ReqMgr (whose URL ReqMgr passes in CouchURL
argument)).
ConfigCacheUrl, if specified, overrides CouchURL. Is backwards compatible,
if not specified, CouchURL is used as before. reqmgr.py Ops ReqMgr client
script enhanced - more workflows template arguments defined
(test/data/ReqMgr/requests)
Fixes #4269
* Ops ReqMgr client script allows to create and assign requests in one go or
separately. Approve is always coupled with create. If necessary can be
separated too, however, if create succeeds, approve must too. Unfortunately,
assignment lacks REST API so the client talks to a webpage.
Ideally (for ReqMgr2), REST API shall support all possible status
transitions, this will then be more easily exposed to reqmgr.py.
Fixes #4247
* Add /store/hidata to the lfn base allowed in the assign page of the
ReqMgr. Fixes 4287
* minor reqmgr.py bugfix, template default values agreed by the Ops, tested
* by the Ops on a private VM instance
commit d97f8c34f500cf8363eff158dac1e2ee94774cc4
Author: Samir <samir@cern.ch>
Date: Mon Nov 19 13:11:31 2012 +0100
Adds a real LogCollect fwjr to the TaskArchiver test and changes to support it better. Changes in the summary document. fixes #4214
commit aaa4d3589157cd1637019242d3a39f6667a414b4
Author: Diego Ballesteros <dballest@fnal.gov>
Date: Mon Nov 19 11:32:51 2012 -0600
Remove restriction of autoapprove on custodial
Removing an unnecessary restriction I added that prevented
custodial subscriptions from being auto approve as well.
The Tier-0 needs it since we auto approve the custodial
subscriptions to CERN.
commit 209e6bf8078c356ef13f16d8dea6d8f576ad46ce
Author: Diego Ballesteros <dballest@fnal.gov>
Date: Fri Nov 16 13:44:01 2012 -0600
Minor bugfix for the resource control script
The update threshold option had the pending and running
slots for a task swapped.
commit 51aafc38fc36b612decce10fc49f15eb493a6af7
Merge: d2e5843 70d790a
Author: Stephen Foulkes <sfoulkes@fnal.gov>
Date: Fri Nov 16 07:10:55 2012 -0800
Merge pull request #4279 from ticoann/add_output_datasets
add output dataset
commit bc5b4e0336de9e4827248a6e04970ca4fa5a9bb7
Author: Diego Ballesteros <dballest@fnal.gov>
Date: Thu Nov 15 13:07:13 2012 -0600
Allow nullauth for WorkloadSummary in local couch
Check for null auth and allow it,
also allow the update of existing documents
this is to provide support for multiple agents
and eventually cleanup. The TaskArchiver already takes care
of not doing funny stuff and uploading stuff twice
unless it's necessary.
commit 70d790aad244d091db7737203545d368439eed70
Author: Seangchan Ryu <sryu@fnal.gov>
Date: Thu Nov 15 15:26:35 2012 -0600
add output dataset
commit d2e58432dba15e2ba3962ca62cebf0fe677baae2
Merge: 7adcc2f 2e52b91
Author: Stephen Foulkes <sfoulkes@fnal.gov>
Date: Thu Nov 15 08:48:36 2012 -0800
Merge pull request #4147 from dballesteros7/job-creator-mysql
Remove Autoincrement check for MySQL
commit 7adcc2f93760c2cf22b911f3cb8446dc66a50e4e
Merge: da3b9f7 41eda72
Author: Stephen Foulkes <sfoulkes@fnal.gov>
Date: Thu Nov 15 08:39:20 2012 -0800
Merge pull request #4277 from dballesteros7/fixResourceControl-binds
Repair resource control interface
commit 41eda723420d9cbf3eabc29e9779b63813ef709f
Author: Diego Ballesteros <dballest@fnal.gov>
Date: Thu Nov 15 09:32:36 2012 -0600
Repair resource control interface
It was blowing on Oracle, MySQL didn't complain but
it wasn't doing the right thing
commit b8870c586c351c20ee2cb6b9600f84b2eea1a236
Author: Marco Mascheroni <marco.mascheroni@cern.ch>
Date: Thu Nov 15 14:52:13 2012 +0100
Check that files contains at least one good lumi before adding them to WWMBS. Fixes #4267
commit da3b9f780520c74f19ea715cb07d505c14b6886d
Author: Stephen Foulkes <sfoulkes@fnal.gov>
Date: Wed Nov 14 16:36:57 2012 -0600
0.9.30
Signed-off-by: Stephen Foulkes <sfoulkes@fnal.gov>
commit d439c7d52a17470083e8fa4ac82d2e72ae017c2d
Merge: cea9f19 c67eaf5
Author: Stephen Foulkes <sfoulkes@fnal.gov>
Date: Wed Nov 14 16:36:15 2012 -0600
Merge branch 'master' of github.com:dmwm/WMCore
commit c67eaf550a3342c696d6d6dd12d43bbeb7761c13
Merge: 6f60062 f3ddd33
Author: Stephen Foulkes <sfoulkes@fnal.gov>
Date: Wed Nov 14 14:35:56 2012 -0800
Merge pull request #4272 from dballesteros7/fix-unit-tests-rc
Fix unit tests broken
commit f3ddd33634a2bb2630d3469f5e401a040cb42292
Author: Diego Ballesteros <dballest@fnal.gov>
Date: Wed Nov 14 16:17:59 2012 -0600
Fix unit tests broken
Change of interface that many test use
Fixes #4271
commit cea9f19c690b49953cd662640da1f9a44b607fc5
Author: Stephen Foulkes <sfoulkes@fnal.gov>
Date: Wed Nov 14 16:25:41 2012 -0600
0.9.29
Signed-off-by: Stephen Foulkes <sfoulkes@fnal.gov>
commit 6f600623d99924c8dd1e07c6bd08f1d3cf91c2ef
Merge: aeb3130 21f9bd4
Author: Stephen Foulkes <sfoulkes@fnal.gov>
Date: Wed Nov 14 14:01:59 2012 -0800
Merge pull request #4270 from dballesteros7/pending-jobs-per-task
Implement pending jobs per task/site functionality
commit 21f9bd486a77d455e4efa8d9ec17a4c72a3f790e
Author: Diego Ballesteros <dballest@fnal.gov>
Date: Mon Nov 12 12:52:35 2012 -0600
Implement Pending jobs per task/site functionality
Add a new column to the rc_threshold table, this indicates the number of pending
jobs to keep for an individual task.
Modify all the concerned DAOs and the JobSubmitter.
Rewrote the JobSubmitterPoller unit tests to make them all non-integration,
break the dependecy of having condor available and add tests for the different
mechanics happening in it such as threshold management, site status,
site whitelists/blacklists. Also includes a performance test.
Finally add a special option to setup the Tier-0 thresholds in one line from the
resource control script.
commit aeb31307e5e8c93ad78bb5bb712954ecdb4d503c
Merge: dc9553b 8258774
Author: Stephen Foulkes <sfoulkes@fnal.gov>
Date: Wed Nov 14 06:40:48 2012 -0800
Merge pull request #4265 from dballesteros7/tier0-runStates
Tier0 run states
commit dc9553b09f91dfa020175c0737258bc73db7ed8a
Merge: 61a1fe3 0962526
Author: Zdenek Maxa <zdenek.maxa@hep.caltech.edu>
Date: Wed Nov 14 00:02:37 2012 -0800
Merge pull request #4237 from zdenekmaxa/reqmgr-request-clone
ReqMgr requests clone implemented.
commit 61a1fe3b0010da4144f40507839cffd6f976a8b3
Merge: d6c68d7 198da7c
Author: Zdenek Maxa <zdenek.maxa@hep.caltech.edu>
Date: Tue Nov 13 14:34:27 2012 -0800
Merge pull request #4253 from PerilousApricot/bugfix/revert-a-revert-a-revert
revert a revert a revert
commit 8258774df9f467bc9b6824e0f366f94a95942fa7
Author: Diego Ballesteros <dballest@fnal.gov>
Date: Fri Nov 9 01:13:25 2012 -0600
Tier0 Plugin implementation
The Tier0 plugin for the AnalyticsDataCollector components
calculates the state of Tier-0 workflows based on subscription
information found in WMBS and updates the request document
in central WMStats with this status.
Additionally this patch fixes some bugs with PluginInterface implementation
patch and renames it from PlugIn to Plugin to conform with the same
name as the BossAir package (PlugIn is a bit harder to remember).
Fix the AnalyticsDataCollector unittest and removed it from the
allowed failing tests.
This requires #4227 to work, it also fixes the unit test breaks from it.
Fixes #4264
commit 2f5e7f87ab99b679062fcabdcaab86d4920dcc5e
Author: Seangchan Ryu <sryu@fnal.gov>
Date: Thu Nov 1 13:29:02 2012 -0500
add PlugIn structure in AnalyticsDataCollector
commit d6c68d7bd75d8c0750ca3278a9b608923567bd44
Merge: 27e0f3d ca944fe
Author: Stephen Foulkes <sfoulkes@fnal.gov>
Date: Tue Nov 13 13:55:56 2012 -0800
Merge pull request #4227 from ticoann/merge_wmagent_summary_delete
deleting jobsummary from wmagent summary when request is announced
commit 27e0f3d742d6f46a39d5f24f926a3953380919ea
Merge: 25a4f3c d08bd4b
Author: Stephen Foulkes <sfoulkes@fnal.gov>
Date: Tue Nov 13 09:29:36 2012 -0800
Merge pull request #4244 from dballesteros7/display-errors-workload-summary
Fix error display in workload summary
commit 25a4f3cb848384d86059f2736fbabd3db21932bd
Merge: 928c20b 2b13ba9
Author: Stephen Foulkes <sfoulkes@fnal.gov>
Date: Tue Nov 13 09:11:35 2012 -0800
Merge pull request #4242 from dballesteros7/input-any-acdc-collection-name
Allow arbitrary collection name on ACDC request
commit 928c20b8fc2ee53eef092e1a4d9e014349a39579
Merge: a05e9a4 22b5967
Author: Stephen Foulkes <sfoulkes@fnal.gov>
Date: Tue Nov 13 08:09:40 2012 -0800
Merge pull request #4239 from yuyiguo/lexicon_fix
lexicon add dbs user
commit a05e9a4bd9b8d52ed9b96f2d68c59b19f6271d58
Merge: 9ffda12 b23c608
Author: Stephen Foulkes <sfoulkes@fnal.gov>
Date: Mon Nov 12 12:44:48 2012 -0800
Merge pull request #4220 from ticoann/jobstate_typo
fix the typo in the jobstatus (affect monitor update in submit paused status)
commit 9ffda1287a32950fa3370a0fd5777bff0cfba3c9
Merge: 1b60a96 af54eb6
Author: Stephen Foulkes <sfoulkes@fnal.gov>
Date: Mon Nov 12 12:44:23 2012 -0800
Merge pull request #4260 from dballesteros7/phedex-inject-override
Implement override for PhEDEx injections
commit 1b60a969e95731c164618e2ff9080d24edbb7320
Merge: c64810b 8d0d93c
Author: Stephen Foulkes <sfoulkes@fnal.gov>
Date: Mon Nov 12 08:08:04 2012 -0800
Merge pull request #4261 from dballesteros7/stageout-code-fix
Fix broken stageout code
commit 8d0d93cbc1a214926b8a6d108ef12eb85ce50503
Author: Diego Ballesteros <dballest@fnal.gov>
Date: Mon Nov 12 09:56:24 2012 -0600
Fix broken stageout code
Combine fixes from @PerilousApricot and @hufnagel, the check
on the new stageout code is done without the use of a helper
to keep consistency with the rest of the executor code as
implemented in #4259.
The unit tests from #4251 are included and working.
commit c64810b216a9c24051ded4965cdf870e9c66c037
Merge: 7bad5fd a83d935
Author: Stephen Foulkes <sfoulkes@fnal.gov>
Date: Mon Nov 12 07:18:51 2012 -0800
Merge pull request #4258 from bbockelm/cvmfs_support
Add support for CVMFS, based on the CRAB implementation
commit a83d9358f0738e33f690158446d697e5fa3e3596
Author: Brian Bockelman <bbockelm@cse.unl.edu>
Date: Sun Nov 11 14:10:25 2012 -0600
Add support for CVMFS, based on HN conversation and identical to the CRAB bootstrapping code.
commit af54eb6de21aced49b5f61176e65c5f8b5941442
Author: Diego Ballesteros <dballest@fnal.gov>
Date: Fri Nov 9 17:57:28 2012 -0600
Implement override for PhEDEx injections
Due to the special setup of the T0/T2 at CERN
and the Bulk/Express processing in the Tier-0. We need
the ability to specify where the files should be registered
to in PhEDEx. This is done per spec, and all files
associated with the workflows from that spec
will be registered to the given site, regardless of the SE
available in the dbsbuffer table or custodial site
entry in dbsbuffer.
Also fix a unit teste in WMWorkload_t
commit ca944fe957e3c3171b1c1621fc50365720040852
Author: Seangchan Ryu <sryu@fnal.gov>
Date: Mon Oct 29 11:34:20 2012 -0500
deleting job summary from wmagent summary on given condition
commit 7bad5fdc7e9526dfcd030e528c10f756795fe88b
Author: Andrew Melo <andrew.melo@gmail.com>
Date: Wed Nov 7 11:17:50 2012 -0600
0.9.28
Signed-off-by: Andrew Melo <andrew.melo@gmail.com>
commit e4d8ecd822884c36a7a1c4aed738895bd93ff403
Merge: 3126707 322f97a
Author: Andrew Melo <andrew.melo@gmail.com>
Date: Wed Nov 7 08:59:40 2012 -0800
Merge pull request #4252 from PerilousApricot/bugfix/revert-a-revert
Do something crazy
commit 198da7c291c5d62708bdd1b61db6f0e835720133
Author: Stephen Foulkes <sfoulkes@fnal.gov>
Date: Tue Oct 30 17:56:15 2012 -0500
Revert Revert Revert
This reverts a previous commit that reverted a previous commit. Solid.
commit 322f97af6b7160446da3e821baeb5f965a03aa1d
Author: Stephen Foulkes <sfoulkes@fnal.gov>
Date: Tue Oct 30 17:53:01 2012 -0500
Do something crazy
I will now revert a revert so something something something
After that I will revert this reverted revert so I can get things back
to normal.
commit 3126707d8e87b0d86c52e9bdb8faaca9b0979bf3
Merge: eb65f8b 2f16328
Author: Zdenek Maxa <zdenek.maxa@hep.caltech.edu>
Date: Wed Nov 7 07:34:45 2012 -0800
Merge pull request #4240 from dballesteros7/fix-reqmgr
Bugfix StdBase DQMHarvest validation
commit 7409d07461039b26f090ca525d005a0ea73a19b0
Author: Andrew Melo <andrew.melo@gmail.com>
Date: Mon Nov 5 11:03:49 2012 +0100
Make component sleeping configurable
Factor out the sleeping part of the component of the component harness.
In some cases, simply sleeping isn't the best idea, this change makes it
so subclasses of BaseWorkerThread can choose how they sleep.
commit d08bd4be17394099726f17e5466574911aec5253
Author: Diego Ballesteros <dballest@fnal.gov>
Date: Thu Nov 1 18:28:28 2012 -0500
Fix error display in workload summary
Display all errors correctly
Fixes #4217
commit 2b13ba953f22ba2f072304615f3f3b2c8588cfbc
Author: Diego Ballesteros <dballest@fnal.gov>
Date: Fri Nov 2 10:08:08 2012 -0500
Allow arbitrary collection name on ACDC request
A CollectionName parameter can be specified in the request arguments
if not specified the collection name defaults to the old request name
as it has done so far.
commit 2f16328c4d16ff24805c1b946aca9f17b27409d5
Author: Diego Ballesteros <dballest@fnal.gov>
Date: Thu Nov 1 15:38:44 2012 -0500
Bugfix StdBase DQMHarvest validation
Simply a code error
commit 22b5967677b08dc5cf4286629c4ad68520944951
Author: yuyi@fnal.gov <yuyi@fnal.gov>
Date: Wed Oct 31 16:07:46 2012 -0500
lexicon add dbs user
commit 0962526adf0e6a0798a24e25db508ca8c492680e
Author: Zdenek Maxa <zdenek.maxa@hep.caltech.edu>
Date: Wed Oct 31 20:58:33 2012 +0100
ReqMgr requests clone implemented.
A few functions / methods signatures changed in the ReqMgr files.
Requests clone implemented: PUT, /reqmgr/reqMgr/clone/requestName, unittest
for clone and reqmgr.py clone interface.
reqmgr.py has --allTests to run over everything the script is capable of for
testing.
Clone request shall fix: #3984, #3944
commit eb65f8bb5da3b8dff7bf5ed0932b3505053951ec
Author: stuartw <stuart.wakefield@gmail.com>
Date: Wed Oct 31 15:16:45 2012 +0000
Remove updates in cancellation - stop conflicts
commit 1df71373fe53a9c37c2e33d6b14802f854889304
Author: Stephen Foulkes <sfoulkes@fnal.gov>
Date: Tue Oct 30 17:56:15 2012 -0500
Revert "Revert "Ops ReqMgr interaction script, ProcConfigCacheID renaming""
This reverts commit 138bf6cf591a23af5cf99ff0d03d0ae1f9fbb0f2.
commit 504fba6863a38c0c3d0b3764a54b3eed1b4cb242
Author: Stephen Foulkes <sfoulkes@fnal.gov>
Date: Tue Oct 30 17:55:04 2012 -0500
0.9.27
Signed-off-by: Stephen Foulkes <sfoulkes@fnal.gov>
commit 138bf6cf591a23af5cf99ff0d03d0ae1f9fbb0f2
Author: Stephen Foulkes <sfoulkes@fnal.gov>
Date: Tue Oct 30 17:53:01 2012 -0500
Revert "Ops ReqMgr interaction script, ProcConfigCacheID renaming"
This reverts commit d01c12d692825c485dab43849765e08e59d237ba.
commit efbcaa71c7a62c6958a05e5142af714eb367f339
Author: Seangchan Ryu <sryu@fnal.gov>
Date: Tue Oct 30 14:29:32 2012 -0500
remove workloadsummary installation from reqmgr and move to reqmon
commit 0bdfa197e0e4eefda3a68ce03bd70c1db8974ad9
Merge: 9d4beeb db9a6a4
Author: Stephen Foulkes <sfoulkes@fnal.gov>
Date: Tue Oct 30 12:27:21 2012 -0700
Merge pull request #4231 from dballesteros7/fix-wmagent-config-phedex
Bugfix for wmagent config
commit 9d4beeb66a702dbb4c09f7e0caefdbf506d562db
Merge: 60c5772 d6ea9d2
Author: Stephen Foulkes <sfoulkes@fnal.gov>
Date: Tue Oct 30 10:52:35 2012 -0700
Merge pull request #4194 from PerilousApricot/feature/stageout-switch
Add per-request switch for stageout
commit db9a6a417b227bd26fd5668bd5354275d7609bc3
Author: Diego Ballesteros <dballest@fnal.gov>
Date: Tue Oct 30 11:23:23 2012 -0500
Bugfix for wmagent config
The spec cache path was not correctly set by the mod-config
commit 60c5772d339f49a167c66f7960a4ee04e3745eca
Merge: 00cf065 0dcb89f
Author: Stephen Foulkes <sfoulkes@fnal.gov>
Date: Tue Oct 30 08:17:59 2012 -0700
Merge pull request #4228 from dballesteros7/fix-taskchain-bug
Bugfix for PrimaryDataset handling in RelVal
commit 0dcb89fae2ea2a77880dbb95f344b426af826207
Author: Diego Ballesteros <dballest@fnal.gov>
Date: Mon Oct 29 13:47:44 2012 -0500
Bugfix for PrimaryDataset handling in RelVal
The PrimaryDataset argument wasn't really suited to be included in the
ParameterStorage decorator used to modify global parameters according
to task-level specifications. Since the PrimaryDataset
is not really a global parameter and always a task one, changed this support
for PrimaryDataset in order to make this actually work.
Added unit test verification of the output datasets to catch these errors.
Fixes #4225
commit 00cf0653636e30ebc46d082956657db8a921036d
Merge: 8820331 d01c12d
Author: Zdenek Maxa <zdenek.maxa@hep.caltech.edu>
Date: Sat Oct 27 16:49:59 2012 -0700
Merge pull request #4223 from zdenekmaxa/reqmgr-ops-scrips-01
Ops ReqMgr interaction script, ProcConfigCacheID renaming
commit d01c12d692825c485dab43849765e08e59d237ba
Author: Zdenek Maxa <zdenek.maxa@hep.caltech.edu>
Date: Sun Oct 28 01:40:01 2012 +0200
Ops ReqMgr interaction script, ProcConfigCacheID renaming
commit 0c2d9ba115cbc39ccf0d740840d5aa452745ee55
Author: Diego Ballesteros <dballest@fnal.gov>
Date: Fri Oct 26 13:31:28 2012 -0500
Implement checksums in LCGImpl backend module
Add support for verifying the adler32 checksum when available
through the lcg-ls -l command. Also add general support
for passing the locally calculated checksums of the files
to the StageOutImpl objects, this will enable the checksum
verification in the rfcp-CERN backend plugin as well.
For the LCGImpl plugin the checksum is obtained as follows:
echo "$LCG_OUTPUT" | sed -nr 's/^.*\s([a-f0-9]{8})\s*\([aA][dD][lL][eE][rR]32\)\s*$/\\1/p'
Where LCG_OUTPUT=`lcg-ls -l -b -D srmv2 <SURL> 2>/dev/null`
The expected format of LCG_OUTPUT is a multi-line string where one of the lines has the following format:
* Checksum: <8-hex character string> (ADLER32)
Or
* Checksum: <8-hex character string> (adler32)
If this line is not present then it is assumed that the checksum is not supported and the check is skipped,
but if a checksum is present then it will be compared and will fail with exit code 63011 if it doesn't match.
Error code is the same as for size mismatch.
Fixes #4185
commit b23c6085b5cdd4a6210c53ad216173ce928158fb
Author: Seangchan Ryu <sryu@fnal.gov>
Date: Thu Oct 25 15:20:40 2012 -0500
fix the typo in the jobstatus
commit 8820331a4e14d54250b9b3d67f1b67c1c1…
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This will delete job summary data which is announced or aborted.
This will have problem of race condition if workflow is splited between agent. So need to provide additional solution in that case. Currently we don't have that case.
Steve please review