ChangeLog.txt

Release Note

* 12/12/2014
  * added junk file cleaner in AtlasPP
  * improved AtlasTS to open output/log datasets

* 12/10/2014
  * to rescue random_seed
  * fixed ddo check in AtlasTR

* 12/9/2014
  * added some debug messages in insertFilesForDataset_JEDI

* 12/5/2014
  * using t1Weight=0 when undefined
  * increased threshold for fax overflow to 0.5 

* 12/4/2014
  * changed AtlasProdJobBroker to send scouts to T1
  * changed AtlasProdJobBroker to count # of jobs regardless of priorities
  * fixed for ddo:

* 12/3/2014
  * using old=True for replica lookup
  * using nEventsPerFile when rucio gives nevents=None

* 12/2/2014
  * added debug messages to AtlasDDM

* 12/1/2014
  * setting ddmBackEnd for ATLAS tasks

* 11/27/2014
  * set frozenTime for pending in Refiner

* 11/26/2014
  * fixed setSplitRule for -
  * using frozenTime for pending
  * added getOldBuildFileSpec
  * checking attribute lengths before registering tasks

* 11/25/2014
  * added unlockTasks
  * added maxAttemptES
  * using merge scout attributes    

* 11/24/2014
  * added pauseTask and resumeTask
  * unthrottling to generate merge jobs

* 11/19/2014
  * fixed AtlasDDM for analysis+rucio
  * skipping setOwner for rucio
  * limitting nRecords with event-splitting in insertFiles
  * using pid to lock in CF

* 11/18/2014
  * fixed AtlasDDM for DQDatasetExistsException
  * change to avoid contention in insertFilesForDataset_JEDI

* 11/17/2014
  * changed JG to (un)set lockedTime
  * changed AtlasProdJobThr to send logging messages 
  * improved throttling in AtlasProdJobThr

* 11/14/2014
  * changed AtlasAnalJB to check core matching
  * fixed reassignTask to not use pending as oldStatus

* 11/13/2014
  * fixed setCloudToTasks_JEDI to avoid late update
  * skipping file lookup for prod merging

* 11/12/2014
  * added random and reusable to dataset attributes

* 11/11/2014
  * fixed getRW
  * removed patch

* 11/10/2014
  * changed meas_date to xrdcp_last_update

* 11/7/2014
  * fixed reassignTaskToSite 

* 11/6/2014
  * introduced respectLB and nEventsPerMergeJob

* 11/5/2014
  * fixed JG to set destinationDBlock
  * fixed AtlasProdJB to check storageToken

* 11/4/2014
  * fixed nGBPerMergeJob
  * changed for lumiBlockNr

* 10/30/2014
  * added child process check in Interaction
  * added more informative message for refine failure

* 10/29/2014
  * added ProcessWrapper
  * fixed JediDBProxy to correctly set superstatus when clouds are assigned
  * fixed insertFiles for filelist on container

* 10/28/2014
  * added memory monitor in Interaction

* 10/27/2014
  * fixed tape lookup in AtlasAnalJB
  * added timeout to join in JG
  * added meas_date limit to NN
  * changed AtlasAnalJB for AthAnalysis
  * changed to allow tape access to users
  * added t1Weight
  * fixed AtlasProdTaskPP to delete intermediate datasets

* 10/24/2014
  * added safety margin in walltime estimation

* 10/21/2014
  * fixed AtlasProdJobThrottler for composit work queues

* 10/20/2014
  * fixed JG for startEvent

* 10/15/2014
  * improved AtlasProdTR to set destination from dst:

* 10/5/2014
  * fixed AtlasProdJobBroker for pre-assigned high prio tasks  

* 10/3/2014
  * fixed JG to propagate middle-name to merged files 

* 9/26/2014
  * improved getTasksToBeFinished_JEDI to avoid row lock contention
  * changed AtlasAnalPostProcessor to skip freezing trn
  * fixed infinite output/log dataset making

* 9/25/2014
  * setting minRamCount and maxCpuCount to pmerge jobs

* 9/24/2014
  * improved AtlasAnalysisPP for multiple threads processing the same task
  * increased timeout of Interaction to 600

* 9/23/2014
  * changed for prod tasks to avalanche when 5 scout jobs succeeded 
  * introduced scoutSuccessRate
  * fixed AtlasProdJobBroker for noInput
  * capped maxDiskCount until maxwdir is used everywhere
  * added capability to reset resource requirements
  * added split rule for merge step

* 9/22/2014
  * removed the hint from getOutputFiles_JEDI
  * added protection for non exiting cliParams to AtlasAnalPP

* 9/16/2014
  * added umNameAtEnd

* 9/12/2014
  * cleanup for reqID
  * added a hint to getOutputFiles_JEDI
  * reduced fetch size in getOutputFiles_JEDI

* 9/11/2014
  * added protection to getScoutData against widely distributed input sizes
  * fixed AtlasAnalJobBroker to take effective input size into account for walltime

* 9/10/2014
  * fixed undefined variables in JediDBProxy
  * added addNthFieldToLFN

* 9/8/2014
  * fixed file lookup for container
  * introduced trn_unknown
  * added protection against changed comm_parameter

* 9/7/2014
  * removing duplicated panda.um

* 9/3/2014
  * changed AtlasDDM to use listDatasets2
  * changed TR to set proper status/error when registration is failed

* 9/2/2014
  * changed AtlasDDM to work with dq2 and rucio
  * updating startTime at submitssion or retry
  * changed pending killer to take startTime into account

* 8/29/2014
  * fixed getAnalSitesWithData
  * added fixedSandbox
  * setting 0 for SKIPEVENTS when unspecified
  * resetting cloud and site

* 8/27/2014
  * fixed retryTask for running inputs
  * not to overwite useScout when incexec
  * disabled file lookup for merge jobs
  * keeping trn_log

* 8/26/2014
  * fixed JG to bunch of datasets for each task in a single cycle
  * introduced max num of datasets in getTasksTobeProcessed 
  * replacing LS in splitRule
  * fixed retryTask

* 8/25/2014
  * fixed JG for merging with only logs

* 8/24/2014
  * reusing lib.tgz when DDM endpoints are shared
  * resetting nFiles and nEvents when incexec
  * using firstSubmission

* 8/22/2014
  * fixed TC to delete only existing params for incexec

* 8/21/2014
  * adding All to email notification when all input succeeded
  * updating nFiles of output/log/lib datasets when the task is done

* 8/20/2014
  * added protection against large --nGBPerJob
  * fixed JG to use MB in maxDiskCount
  * fixed JG not to include input size in maxDiskCount for remoteIO sites
  * fixed retryTask for task with multiple inputs

* 8/19/2014
  * reduced cutoff to 15 for selection of WAN access sites
  * updating task when incexec

* 8/18/2014
  * introduced fairshare in tobeProcessed
  * allowing retry for maxAttempt<=attemptNr
  * fixed JG to register all output datasets 

* 8/17/2014
  * fixed PP to check only master inputs
  * sending task view in notification
  * setting retry in history
  * added safety mergin in workDiskCoung
  * fixed getScoutData
  * fixed AtlasAnalJB to correctly use scout data

* 8/16/2014
  * locking task in JG
  * fixed for xmlConfig

* 8/15/2014
  * fixed for loadXML
  * fixed for retry
  * fixed incexec to use comm_parameters
  * changed AtlasProdPP to freeze output and log datasets
  * decreased limit for instance renewal in Interaction

* 8/14/2014
  * changed to allow reactivation of aborted tasks 
  * changed some method to avoid row contention
  * fixed AtlasDDM for sites which share the same DDM endpoints 

* 8/13/2014
  * removed delay after scouts
  * fixed AtlasAnalJB for nThrottle
  * changed to lock tasks before prepare or process
  * added more debug info to AtlasAnalJB
  * removing currentPrrio limit when counting jobs in AtlasAnalJB

* 8/12/2014
  * fixed file rescue for pseudo input
  * improved performance of getTasksToBeProcessed
  * fixed TaskRefinerBase for GRL
  * fixed file rescue for orphaned merging
  * added lock against multiple processing
  * using tobroken in JG

* 8/11/2014
  * fixed AtlasAnalJB for remote access sites

* 8/6/2014
  * fixed JG for ES jobs to have temporary log file names

* 8/5/2014
  * added campaign info
  * skipping some checks in AtlasProdJobBrokerage when site is preassigned

* 8/1/2014
  * introduced waitInput for Event Picking
  * fixed scouts to take nEventsPerJob into account

* 7/30/2014
  * added a protection against multiple registration of the same task

* 7/29/2014
  * modified scout data calculation to take coreCount into account

* 7/25/2014
  * added more patterns for hospital queues

* 7/24/2014
  * setting superStatus

* 7/23/2014
  * added retryChildTasks_JEDI
  * added stateChangeTime
  * setting maxAttempt for secondary files as well

* 7/22/2014
  * removed US constraint from network-aware brokerage

* 7/19/2014
  * added a protection to CF to set ready to task when only secondary is checked
  * fixed maxDiskCout

* 7/10/2014
  * fixed retryTask
  * changed status=timeout to aborted
  * fixed calculation for effective walltime
  * removed <> from http logging messages
  * added refresh() to *Base
  * added http logging
  * changed to propagate aborting to DEFT
  * added caching for SiteMapper

* 7/8/2014
  * changed PP to kill child tasks when parent is aborted/failed/broken

* 7/7/2014
  * changed CF to remove lost files

* 7/3/2014
  * added consistency check for input datasets

* 7/2/2014
  * implemented to process dependent tasks before parent is completed

* 6/30/2014
  * added vo info to FQAN list when submtting jobs
  * using basename as LFN for useListPFN
  * added ticketID and ticketSystemType
  * fixed Booster in AtlasProdWatchDog

* 6/27/2014
  * fixed TaskRefiner to properly tasks when parent task failed to complete

* 6/26/2014
  * changed AtlasProdJobBroker to work with old fairsharepolicy format

* 6/25/2014
  * added support for multiple consumers

* 6/20/2014
  * fixed for pre-processing

* 6/11/2014
  * added templates for cron and logrotate
  * modified email notification to use BigPandaMon and JIRA

* 6/1/2014
  * added eventService to JEDI_Tasks

* 5/30/2014
  * added plugins for general purpose
  * fixed job_retry_history

* 5/27/2014
  * added multiple attempts to ctl script

* 5/26/2014
  * changed error level of import failure in FactoryBase

* 5/21/2014
  * changed ATLAS anal brokerage to use CVMFS check for nightlies
  * added addEventServiceTask.py

* 5/13/2014
  * added reassignTask

* 5/8/2014
  * added a build number to version name

* 5/6/2014
  * fixed for t_task.total_events

* 4/29/2014
  * use startEvent and endEvent of files for event service

* 4/23/2014
  * changed CC to allow nEventsPerInputFile

* 4/21/2014
  * initializing nevents, total_req_jobs and metadata

* 4/17/2014
  * introduced toabort
  * fixed pp to set failed to datasets for failed tasks

* 4/14/2014
  * fixed for task broker

* 4/9/2014
  * added AtlasAnalWatchDog
  * improved log in AtlasProdJobBroker
  * added randomSleep to all workers

* 4/7/2014
  * fixed RucioFC lookup
  * fixed to set higher priorities to pmerge
  * fixed to use proper currentPriority in AtlasAnalJobBroker
  * changed to reuse lib.tgz in analysis
  * fixed splitter for merging
  * changed AtlasAnalJobBroker to take throttled jobs into account

* 3/25/2014
  * fixed to avoid duplicated usage of premerged datasets
  * fixed to set status=scouting even if scout jobs run on multiple datasets

* 3/20/2014
  * changed xrd to fax

* 3/19/2014
  * chaned to set t_task.start_time

* 3/15/2014
  * chaned to propagate aborted to DEFT

* 3/14/2014
  * changed to set parent_tid=taskid when no parent
  * changed the initial task status to waiting from submit
  * chaned to propagate more task status to DEFT
  * reset splitRule when waiting for parent

* 3/12/2014
  * fixed notification for merge

* 3/10/2014
  * improved ATLAS job brokerage for --site=A,B,C,AUTO
  * added a mechanism for task chain
  * fixed JEDI email notification to show process and dataset summary

* 3/7/2014
  * fixed wrong errorDiaglog="No build jobs succeeded" for group anal

* 3/4/2014
  * added protection to TR against errors happen before impl or taskSpec is generated
  * changed finished/partial to done/finished

* 3/3/2014
  * changed watchdog to allow vo/label specific acction
  * enabled FAX

* 2/20/2014
  * changed ATLAS job brokerage to allow jobs go to brokeroff when preassigned 
  * removed disk size check for directIO sites from ATLAS job brokerage
  * enabled rucio endpoint

* 2/17/2014
  * fixed disk threshold for anal brokerage
  * fixed for transHome=None

* 2/13/2014
  * changed for new columns in t_task

* 2/11/2014
  * ignoring panda internal datasets
  * added protection for unknown dataset
  * support for wild-card in input dataset/container names

* 2/7/2014
  * fixed getScoutData to calculate data per MB

* 2/6/2014
  * fixed AtlasJB to use LOCALGROUPDISK

* 2/4/2014
  * adding attemptNr to output/log LFN
  * fixed for log+ObjectStore

* 2/3/2014
  * implemented event service functions
  * fixed ATLAS pp to delete transient or empty datasets
  * fixed ATLAS job brokerge for nightlies 

* 1/29/2014
  * added protection against RUCIOTEST
  * making new lib when previous attempt failed at the same site

* 1/22/2014
  * added pfnList
  * fixed disableAutoRetry to disable JEDI retry too
  * enabled recordRetryHistory_JEDI

* 1/19/2014
  * added disableAutoRetry

* 1/17/2014
  * added support for loadXML
  * using Job_Retry_History

* 1/15/2014
  * added safety merging to the output offset
  * fixed to set scope for lib

* 1/14/2014
  * fixed getScoutData to use lib size as work dir size
  * fixed InputChunk to use the output offset

* 1/13/2014
  * not chaning task status by merging
  * closing libDS
  * fixed to send email when no scouts succeeded
  * fixed to use unique name for lib.tgz
 
* 1/9/2014
  * fixed JG for cache

* 12/20/2013
  * fixed for multiple input streams

* 12/19/2013
  * added --noEmail
  * incexec with dsForXYZ

* 12/17/2013
  * changed to work with T_TASK
  
* 12/13/2013
  * added include/exclude in CF
  * set containerName when expand

* 12/12/2013
  * fixed for input container
  * fixed for DBR caching
  * added protection against task duplication
  * fixed for retry

* 12/11/2013
  * fixed AnalRefiner for DBR
  * fixed AtlasAnalJobBroker for data locality
  * fixed makeJobParameters for merging

* 12/10/2013
  * added pandamon URLs to AtlasAnalPostProcessor
  * fixed AtlasAnalJobBrokerage for remote access
  * implemented TaskSetupper for ATLAS analysis

* 12/2/2013 
  * set task status to broken when no scout jobs succeeded
  * fixed retry for scouting

* 12/1/2013 
  * increased the limit on the number of files 

* 11/25/2013 
  * refactored PostProcessor
  * added email notification

* 11/22/2013 
  * removed direct LFC dependence from AtlasDDMClient

* 11/21/2013 
  * setting transferType and sourceSite when remote access is used
  * implemented eventPicking and GRL

* 11/18/2013
  * changed getBestNNetworkSites_JEDI

* 11/15/2013
  * implemented task retry and incexec
  * added a protection to CF for too many input

* 11/9/2013
  * fixed getBestNNetworkSites_JEDI for table merging

* 11/7/2013
  * implemented merging

* 10/29/2013
  * added containerExpansion

* 10/20/2013
  * added nEventsPerRange
  * fixed CF to take offset for input into account

* 10/11/2013
  * added check if DBR is well replicated
  * fixed for /E
  * set jobsetID
  * added LATEST DBR lookup

* 10/10/2013
  * improved JG for pathena

* 10/4/2013
  * implemented getSatelliteSites
 
* 9/26/2013
  * added preprocessing function

* 9/9/2013
  * added analysis functions
  * tagged 0.0.1

* 9/4/2013
  * fixed first event to start from 1
  * fixed random seed

* 8/31/2013
  * fixed prepareTask

* 8/29/2013
  * fixed Watchdoc
  * uploading logs when setupper is failed

* 8/28/2013
  * added support for log merge

* 8/19/2013
  * added timeout for pending

* 8/12/2013
  * fixed CF for nFilesPerJob + scouting 

* 8/9/2013
  * fixed splitter to respect nFilesPerJob even in scouting 

* 8/8/2013
  * added a capability to make build jobs

* 7/31/2013
  * added fullSimulation for job splitter 
  * removed defualt walltime in AtlasProdTaskRefiner
  * improved AtlasJobBroker to upload log snippet
  * changed AtlasJobBroker to take max/minmemory and mintime into account

* 7/29/2013
  * added an error message to CF when files are missing
  * changed FR to take oldAccompanyDatasetNames into account

* 7/22/2013
  * added TaskSetupper

* 7/19/2013
  * added TG for FileRecovery

* 7/16/2013
  * fixed JG for FileRecovery

* 7/11/2013
  * fixed getScoutJobData

* 7/10/2013
  * added FileRecovery

* 7/3/2013
  * alpha version 

* 6/28/2013
  * added missing file remover to AtlasProdTaskBroker 

* 6/25/2013
  * fixed JG to take maxNumJob into account
  * added reniceJEDI

* 6/21/2013
  * fixed JobThrottler for maxNumJob and minPriority

* 6/20/2013
  * added jediTaskID to the WHERE clause for all UPDATE

* 6/19/2013
  * improved CF to directly avalanche when skipScout=True
  * improved JG to reset unused files
  * improved some queries to use AUX table
  * fixed AtlasDDMClient to ignore duplicated files with different attempt number
  * added support for non-input tasks

* 6/14/2013
  * fixed to support secondary datasets which has non-integer ratio to master 

* 6/11/2013
  * fixed WorkQueueMapper

* 6/6/2013
  * added ZombiCleaner

* 6/3/2013
  * changed schema names configurable
  * fixed CF for broken datasets

* 6/1/2013
  * added TaskCommando

* 5/27/2013
  * added TaskBroker

* 5/22/2013
  * renamed taskID to jediTaskID

* 5/10/2013
  * first version