forked from DIRACGrid/DIRAC
-
Notifications
You must be signed in to change notification settings - Fork 0
/
release.notes
8120 lines (5907 loc) · 283 KB
/
release.notes
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
[v7r0p6]
*Core
NEW: Add environment.yml file for preparing an environment with conda
CHANGE: Use subprocess32 if possible for significantly better performance
FIX: TypeLoader should not be a singleton as it has a cache
*FrameworkSystem
FIX: Avoid leaking std(in|out|err) file handles when running runsvdir
*WorkloadManagementSystem
NEW: To activate the pilot logging mechanism the following option must be present in CS: Pilot/PilotLogging with a value set to "true", "yes" or "y".
*Tests
NEW: CI for unit/lint/integration testing with GitHub actions
NEW: Start using shell linting of the test scripts
FIX: Exit quickly if the installation fails to make it easier to find issues
*Docs
NEW: Document how to run integration tests in docker
CHANGE: The options in the command reference sections: 'ignore' and 'scripts', are replaced by 'exclude' and 'manual' respectively. The entries in 'exclude' will now reject scripts that are otherwise picked up by the 'patterns'. Entries in 'manual' will be added to the indexFile, but their rst file has to be provided by hand. Fixes #4345
[v7r0p5]
FIX: changes from v6r22p15 included
[v7r0p4]
*Core
FIX: (#4337) remove unexisting import from dirac-info
*WMS
CHANGE: (#4317) dealing with min and max number of processors from the job
*docs
CHANGE: (#4317) added WMS admin doc (job state machine)
[v7r0p3]
*Core
FIX: (#4298) install_site.sh : set UpdatePilotCStoJSONFile=False for initial installations
*WMS
FIX: (#4294) JobState - restored logic of having database clients at the class level
*TS
CHANGE: (#4308) Rename Operations option Productions/ProductionFilesMaxResetCounter to Transformations/FilesMaxResetCounter
*Resources, WMS, tests
CHANGE: (#4295) save the number of processors as jobLimits in pilot cfg
*tests
CHANGE: (#4291) removed pilot2 tests
*docs
NEW: (#4309) Added how to create a dedicated dirac file catalog
[v7r0p2]
*Accounting
FIX: (#4290) Allow longer user and site names.
*WMS
NEW: (#4287) dirac-wms-get-wn and dirac-wms-pilot-job-info scripts (ported from LHCbDIRAC)
*docs
CHANGE: (#4266) the AdministratorGuide has been restructured
[v7r0p1]
FIX: Removing some deprecated codes
[v7r0]
FIX: (#3962) use print function instead of print statement
FIX: (#3962) remove the usage of the long suffix to distinguish between long and int
FIX: (#3962) convert octal literals to new syntax
NEW: (#3962) Add pylint test to check python 3 compatibility
*Configuration
CHANGE: (#4249) The flag UpdatePilotCStoJSONFile has been moved to Operations/[]/Pilot section
NEW: (#4255) Resources - added getStorageElements() method
*Framework
CHANGE: (#4180) Removed local MySQL handling (DIRACOS won't have install it)
CHANGE: (#4180) Removed setup of old portal
*Accounting
CHANGE: (Multi)AccountingDB - Grouping Type and Object loader together with the MonitoringSystem ones.
*WorkloadManagementSystem
NEW: Add JobElasticDB.py with getJobParameters and setJobParameter methods to work with ElasticSearch (ES) backend.
NEW: Add gJobElasticDB variable and indicates the activation of ES backend.
CHANGE: Modify export_getJobParameter(s) to report values from ES if available.
CHANGE: (#3748) Reduced (removed, in fact) interactions of Optimizers with JobParameters, using only OptimizerParameters
NEW: (#3760) Add Client/JobStateUpdateClient.py
NEW: (#3760) Add Client/JobManagerClient.py
CHANGE: (#3760) Use the above Client classes instead of invoking RPCClient()
NEW: Added ES backend to WMSAdministratorHandler to get JobParameters.
NEW: Added separate MySQL table for JobsStatus in JobDB, and modified code accordingly.
CHANGE: ElasticJobDB.py: Modify setJobParameter method to register JobAttributes like Owner, Proxy, JobGroup etc.
CHANGE: ElasticJobDB.py: added getJobParametersAndAttributes method to retrieve both parameters and attributes for a given JobID.
CHANGE: Pilot Watchdog - using python UNIX services for some watchdog calls
CHANGE: (#3890) JobState always connects to DBs directly
CHANGE: (#3890) remove JobStateSync service
CHANGE: (#3873) Watchdog: use Profiler instead of ProcessMonitor
NEW: (#4163) SiteDirectors send by default Pilot3 (flag for pilot2 is still kept)
FIX: (#4163) removed unicode_literals (messes up things with DEncode)
NEW: (#4224) PilotWrapper can get the Pilot files from a list of locations
CHANGE: (#4224) PilotWrapper is compatible with several python versions
NEW: (#4260) New dirac-wms-match command to test a given job for matching computing resources
NEW: (#4260) QueueUtilities - utilities relevant to queues manipulation
FIX: (#4265) urllib.urlopen() call with and without the 'context' parameter in Utilities/PilotWrapper.py
according to the version of urllib instead of the python version
CHANGE: (#4244) Modified flag for using the JobParameters on ElasticSearch database
*Core
NEW: (#3744) Add update method to the ElasticSearchDB.py to update or if not available create the values
sent from the setJobParameter function. Uses update_by_query and index ES APIs.
CHANGE: (#3744) generateFullIndexName() method made static under the ElasticSearchDB class.
CHANGE: (#3744) removed unused/outdated stuff from Distribution module
FIX: check for empty /etc/grid-security/certificates dir
NEW: (#3842) Add createClient decorator to add wrapper for all export_ functions automatically
NEW: (#3678) dirac-install can install a non released code directly from the git repository
FIX: (#3931) AuthManager - modified to work with the case of unregistered DN in credDict
FIX: (#4182) Add CountryName OID and be more permissive in the String type for decoding VOMSExtensions
FIX: (#4211) change orders of @deprecated and @classmethod decorators in X509Chain
NEW: (#4229) dirac-install defines XRD_RUNFORKHANDLER environment variable
*TransformationSystem
NEW: (#4124) InputDataQuery and OutputDataQuery parameters can be set for Transformations before
they are added to the transformation system. Solves #4071
CHANGE: (#4238) TransformationDB.getTransformationMetaQuery returns ENOENT if no meta query exists
FIX: (#4238) InputDataAgent: silence warnings about transformations not having a input data query
*ProductionManagement
NEW: (#3703) ProductionManagement system is introduced
*Interfaces
CHANGE: (#4163) removed deprecated methods from Dirac.py
*Resources
FIX: (#4229) add proxy location as a virtual user for xroot urls
FIX: (#4234) future import in executeBatch script that prevented the pilot deployment on SSH CE, fixes #4233
FIX: (#4231) SE __executeMethod: only pass protocols parameter to getTransportURL
NEW: (#4255) New dirac-resource-info command to display computing and storage resources available
to a given VO
*ResourceStatusSystem
CHANGE: (#4177) use the ObjectLoader for Clients, in PEP
*Monitoring
FIX: MonitoringReporter - make processResords() method thread safe, fixes #4193
*tests
NEW: Add ChangeESFlag.py script to modify useES flag in dirac.cfg. To be integrated with Jenkins code.
CHANGE: (#3760) Use the above Client classes instead of invoking RPCClient()
NEW: (#3744) Added performance tests for ES and MySQL for WMS DB backends
NEW: (#3744) Added miniInstallDIRAC function for Jenkins jobs
FIX: (#4177) restored test of JobsMonitor().JobParameters
NEW: (#4224) added a test for PilotWrapper
NEW: (#4244) integration test for JobParameters on ElasticSearch database
*Docs
FIX: Better dirac.cfg example configuration for web
NEW: (#3744) Add WMS documentation in DeveloperGuide/Systems
NEW: Added script (docs/Tools/UpdateDiracCFG.py) to collate the ConfigTemplate.cfg
files into the main dirac.cfg file
CHANGE: (#4110) updated basic tutorial for CC7 instead of SLC6.
NEW: (#4170) added Production system documentation
CHANGE: (#4224) Pilot 3 is the default
NEW: (#4244) Added a few notes on using the JobParameters on ElasticSearch database
[v6r22p16]
*Core
CHANGE: remove many calls to gLogger.debug() of low level DB modules (in particular MySQL.py) that may slowdown services
*FrameworkSystem
FIX: PRIVATE_LIMITED_DELEGATION role can download its own proxies
*ResourceStatusSystem
FIX: PublisherHandler: Convert set to list, because set can not be serialized using DEncode.
[v6r22p15]
*Resources
NEW: GFAL2_StorageBase: Disable GRIDFTP SESSION_REUSE by default. It can be enabled via
an environment variable export DIRAC_GFAL_GRIDFTP_SESSION_REUSE=True. This export
should be added in server bashrc files.
[v6r22p14]
*Framework
FIX: (#4318) NotificationHandler: fix name of initializeHandler function so the
handler is properly initialized and the periodicTasks are created
CHANGE: (#4325) InstalledComponentsDB.addInstalledComponents: Instead of creating a new
Host entry if only the CPU model changed, update the CPU field in the DBCHANGE:
InstalledComponentsDB.addInstalledComponents: Instead of creating a new Host entry
if only the CPU model changed, update the CPU field in the DB
*WMS
FIX: (#4329) Fix exception for PilotManager or PilotStatusAgent: itertems -> iteritems
*DMS
FIX: (#4327) avoid introducing inconsistencies in the DB due to the FTS inconsistencies
*Resources
CHANGE: (#4313) increase the SE logging level for plugin errors
*TS
CHANGE: (#4315) DataRecoveryAgent: Tweak information printout for email formatting;
clarify use of default values; reduce logging verbosity
[v6r22p13]
*Resources
FIX: (#4299) CREAMComputingElement - possibility to defined CEQueueName to be used in the pilot submission command
CHANGE: (#4297) SingularityCE: Delete workDir at end of job by default.
FIX: (#4297) SingularityCE: Always stop proxy renewal thread at end of job.
*TS
NEW: (#4301) DataRecoveryAgent to perform and apply consistency checks for transformations
NEW: (#4301) dirac-transformation-recover-data : script to manually run consistency checks on
individual transformations, for debugging and testing of the DataRecoveryAgent
[v6r22p12]
FIX: fixes from v6r21p16
[v6r22p11]
*Interfaces
FIX: (#4277) Dirac.py - fix error handling in getJobParameters()
[v6r22p10]
*Resources
FIX: (#4271) StorageElement - loadObject of occupancyPlugin is called with the path to OccupancyPlugins
[v6r22p9]
*Core
FIX: (#4269) Workflow - revert changes introduced in #4253
[v6r22p8]
*WMS
FIX: (#4256) handle empty results of getJobParameter in StalledJobAgent
*Resources
NEW: (#4223) Storage - occupancy plugin for WLCG standard accounting JSON
FIX: (#4259) SingularityComputingElement - ensure lcgBundle is installed in container if LCGBundleVersion is set.
[v6r22p7]
*Core
FIX: (#4253) Workflow - accept unicode file path to workflow XML
*WMS
NEW: (#4205) Statistics of Pilot submission is sent to Accounting System by SiteDirector.
FIX: (#4250) use host credentials to query the SandboxMetadataDB for async removal
*Resources
CHANGE: (#4242) DFC SEManagerDB: only acquire the lock if the cache needs to be modified
FIX: (#4251) preamble attribute was missing in SSHBatchComputingElement causing an error
in _submitJobHost()
*DMS
FIX: (#4248) SEManager correct order of calls at init
[v6r22p6]
*Core
CHANGE: (#4203) A VO with diracos extension, must be able to install the main diracos from DIRAC repository.
*TS
CHANGE: (#4218) do not set Job type 'hospital'
[v6r22p5]
*WMS
CHANGE: (#4217) Sandbox: Adding OwnerDN and OwnerGroup for DelayedExternalDeletion
*DMS
CHANGE: (#4202) relax permissions on getLFNForGUID
*Resources
FIX: (#4200) re-allow the use of the gsiftp cache in the SE
CHANGE: (#4206) Storage - improved treatment for the case without SpaceToken in SpaceOccupancy
[v6r22p4]
*Core
NEW: (#4181) Allow to install an extension of DIRACOS
*Docs
FIX: (#4187) Create a TransformationAgent Plugin -- more concrete
descriptions for adding a plugin to AllowedPlugins
[v6r22p3]
*WMS
CHANGE: (#4175) added function to get the number of Processors, using it in dirac-wms-get-wn-parameters (invoked by the pilots)
CHANGE: (#4175) changed port of PilotManager from 9129 to 9171
*docs
CHANGE: (#4174) update WebApp administrator docs
CHANGE: (#4175) removed mentions of MPIService
[v6r22p2]
*Core
FIX: (#4165) $Id$ keyword must be replaced only at the beginning of the file
*RSS
FIX: (#4169) PublisherHandler fix (UnboundLocalError)
*Docs
NEW: (#4167) /Operations/DataManagement - Added a link to the documentation on "Multi Protocol"
[v6r22p1]
*WMS
NEW: (#4147) added option for specifying a LocalCEType (which by default is "InProcess")
*Interfaces
NEW: (#4146) added setNumberOfProcessors method to Job() API
*Resources
NEW: (#4159) add dav(s) in the protocol lists of GFAL2_HTTPS
*tests
NEW: (#4154) Using variable for location of INSTALL_CFG_FILE (useful for extensions)
[v6r22]
*WorkloadManagementSystem
NEW: (#4045) PilotsHandler service (to interact with PilotAgentsDB)
CHANGE: (#4049) Pilot wrapper for pilot3: download files at runtime
CHANGE: (#4119) removed last bit looking into /Registry/VOMS/ section
CHANGE: (#4119) VOMS2CSAgent: mailFrom option is invalid, use MailFrom instead
FIX: (#4074) fixed PilotManager service name in ConfigTemplate.cfg
FIX: (#4100) use CAs to upload the pilot files to a dedicated web server using requests
FIX: (#4106) fixes for logging messages for Matcher
FIX: (#4106) fixes for logging messages for Optimizers
NEW: (#4136) added DIRACOS option (for SiteDirectors)
*ConfigurationSystem
NEW: (#4053) VOMS2CSSynchronizer/VOMS2CSAgent - enable automatic synchronization of the
Suspended user status with the VOMS database
CHANGE: (#4113) VOMS2CSSynchronizer: considering the case when no email is provided by VOMS
*Core
NEW: (#4053) AuthManager - interpret the Suspended user status considering suspended
users as anonymous visitors
CHANGE: (#4053) The use of CS.py module is replaced by the use of Registry.py and CSGlobals.py
*Interfaces
CHANGE: (#4098) removed dirac-admin-get-site-protocols.py as it could give potentially wrong results (use dirac-dms-protocol-matrix instead)
*Resources
NEW: (#4142) enable to get SE occupancy from plugin
NEW: (#4142) add occupancy plugin to retrieve the info from BDII
CHANGE: (#4142) GFAL2_SRM2Storage.getOccupancy() calls parent if SpaceToken is not given
*ResourceStatusSystem
CHANGE: clients: using DIRAC.Core.Base.Client as base
CHANGE: (#4098) SiteInspectorAgent runs only on sites with tokenOwner="rs_svc"
CHANGE: (#4098) remove SRM dependencies
CHANGE: (#4136) downtimeCommand will use the GOCServiceType only for SRM SEs
FIX: (#4139) only take the first endpoint for the SpaceOccupancy
*DataManagementSystem
CHANGE: (#4136) Moved methods from ResourceStatusSystem/CSHelpers to DMSHelpers
CHANGE: (#4138) FTS3 is now the default
*docs
NEW: (#4099) Instructions about setting up the DIRAC web server for pilot3
CHANGE: (#4119) added note on MultiProcessor jobs preparation
*test
FIX: (#4119) Not lhcb but dteam (for DIRAC certification)
FIX: (#4139) client_dms.sh not lhcb specific
CHANGE:(#4140) adapt transformation certification tests to dteam VO
[v6r21p16]
*Resources
FIX: (#4292) SSHComputingElement - define X509_USER_PROXY in case of gsissh access
*WMS
FIX: (#4292) SiteDirector - do not use keyword arguments when making setPilotStatus call
[v6r21p15]
*WMS
CHANGE: (#4214) Add an argument to the constructor of SandboxStoreClient for using in scripts
that cannot use the DB directly
*DMS
NEW: (#4171) ArchiveFiles Request Operation: to create a tarball out of a list of LFNs
NEW: (#4171) CheckMigration Request Operation to hold the request progress until the attached
LFNs are migrated to tape
NEW: (#4171) FCOnlyStorage StorageElement plugin to register LFNs without physical replica to
conserve LFN metadata when they are archived, for example
NEW: (#4171) dirac-dms-create-archive-request command to create a request to archive a list of
LFNs and remove their physical copies
NEW: (#4171) dirac-dms-create-moving-request command to move LFNs from a to b with optional
"CheckMigration" step. as it uses the ReplicateAndRegister operation, transfer via
FTS is also possible
FIX: (#4171) FileCatalogClient: add "addFileAncestors" to list of "write functions"
[v6r21p14]
*DMS
FIX: (#4192) dirac-dms-clean-directory correct usage message for list of arguments
FIX: (#4192) dirac-dms-clean-directory now properly prints error messages
FIX: (#4192) dirac-dms-clean-directory will now also clean empty directories
FIX: (#4192) FileCatalog.DirectoryMetadata: prevent error when removeMetadataDirectory is
called on empty list of directories, triggered when calling removeDirectory
on non-existing directory
FIX: (#4192) FileCatalog.DirectoryTreeBase: prevent maximum recursion depth exception
when calling remove directory with illegal path
*Resources
CHANGE: (#4191) Storages: catch specific DPM/Globus error code when creating existing directory
[v6r21p13]
*RSS
FIX: (#4173) only use the hostname of FTS servers in the RSS commands
[v6r21p12]
*WMS
FIX: (#4155) JobDB.getAttributesForJobList: Return S_ERROR if unknown attributes are requested.
Instead of potentially returning garbage a clear error message is returned.
[v6r21p11]
*Transformation
FIX: (#4144) fixed a logic bug in counting numberOfTasks in MCExtension which is expected
to limit the total number of tasks for MC transformations
*Accounting
FIX: (#4151) In AccountingDB.insertRecordThroughQueue fix bad dictionary key "0K"
[v6r21p10]
*Core
FIX: (#4133) dirac-install: correct location for ARC plugins with DIRACOS
*WMS
CHANGE: (#4136) JobStateUpdateHandler - restoring the job status to Running after hearbeat
*Docs
NEW: (#4134) Added /Operations/DataManagement parameters for protocols
[v6r21p9]
*Core
FIX: (#4130) correct symlinks in dirac-deploy
[v6r21p8]
*WMS
CHANGE: (#4111) Better logging in JobWrapper
CHANGE: (#4114) JobScheduling - allow both Tag and Tags option in the job JDL
*DMS
FIX: (#4101) FileManagerBase - use returned ID:LFN dict instead of the LFN list. Fixes the bug in
getReplicasByMetadata reported in #4058
*TransformationSystem
FIX: (#4112) TaskManager.py - testing if the request ID is correct was not done properly, test the numerical value
[v6r21p7]
*Core
FIX: (#4076) A certain module like WebAppDIRAC must be checked out from the code repository once.
*Resources
FIX: (#4078) Fix the exception when StorageElement objects are created with a list of plugins
*SMS
NEW: (#4086) StageMonitorAgent: new option StoragePlugins to limit the protocols used to contact storagelements for staged files.
*WMS
FIX: (#4093) better logging from services
*TS
CHANGE: (#4095) ConfigTemplate for RequestTaskAgent now contains all options
CHANGE: (#4096) the Broadcast TransformationPlugin no longer requires a SourceSE to be set. If it is set, the behaviour is unchanged
CHANGE: (#4096) dirac-transformation-replication: change default pluging back to Broadcast (reverts #4066)
*Docs:
CHANGE: (#4095) AdministratorGuide install TS tutorial: added options MonitorFiles and MonitorTasks for TaskAgents
[v6r21p6]
*CS
FIX: (#4064) Fix exception when calling dirac-admin-add-shifter with already existing values
*Core
NEW: (#4065) getIndexInList utility in List.py
*Resources
NEW: (#4065) add a SpaceReservation concept to storages
NEW: (#4065) add a getEndpoint method to StorageBase
*RSS
CHANGE: (#4065) CSHelpers.getStorageElementEndpoint returns the endpoint or non srm protocol
CHANGE: (#4065) add the SpaceReservation to the FreeDiskSpaceCommand result
*TS
FIX: (#4066) The dirac-transformation-replication script will now create valid transformations
given only the required arguments. Instead of the 'Broadcast' plugin, the 'Standard' plugin
is created if not SourceSE is given. If a value for the plugin argument is given, that will
be used.
*docs
CHANGE: (#4069) DIRAC installation procedure is updated taking account DIRACOS
CHANGE: (#4094) Pilots3 options: moved to /Operation/Pilot section
[v6r21p5]
*Core
NEW: (#4046) allow install_site to install DIRACOS
FIX: (#4047) dirac-deploy-scripts uses correct regex to find scripts
NEW: (#4047) dirac-deploy-scripts can use symplink instead of wrapper
CHANGE: (#4051) use debug level for logs in the ProcessPool
*RequestManagementSystem
CHANGE: (#4051) split log messages
*ResourceStatusSystem
FIX: (#4050) fix reporting from EmailAgent
CHANGE: (#4051) split log messages in static and dynamic parts
*Docs
CHANGE: (#4034) Add magic runs to setup DIRAC in example scripts, so they work out of the box.
NEW: (#4046) add a tuto for setting up a basic installation
NEW: (#4046) add a tuto for setting up two Dirac SEs
NEW: (#4046) add a tuto for setting up the DFC
NEW: (#4046) add a tuto for managing identities
NEW: (#4046) add a tuto for setting up the RMS
NEW: (#4046) add a tuto for doing DMS with TS
*ConfigurationSystem
CHANGE: (#4044) dirac-configure: forcing update (override, in fact) of CS list
*WorkloadManagementSystem
FIX: (#4052) SiteDirector - restore the logic of limiting the number of pilots to submit due to the
WaitingToRunningRatio option
FIX: (#4052) Matcher - if a pilot presents OwnerGroup parameter in its description, this is interpreted
as a pilot requirement to jobs and should not be overriden.
CHANGE: (#4027) Improve scalability of HTCondorCE jobs
*Accounting
CHANGE: (#4033) accounting clients use DIRAC.Core.Base.Client as base
*DataManagementSystem
FIX: (#4042) add exit handler for stored procedure
FIX: (#4048) correct the header of the CSV file generated by dirac-dms-protocol-matrix
*TransformationSystem
FIX: (#4038) TransformationCleaningAgent cancels the Request instead of removing them
*Resources
CHANGE: (#4048) SE: give preference to native plugins when generating third party URLS
[v6r21p4]
WorkloadManagementSystem
CHANGE: (#4008) Modification of utility function PilotCStoJSONSynchronizer. The modification
allows to add information to created json file about the DNs fields of users belonging
to 'lhcb_pilot' group. This information is needed for the second level authorization
used in the Pilot Logger architecture. Also, some basic unit tests are added.
*Docs
CHANGE: (#4028) update instructions to install and setup runit
*TransformationSystem
FIX: (#4022) when a site was requested inside the job workflow description, and BulkSubmission was used, such site was not considered.
*Resources
FIX: (#4006) Resources/MessageQueue: add a dedicated listener ReconnectListener
[v6r21p3]
*Core
FIX: (#4005) getDiracModules is removed, class member is used instead.
FIX: (#4013) Use getCAsLocation in order to avoid non-exist os.environ['X509_CERT_DIR']
*ConfigurationSystem
FIX: (#4004) BDII2CSAgent: fix for CEs with incomplete BDII info
*WorkloadManagementSystem
NEW: (#4016) JobAgent - added possibility to try out several CE descriptions when
getting jobs in one cycle
NEW: (#4016) Matcher - MultiProcessor tag is added to the resource description if appropriate
NEW: (#4016) JobScheduling - MultiProcessor tag is added to the job description if it
specifies multiple processor requirements
FIX: (#4018) JobMonitoring.getJobParameter cast to int
NEW: (#4019) added WMSAdministratorClient module, and using it throughout the code
*Resources/MessageQueue
CHANGE: (#4007) change the way of defining identifier format for MQ resources:
accepted values are 'Topics' or 'Queues'.
*DataManagementSystem
CHANGE: (#4017) DIP handler internally uses bytes instead of MB
NEW: (#4010) add dirac-dms-protocol-matrix script
CHANGE: (#4010) remove dirac-dms-add-files script
*Resources
NEW: (#4016) PoolComputingElement - getDescription returns a list of descriptions
with different requirements to jobs to be matched
CHANGE: (#4017) Standardize sizes returned by StoragePlugins in Bytes
CHANGE: (#4011) MQ: randomzied the broker list when putting message
[v6r21p2]
*Core
CHANGE: (#3992) dirac-install does not define REQUESTS_CA_BUNDLE in the bashrc anymore
NEW: (#3998) dirac-install if DIRACOS already installed and DIRACOS is not requested,
it will force to install it
CHANGE: (#3992) specify the ca location when calling requests
CHANGE: (#3991) MySQL class prints only debug logs
FIX: (#4003) dirac-install - if the DIRACOS version is not given then use the proper
release version
*WorkloadManagementSystem
CHANGE: (#3992) specify the ca location when calling requests
FIX: (#4002) Local protocols are retrieved as a list in InputDataResolution
*Interfaces
FIX: (#4000) Dirac.py - bug fixed: return value of getJobParameters changed that
should be taken into account by the clients
[v6r21p1]
*WorkloadManagementSystem
CHANGE: (#3989) JobDB.py - do not add default SubmitPool parameter to a job description
FIX: (#3989) dirac-admin-get-site-mask - show only sites in Active state
*DataManagementSystem
CHANGE: (#3985) FTS3DB: getActiveJobs, those jobs are now selected that have been monitored the longest time ago. Ensure better cycling through FTS Jobs
FIX: (#3987) check missing file with another string
[v6r21]
*Core
NEW: (#3921) DictCache - allow threadLocal cache
FIX: (#3936) DictCache - Fix exception upon delete
FIX: (#3922) allow Script.py to accommodate specific test calls with pytest
CHANGE: (#3940) dirac-install - instrument to support DiracOS
FIX: (#3945) set DIRACOS environment variable before souring diracosrc
CHANGE: (#3949) Removed unattended dirac-install-client.py
CHANGE: (#3950) File.py - do not follow links when getting files list or size
in directory via getGlobbedFiles and getGlobbedTotalSize
CHANGE: (#3969) Use EOS for installing DIRAC software
*FrameworkSystem
FIX: (#3968) removed the old logging
*ResourceStatusSystem
FIX: (#3921) fix logic of the RSSCache leading in expired keys
*Accounting
CHANGE: (#3933) Change the columns size of the FinalMinorStatus
*WorkloadManagementSystem
CHANGE: (#3923) Clean PYTHONPATH from *rc when installing DIRAC from the pilot
NEW: (#3941) JobDB: getJobParameters work also with list on job IDs
CHANGE: (#3941) JobCleaningAgent queries for job parameters in bulk
CHANGE: (#3941) Optimizers only set optimizers parameters (backported from v7r0)
CHANGE: (#3970) streamlining code in OptimizerModule. Also pep8 formatting (ignore white spaces for reviewing)
FIX: (#3976) fixed Banned Sites matching in TaskQueueDB
FIX: (#3970) when an optimizer agent was instantiating JobDB (via the base class) and the machine
was overloaded, the connection to the DB failed but this was not noticed and the agent was
not working until restarted after max cycles. Now testing JobDB is valid in OptimizerModule
base class and exit if not valid.
*TransformationSystem
CHANGE: (#3946) Remove directory listing from ValidateOutputDataAgent
CHANGE: (#3946) Remove directory listing from TransformationCleaningAgent
FIX: (#3967) TransformationCleaningAgent: don't return error if log directory does not exist
*Interfaces
CHANGE: (#3947) removed old methods going through old RMS
CHANGE: (#3960) Dirac.py - getLFNMetadata returns result for both file and directory LFNs
FIX: (#3966) Dirac: replace the use of deprecated function status by getJobStatus
*DataManagementSystem
FIX: (#3922) Fixes FTS3 duplicate transfers
FIX: (#3982) respect the source limitation when picking source for an FTS transfer
*MonitoringSystem
CHANGE: (#3956) Change the bucket size from week to day.
*Resources
CHANGE: (#3933) When crating a consumer or producer then the error message must be
handled by the caller.
CHANGE: (#3937) MessageQueue log backends is now set to VERBOSE instead of DEBUG
NEW: (#3943) SSHComputingElement - added Preamble option to define a command to be
executed right before the batch system job submission command
NEW: (#3953) Added the possibility to add filters to log backends to refine the
output shown/stored
NEW: (#3953) Resources.LogFilters.ModuleFilter: Filter that allows one to set the
LogLevel for individual modules
NEW: (#3953) Resources.LogFilter.PatternFilter: Filter to select or reject log
output based on words
FIX: (#3959) PoolComputingElement - bug fix: initialize process pool if not yet
done in getCEStatus()
*test
CHANGE: (#3948) integration tests run with unittest now exit with exit code != 0 if failed
*docs
NEW: (#3974) Added HowTo section to the User Guide
[v6r20p28]
*WorkloadManagementSystem
FIX: (#4092) pilotTools - Ensure maxNumberOfProcessors is an int
[v6r20p27]
*WMS
FIX: (#4020) SiteDirector - do not use keywords in addPilotTQReference/setPilotStatus calls
[v6r20p26]
*WorkloadManagementSystem
FIX: (#3932) MutiProcessorSiteDirector: get platform is checkPLatform flag is true
*DataManagementSystem
FIX: (#3928) `FileCatalogClient` now properly forwards function docstrings through
`checkCatalogArguments` decorator, fixes #3927
CHANGE: (#3928) `Resources.Catalog.Utilities`: use functool_wraps in `checkCatalogArguments`
*TransformationSystem
CHANGE: (#3934) make the recursive removal of the log directory explicit in the TransformationCleaningAgent
[v6r20p25]
*Core
FIX: (#3909) DISET - always close the socket even in case of exception
*FrameworkSystem
FIX: (#3913) NotificationHandler - bugfixed: changed SMTPServer to SMTP
FIX: (#3914) add extjs6 support to the web compiler
*docs
NEW: (#3910) Added documentation on MultiProcessor jobs
*WorkloadManagementSystem
FIX: (#3910) TaskQueueDB - fixed strict matching with tags, plus extended the integration test
*DataManagementSystem
CHANGE: (#3917) FTS3: speedup by using subqueries for the Jobs table
*TransformationSystem
CHANGE: (#3916) use SE.isSameSE() method
*Resources
NEW: (#3916) Add isSameSE method to StorageElement which works for all protocols
[v6r20p24]
*WorkloadManagementSystem
FIX: (#3904) SiteDirector fixed case with TQs for 'ANY' site
[v6r20p23]
*TransformationSystem
NEW: (#3903) do not remove archive SEs when looking at closerSE
*CORE
NEW: (#3902) When the environment variable DIRAC_DEPRECATED_FAIL is set to a non-empty value,
the use of deprecated functions will raise a NotImplementError exception
*ConfigurationSystem
FIX: (#3903) ServiceInterface - fix exception when removing dead slave
*FrameworkSystem
FIX: (#3901) NotificationClient - bug fix
[v6r20p22]
*Core
FIX: (#3897) ObjectLoader returns DErrno code
FIX: (#3895) more debug messages in BaseClient
*ResourceStatusSystem
FIX: (#3895) fixed bug in dirac-rss-set-token script
*WorkloadManagementSystem
FIX: (#3897) SiteDirector: using checkPlatform flag everwhere needed
CHANGE: (#3894) Using JobStateUpdateClient instead of RPCClient to it
CHANGE: (#3894) Using JobManagerClient instead of RPCClient to it
[v6r20p20]
*Core
CHANGE: (#3885) Script.parseCommandLine: the called script is not necessarily the first in sys.argv
*ConfigurationSystem
CHANGE: (#3887) /Client/Helpers/Registry.py: Added search dirac user for ID and CA
*MonitoringSystem
FIX: (#3888) mqProducer field in MonitoringReporter can be set to None, and the comparison was broken.
It is fixed. Also some additional checks are added.
*WorkloadManagementSystem
CHANGE: (#3889) removed confusing Job parameter LocalBatchID
CHANGE: (#3854) TQ matching (TaskQueueDB.py): when "ANY" is specified, don't exclude task queues
(fix with "Platforms" matching in mind)
CHANGE: (#3854) SiteDirector: split method getPlatforms, for extension purposes
*DataManagementSystem
FIX: (#3884) restore correct default value for the SEPrefix in the FileCatalogClient
FIX: (#3886) FTS3: remove the hardcoded srm protocol for registration
FIX: (#3886) FTS3: return an empty spacetoken if SRM is not available
*TransformationSystem
CHANGE: (#3891) ReplicationTransformation.createDataTransformation: returns S_OK with the
transformation object when it was successfully added, instead of S_OK(None)
*Resources
NEW: (#3886) SE - return a standard error in case the requested protocol is not available
[v6r20p18]
*DataManagementSystem
CHANGE: (#3882) script for allow/ban SEs now accepting -a/--All switch, for allo status types
*Core
FIX: (#3882) ClassAdLight - fix to avoid returning a list with empty string
*Resources
FIX: (#3882) Add site name configuration for the dirac installation inside singularity CE
*test
FIX: (#3882) fully activating RSS in Jenkins tests
[v6r20p17]
*Core
CHANGE: (#3874) dirac-create-distribution-tarball - add tests directory to the tar file and fix pylint warnings.
FIX: (#3876) Add function "discoverInterfaces" again which is still needed for VMDIRAC
*ConfigurationSystem
CHANGE: (#3875) Resources - allow to pass a list of platforms to getDIRACPlatform()
*WorkloadManagement
CHANGE: (#3867) SandboxStoreClient - Returning file location in output of getOutputSandbox
CHANGE: (#3875) JobDB - allow to define a list of Platforms in a job description JDL
*ResourceStatusSystem
CHANGE: (#3863) deprecated CSHelpers.getSites() function
*Interfaces
NEW: (#3872) Add protocol option to dirac-dms-lfn-accessURL
CHANGE: (#3864) marked deprecated some API functions (perfect replace exists already, as specified)
*Resources
FIX: (#3868) GFAL2_SRM2Storage: only set SPACETOKENDESC when SpaceToken is not an empty string
*Test
CHANGE: (#3863) Enable RSS in Jenkins
*DataManagementSystem
FIX: (#3859) FTS3: resubmit files in status Canceled on the FTS server
NEW: (#3871) FTS submissions can use any third party protocol
NEW: (#3871) Storage plugin for Echo (gsiftp+root)
FIX: (#3871) replace deprecated calls to the gfal2 API
NEW: (#3871) Generic implementation for retrieving space occupancy on storage
*TransformationSystem
FIX: (#3865) fixed submission of parametric jobs with InputData from WorkflowTask
FIX: (#3865) better logging for parametric jobs submission
*StorageManagamentSystem
FIX: (#3868) Fix StageRequestAgent failures for SEs without a SpaceToken
*RequestManagementSystem
FIX: (#3861) tests do not re-use File objects
[v6r20p16]
*WorkloadManagementSystem
CHANGE: (#3850) the platform discovery can be VO-specific.
*Interfaces
CHANGE: (#3856) setParameterSequence always return S_OK/S_ERROR
*TransformationSystem
FIX: (#3856) check for return value on Job interface and handle it
*ResourceStatusSystem
FIX: (#3852) site may not have any SE
[v6r20p15]
*Interface
FIX: (#3843) Fix the sandbox download, returning the inMemory default.
*WorkloadManagementSystem
FIX: (#3845) late creation on RPC in JobMonitoringClient and PilotsLoggingClient
*DataManagementSystem
FIX: (#3839) Update obsolete dirac-rms-show-request command in user message displayed when running dirac-dms-replicate-and-register-request
*FrameworkSystem
FIX: (#3845) added setServer for NotificationClient
*Docs
NEW: (#3847) Added some info on parametric jobs
[v6r20p14]
CHANGE: (#3826) emacs backup file pattern added to .gitignore
*MonitoringSystem
CHANGE: (#3827) The default name of the Message Queue can be changed
*Core
FIX: (#3832) VOMSService.py: better logging and error prevention
*ConfigurationSystem
FIX: (#3837) Corrected configuration location for Pilot 3 files synchronization
*FrameworkSystem
FIX: (#3830) InstalledComponentDB.__filterFields: fix error in "Component History Web App" when filter values are unicode
*Interface
CHANGE: (#3836) Dirac.py API - make the unpacking of downloaded sandboxes optional
*Accounting
CHANGE: (#3831) ReportGenerator: Authenticated users without JOB_SHARING will now only get plots showing their own jobs, solves #3776
*ResourceStatusSystem
FIX: (#3833) Documentation update
CHANGE: (#3838) For some info, use DMSHelper instead of CSHelper for better precision
*RequestManagementSystem
FIX: (#3829) catch more exception in the ReqClient when trying to display the associated FTS jobs
[v6r20p13]
*FrameworkSystem
FIX: (#3822) obsolete parameter maxQueueSize in UserProfileDB initialization removed
*WorkloadManagementSystem
FIX: (#3824) Added Parameter "Queue" to methods invoked on batch systems by LocalComputingElement
FIX: (#3818) Testing parametric jobs locally now should also work for parametric input data
NEW: (#3818) Parameters from Parametric jobs are also replaced for ModuleParameters,
and not only for common workflow parameters
*DataManagementSystem
FIX: (#3825) FileCatalogCLI: print error message when removeReplica encounters weird return value
FIX: (#3819) ReplicateAndRegister: fix a problem when transferring files to multiple storage
elements, if more than one attempt was needed the transfer to all SEs was not always
happening.
CHANGE: (#3821) FTS3Agent: set pool_size of the FTS3DB
*TransformationSystem
FIX: (#3820) Fix exception in TransformationCleaningAgent: "'str' object not callable"
*ConfigurationSystem
FIX: (#3816) The VOMS2CSAgent was not sending notification emails when the DetailedReport
option was set to False, it will now send emails again when things change for a VO.
CHANGE: (#3816) VOMS2CSAgent: Users to be checked for deletion are now printed sorted and line
by line
NEW: (#3817) dirac-admin-check-config-options script to compare options and values between
the current Configuration and the ConfigTemplates. Allows one to find wrong or
missing option names or just see the difference between the current settings and
the default values.
[v6r20p12]
*Core
FIX: (#3807) Glue2 will return a constant 2500 for the SI00 queue parameter,
any value is needed so that the SiteDirector does not ignore the queue, fixes #3790
*ConfigurationSystem
FIX: (#3797) VOMS2CSAgent: return error when VO is not set (instead of exception)
FIX: (#3797) BDII2CSAgent: Fix for GLUE2URLs option in ConfigTemplate (Lower case S at the end)
*DataManagementSystem
FIX: (#3814) SEManager - adapt to the new meaning of the SE plugin section name