-
Notifications
You must be signed in to change notification settings - Fork 641
/
NEWS
15150 lines (14809 loc) · 860 KB
/
NEWS
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
This file describes changes in recent versions of Slurm. It primarily
documents those changes that are of interest to users and administrators.
* Changes in Slurm 24.11.0rc1
=============================
-- Improve error type logged from partial or incomplete reading from socket or
pipe to avoid potentially logging an error from a previous syscall.
-- slurmrestd - Improve the handling of queries when unable to connect to
slurmdbd by providing responses when possible.
-- slurmrestd,sackd,scrun - Avoid rare hangs related to I/O.
-- scrun - Add support '--all' argument for kill subcommand.
-- Remove srun --cpu-bind=rank.
-- Add resource_spec/cpus and resource_spec/memory entry points in data_parser
to print the CpuSpecList and MemSpecLimit in sinfo --json.
-- sinfo - Add '.sinfo[].resource_spec.cpus' and
'.sinfo[].resource_spec.memory' fields to print the CpuSpecList and
MemSpecLimit dumped by 'sinfo --{json|yaml}'.
-- Increase efficency of sending logs to syslog.
-- Switch to new official YAML mime type "application/yaml" in compliance with
RFC9512 as primary mime type for YAML formatting.
-- slurmrestd - Removed deprecated fields from the following endpoints:
'.result' from 'POST /slurm/v0.0.42/job/submit'.
'.job_id', '.step_id', '.job_submit_user_msg' from 'POST /slurm/v0.0.42/job/{job_id}'.
'.job.exclusive', '.jobs[].exclusive' to 'POST /slurm/v0.0.42/job/submit'.
'.jobs[].exclusive' from 'GET /slurm/v0.0.42/job/{job_id}'.
'.jobs[].exclusive' from 'GET /slurm/v0.0.42/jobs'.
'.job.oversubscribe', '.jobs[].oversubscribe' to 'POST /slurm/v0.0.42/job/submit'.
'.jobs[].oversubscribe' from 'GET /slurm/v0.0.42/job/{job_id}'.
'.jobs[].oversubscribe' from 'GET /slurm/v0.0.42/jobs'.
-- scontrol - Removed deprecated fields '.jobs[].exclusive' and
'.jobs[].oversubscribe' from 'scontrol show jobs --{json|yaml}'.
-- squeue - Removed deprecated fields '.jobs[].exclusive' and
'.jobs[].oversubscribe' from 'squeue --{json|yaml}'.
-- Improve the way to run external commands and fork processes to avoid
non-async-signal safe calls between a fork and an exec. We fork ourselves
now and executes the commands in a safe environment. This includes spank
prolog/epilog executions.
-- Improve MaxMemPerCPU enforcement when exclusive jobs request per node memory
and the partition has heterogeneous nodes.
-- Remove a TOCTOU where multiple steps requesting an energy reading at the
same time could cause too frequent accesses to the drivers.
-- Limit SwitchName to HOST_NAME_MAX chars length.
-- For scancel --ctld and the following rest api endpoints:
'DELETE /slurm/v0.0.40/jobs'
'DELETE /slurm/v0.0.41/jobs'
'DELETE /slurm/v0.0.42/jobs'
Support array expressions in the responses to the client.
-- salloc - Always output node names to the user when an allocation is granted.
-- slurmrestd - Removed all v0.0.39 endpoints.
-- select/linear - Reject jobs asking for GRES per job|socket|task or
cpus|mem per GRES.
-- Add /nodes POST endpoint to REST API, supports multiple node update whereas
previously only single nodes could be updated through /node/<nodename>
endpoint:
'POST /slurm/v0.0.42/nodes'
-- Do not allow changing or setting PreemptMode=GANG to a partition as this is
a cluster-wide option.
* Changes in Slurm 24.05.2
==========================
-- Fix energy gathering rpc counter underflow in _rpc_acct_gather_energy when
more than 10 threads try to get energy at the same time. This prevented
the possibility to get energy from slurmd by any step until slurmd was
restarted, so losing energy accounting metrics in the node.
-- accounting_storage/mysql - Fix issue where new user with wckey did not
have a default wckey sent to the slurmctld.
-- slurmrestd - Prevent slurmrestd segfault when handling the following
endpoints when none of the optional parameters are specified:
'DELETE /slurm/v0.0.40/jobs'
'DELETE /slurm/v0.0.41/jobs'
'GET /slurm/v0.0.40/shares'
'GET /slurm/v0.0.41/shares'
'GET /slurmdb/v0.0.40/instance'
'GET /slurmdb/v0.0.41/instance'
'GET /slurmdb/v0.0.40/instances'
'GET /slurmdb/v0.0.41/instances'
'POST /slurm/v0.0.40/job/{job_id}'
'POST /slurm/v0.0.41/job/{job_id}'
-- Fix IPMI energy gathering when no IPMIPowerSensors are specified in
acct_gather.conf. This situation resulted in an accounted energy of 0
for job steps.
-- Fix a minor memory leak in slurmctld when updating a job dependency.
-- scontrol,squeue - Fix regression that caused incorrect values for
multisocket nodes at '.jobs[].job_resources.nodes.allocation' for
'scontrol show jobs --(json|yaml)' and 'squeue --(json|yaml)'.
-- slurmrestd - Fix regression that caused incorrect values for
multisocket nodes at '.jobs[].job_resources.nodes.allocation' to be dumped
with endpoints:
'GET /slurm/v0.0.41/job/{job_id}'
'GET /slurm/v0.0.41/jobs'
-- jobcomp/filetxt - Fix truncation of job record lines > 1024 characters.
* Changes in Slurm 24.05.1
==========================
-- Fix slurmctld and slurmdbd potentially stopping instead of performing a
logrotate when recieving SIGUSR2 when using auth/slurm.
-- switch/hpe_slingshot - Fix slurmctld crash when upgrading from 23.02.
-- Fix "Could not find group" errors from validate_group() when using
AllowGroups with large /etc/group files.
-- Prevent an assertion in debugging builds when triggering log rotation
in a backup slurmctld.
-- Add AccountingStoreFlags=no_stdio which allows to not record the stdio
paths of the job when set.
-- slurmrestd - Prevent a slurmrestd segfault when parsing the crontab field,
which was never usable. Now it explicitly ignores the value and emits a
warning if it is used for the following endpoints:
'POST /slurm/v0.0.39/job/{job_id}'
'POST /slurm/v0.0.39/job/submit'
'POST /slurm/v0.0.40/job/{job_id}'
'POST /slurm/v0.0.40/job/submit'
'POST /slurm/v0.0.41/job/{job_id}'
'POST /slurm/v0.0.41/job/submit'
'POST /slurm/v0.0.41/job/allocate'
-- mpi/pmi2 - Fix communication issue leading to task launch failure with
"invalid kvs seq from node".
-- Fix getting user environment when using sbatch with "--get-user-env" or
"--export=" when there is a user profile script that reads /proc.
-- Prevent slurmd from crashing if acct_gather_energy/gpu is configured but
GresTypes is not configured.
-- Do not log the following errors when AcctGatherEnergyType plugins are used
but a node does not have or cannot find sensors:
"error: _get_joules_task: can't get info from slurmd"
"error: slurm_get_node_energy: Zero Bytes were transmitted or received"
However, the following error will continue to be logged:
"error: Can't get energy data. No power sensors are available. Try later"
-- sbatch, srun - Set SLURM_NETWORK environment variable if --network is set.
-- Fix cloud nodes not being able to forward to nodes that restarted with new
IP addresses.
-- Fix cwd not being set correctly when running a SPANK plugin with a
spank_user_init() hook and the new "contain_spank" option set.
-- slurmctld - Avoid deadlock during shutdown when auth/slurm is active.
-- Fix segfault in slurmctld with topology/block.
-- sacct - Fix printing of job group for job steps.
-- scrun - Log when an invalid environment variable causes the job submission
to be rejected.
-- accounting_storage/mysql - Fix problem where listing or modifying an
association when specifying a qos list could hang or take a very long time.
-- gpu/nvml - Fix gpuutil/gpumem only tracking last GPU in step. Now,
gpuutil/gpumem will record sums of all GPUS in the step.
-- Fix error in scrontab jobs when using slurm.conf:PropagatePrioProcess=1.
-- Fix slurmctld crash on a batch job submission with "--nodes 0,...".
-- Fix dynamic IP address fanout forwarding when using auth/slurm.
-- Restrict listening sockets in the mpi/pmix plugin and sattach to the
SrunPortRange.
-- slurmrestd - Limit mime types returned from query to 'GET /openapi/v3' to
only return one mime type per serializer plugin to fix issues with OpenAPI
client generators that are unable to handle multiple mime type aliases.
-- Fix many commands possibly reporting an "Unexpected Message Received" when
in reality the connection timed out.
-- Prevent slurmctld from starting if there is not a json serializer present
and the extra_constraints feature is enabled.
-- Fix heterogeneous job components not being signaled with scancel --ctld and
'DELETE slurm/v0.0.40/jobs' if the job ids are not explicitly given,
the heterogeneous job components match the given filters, and the
heterogeneous job leader does not match the given filters.
-- Fix regression from 23.02 impeding job licenses from being cleared.
-- Move error to log_flag which made _get_joules_task error to be logged to the
user when too many rpcs were queued in slurmd for gathering energy.
-- For scancel --ctld and the associated rest api endpoints:
'DELETE /slurm/v0.0.40/jobs'
'DELETE /slurm/v0.0.41/jobs'
Fix canceling the final array task in a job array when the task is pending
and all array tasks have been split into separate job records. Previously
this task was not canceled.
-- Fix power_save operation after recovering from a failed reconfigure.
-- slurmctld - Skip removing the pidfile when running under systemd. In that
situation it is never created in the first place.
-- Fix issue where altering the flags on a Slurm account (UsersAreCoords)
several limits on the account's association would be set to 0 in
Slurm's internal cache.
-- Fix memory leak in the controller when relaying stepmgr step accounting to
the dbd.
-- Fix segfault when submitting stepmgr jobs within an existing allocation.
-- Added "disable_slurm_hydra_bootstrap" as a possible MpiParams parameter in
slurm.conf. Using this will disable env variable injection to allocations
for the following variables: I_MPI_HYDRA_BOOTSTRAP,
I_MPI_HYDRA_BOOTSTRAP_EXEC_EXTRA_ARGS, HYDRA_BOOTSTRAP,
HYDRA_LAUNCHER_EXTRA_ARGS.
-- scrun - Delay shutdown until after start requested. This caused scrun
to never start or shutdown and hung forever when using --tty.
-- Fix backup slurmctld potentially not running the agent when taking over as
the primary controller.
-- Fix primary controller not running the agent when a reconfigure of the
slurmctld fails.
-- slurmd - fix premature timeout waiting for REQUEST_LAUNCH_PROLOG with large
array jobs causing node to drain.
-- jobcomp/{elasticsearch,kafka} - Avoid sending fields with invalid date/time.
-- jobcomp/elasticsearch - Fix slurmctld memory leak from curl usage
-- acct_gather_profile/influxdb - Fix slurmstepd memory leak from curl usage
-- Fix 24.05.0 regression not deleting job hash dirs after MinJobAge.
-- Fix filtering arguments being ignored when using squeue --json.
-- switch/nvidia_imex - Move setup call after spank_init() to allow namespace
manipulation within the SPANK plugin.
-- switch/nvidia_imex - Skip plugin operation if nvidia-caps-imex-channels
device is not present rather than preventing slurmd from starting.
-- switch/nvidia_imex - Skip plugin operation if job_container/tmpfs
is configured due to incompatibility.
-- switch/nvidia_imex - Remove any pre-existing channels when slurmd starts.
-- rpc_queue - Add support for an optional rpc_queue.yaml configuration file.
* Changes in Slurm 24.05.0
==========================
-- Fix regresssion in rc1 causing power_save thread to spin continuously.
-- Improve ctld_relay shutdown sequence.
-- Fixed 'make distclean' behavior for contribs/perlapi.
-- slurmrestd - Avoid ignoring numerical only endpoints during startup with
older libjson-c due to type parsing mismatching.
-- Reject non-stepmgr job allocations requesting --resv-ports from the ctld.
-- slurmrestd - Add fields '.job.resv_ports' '.jobs[].resv_ports' to
'POST /slurm/v0.0.41/job/submit' and 'POST /slurm/v0.0.41/job/allocate'.
-- slurmstepd - Fix crash when cleaning up on shutdown with
--enable-memory-leak-debug.
-- Fix segfault in switch/hpe_slingshot plugin due to initialization squence.
-- scrun - Fix regression in rc1 that caused scrun to crash.
-- Prevent unnecessary log statement when free'ing ports.
-- Fix regression in rc1 causing communication problems when sending large
responses from slurmctld.
-- sreport - fix parsing of 'format=Planned' to prevent it from being
misinterpreted as 'PlannedDown'. 'PlannedDown' is now also known as
'PLNDDown' to match what is printed as the column title.
-- topology/block - Always return an error when the segment size does not
match the system or job specification.
-- Add previously missing timers for Prolog and Epilog scripts when RunInJob is
set.
-- Show an error when PrologFlags RunInJob and Serial are used together.
PrologFlags=Serial is not compatible with how RunInJob operates.
-- Fix memory leak on shutdown when using --enable-memory-leak-debug and
freeing cons_tres node usage.
-- Rename src/stepmgr/gres_ctld.[ch] to src/stepmgr/gres_stepmgr.[ch].
-- Fix various cosmetic issues with states in sinfo.
-- slurmrestd - Avoid crash due to associations query.
-- Calculate a job's min_cpus with consideration to --cpus-per-gpu.
-- Fix scancel request when specifying individual array tasks in combination
with filtering options (in both regular and --interactive mode).
-- Enable MaxStepCount in stepmgr.
-- Enable AccountingStorageEnforce=nojobs,nosteps in stepmgr.
-- Add AccountingStorageParameters=max_step_records to limit how many steps
are recorded in the database for each job -- excluding batch, extern, and
interactive steps.
-- switch/hpe_slingshot - allocate VNIs on the controller for stepmgr jobs and
pass to the stepmgr for steps to use.
-- switch/hpe_slingshot - fix assertion when restarting the controller.
-- switch/hpe_slingshot - fix calcuation of free vnis when restarting the
controller with running jobs.
-- Improve default job reserve MPI port allocations that use overcommit or do
not specify a task count for stepmgr enabled jobs.
-- Fix a regression in rc1 resulting in scrun occasionally deadlocking when
the --enable-memory-leak-debug configure option was used.
-- topology/default - Prevent segfault in slurmctld on 'scontrol show topo'.
-- slurmrestd - Avoid creating or requring a connection to slurmdbd for the
'GET /openapi/v3' endpoint, fixing a regression in rc1.
-- scrun - Fix setting and getting environment via SPANK plugins.
-- sview - Fix nodes tab if a node has RestrictedCoresPerGPU configured.
-- slurmrestd - Add --generate-openapi-spec argument.
-- sview - Prevent segfault when retrieving slurmdbd configuration.
-- Avoid canceling rejected heterogeneous jobs without job write lock.
-- Fix slurmctld crash when reconfiguring with a PrologSlurmctld is running.
-- Fix slurmctld crash after a job has been resized.
* Changes in Slurm 24.05.0rc1
=============================
-- Make slurmstepd retry REQUEST_COMPLETE_BATCH_SCRIPT indefinitely.
-- Always load serializer/json when using any data_parser plugins.
-- slurmrestd - Reject single http query with multiple path requests.
-- slurmrestd - Add time/planned field to slurmdb/v0.0.41/job/{job_id}.
-- Improve Power Save's Resume/Suspend rate limiting.
-- slurmrestd - Improve reliablity under high memory pressure by closing
connections instead of forcing a fatal exit due to lack of memory.
-- data_parser/v0.0.41 - Avoid aborting when invoking a not implemented
parser.
-- data_parser/v0.0.41 - Fix how nice values are parsed for job submissions.
-- data_parser/v0.0.41 - Fix regression where parsing error did not result in
invalid request being rejected.
-- Print an error message in 'scontrol reboot' when a node reboot request
is ignored due to the current node state.
-- squeue - Add "--notme" option.
-- data_parser/v0.0.41 - change "association.id" to just include the int "id"
rather than include redundant assoc info (cluster, user, partition, account)
that's already included in the "association" object.
-- data_parser/v0.0.41 - Improve parsing of numeric user id.
-- data_parser/v0.0.41 - Improve parsing of numeric group id.
-- slurmrestd - Generated openapi.json will only populate "deprecated" fields
if true. False is the default value and does not require being present.
-- slurmrestd - Populate missing "deprecated" fields in openapi.json.
-- slurmrestd - Corrected deprecated fields in generated openapi.json not
getting populated.
-- slurmrestd - Generated openapi.json will have reduced number of "$ref"
fields. Where there was only 1 reference for the schema, the "$ref" schema
will be directly populated in place.
-- slurmrestd - Rename *_NO_VAL schemas in generated openapi.json to have
_struct and to pass along correct integer sizing when possible.
-- slurmrestd - Correct description fields in generated openapi.json where
descriptions were not present or too generic.
-- Remove support for Cray XC ("cray_aries") systems.
-- Prevent backup slurmctld from taking over if the heartbeat file is
still being updated. Failure to ping may have been due to clock skew.
-- serializer/yaml - Converted to new parsing interface in libyaml to improve
parsing compatibility.
-- Removed TopologyPlugin tree and dragonfly support from select/linear.
If those topology plugins are desired please switch to select/cons_tres.
-- Changed slurmrestd.service to only listen on TCP socket by default.
Environments with existing drop-in units for the service may need
further adjustments to work after upgrading.
-- Fix how gres are allocated per job when using multiple gres types.
-- Log an error when UnkillableStepTimeout is less than five times
MessageTimeout.
-- Avoid step gres dealloc count underflow errors after reconfiguring or
restarting slurmctld.
-- Fix controller not validating periodic dynamic future registrations.
-- Fix dynamic future nodes registering as new node when specifying -N<name>.
-- Fix sbcast (or srun --bcast) --send-libs when it is used multiple times in
the same job. Previously, subsequent calls to sbcast --send-libs would
overwrite the libraries for the first executable.
-- Add support for sbcast --preserve when job_container/tmpfs configured
(previously documented as unsupported).
-- Changed the default value for UnkillableStepTimeout to 60 seconds or five
times the value of MessageTimeout, whichever is greater.
-- slurmctld - Check if --deadline has been reached and not satisfied on held
jobs, otherwise they could remain without automatic cancellation until after
the job is released.
-- scrun/slurmrestd/sackd - Avoid closing all listening sockets when
interrupted from signal such as SIGALRM. Normal shutdown remains unaffected.
-- Remove systemd AbandonScope() logic for scope units as it is not needed.
-- Fix GresUsed output from `scontrol show nodes --details` showing GRES types
that are not configured on a node.
-- slurmrestd - Fatal during start up when loading content plugin fails.
-- slurmrestd - Reduce complexity in URL path matching.
-- data_parser/v0.0.41 - Emit a warning instead of an error if a disabled
parser is invoked.
-- Federation - allow client command operation when slurmdbd is unavailable.
-- Enforce mutual exclusivity of --systemd and -D when launching daemons
-- slurmctld - remove -d option
-- burst_buffer/lua - Trigger a burst_buffer event for strigger when the
real_size function fails.
-- burst_buffer/lua - Added two new hooks: slurm_bb_test_data_in and
slurm_bb_test_data_out. The syntax and use of the new hooks are documented
in etc/burst_buffer.lua.example. These are required to exist. slurmctld now
checks on startup if the burst_buffer.lua script loads and contains all
required hooks; slurmctld will exit with a fatal error if this is not
successful. Added PollInterval to burst_buffer.conf. Removed the arbitrary
limit of 512 copies of the script running simultaneously.
-- sackd/slurmrestd/scrun - Avoid using empty string while logging unix socket
connections from a listening connection.
-- Fix 20 character username limit from 'sacctmgr show events'
-- Log an error if UsePss or NoShare are configured with a plugin other than
jobacct_gather/linux. In such case these parameters are ignored.
-- helpers.conf - Added Flags=rebootless parameter allowing feature changes
without rebooting compute nodes.
-- scontrol - Add new subcommand 'power' for node power control.
-- data_parser/v0.0.41 - Implement parser of distribution for
/slurm/v0.0.41/job/submit.
-- data_parser/v0.0.41 - Change distribution_plane_size field type from UINT16
to UINT16_NO_VAL for /slurm/v0.0.41/job/submit.
-- topology/block - Replaced the BlockLevels with BlockSizes in topology.conf.
-- Fix slurmd cgroup/v2 startup race with systemd and cgroupfs.
-- Add SystemdTimeout= parameter in cgroup.conf.
-- Add QOS limit MaxTRESRunMinsPerAccount.
-- Add QOS limit MaxTRESRunMinsPerUser.
-- jobcomp/{elasticsearch,kafka} - Send priority alongside the rest of fields.
-- Add contain_spank option to SlurmdParameters. When set, spank_user_init(),
spank_task_post_fork(), and spank_task_exit() will execute within the
job_container/tmpfs plugin namespace.
-- Update job reason appropriately when bf_licenses is used.
-- slurmrestd - Tagged `script` field as deprecated in
'POST /slurm/v0.0.41/job/submit' in anticipation of removal in future
OpenAPI plugin versions.
-- Fix salloc/sbatch/srun crashing with certain invalid nodelist requests.
-- Optimize jobacctgather by not iterating every time over pids that have
already finished.
-- Remote SPANK callbacks invoked by srun get called once instead of twice.
-- auth/slurm - Support multiple keys through slurm.jwks.
-- sched/backfill - Fix issue with bf_continue where a job partition request
could be incorrectly reset back to a partition that is no longer specified
after a job partition update processed during a lock yield time window.
-- slurmrestd - Explicitly set process as dumpable (and ptrace-able) at
startup for systems where suid_dumpable is not 2.
-- slurmrestd - Tag all /slurm/v0.0.39/ and /slurmdb/v0.0.39/ endpoints as
deprecated in anticipation of removal in Slurm 24.11.
-- Add ELIGIBLE environment variable to jobcomp/script plugin.
-- slurmrestd,sackd,scrun - Improve outgoing data efficiency using
non-contiguous write support in kernel.
-- sackd - Add support for SACKD_DEBUG, SACKD_STDERR_DEBUG, and
SACKD_SYSLOG_DEBUG environment variables to control logging.
-- mpi/pmi2 - PMI_process_mapping values have been adapted for executions
where arbitrary distribution/SLURM_HOSTFILE is used. Now it can take into
account multiple instances of the same node inside SLURM_HOSTFILE.
-- Avoid wrong limit oriented (i.e. QosMaxGresPer*) job pending reason for jobs
actually pending on Resources when GPUs are requested per job.
-- Fix --ntasks-per-node not being treated as a max count of tasks per node
when used in combination with --ntasks. --ntasks option will now take
precedence as it is documented.
-- Accept X11 cookies that do not have a display number associated with it.
-- Always use the QOS name for SLURM_JOB_QOS environment variables.
Previously the batch environment would use the description field,
which was usually equivalent to the name.
-- slurmrestd - Add "CRON_JOBS" as possible flag value to the following:
'DELETE /slurm/v0.0.40/jobs' flags field.
'DELETE /slurm/v0.0.41/jobs' flags field.
'DELETE /slurm/v0.0.40/job/{job_id}?flags=' flags query parameter.
'DELETE /slurm/v0.0.41/job/{job_id}?flags=' flags query parameter.
-- Fix ScronParameters=explicit_scancel when using the rest api DELETE jobs
query: if the CRON_JOBS flag is not used then cron jobs will not be
cancelled. The NO_CRON_JOBS flag is ignored in v0.0.40 and removed in
v0.0.41.
-- Pass multi-partition job priorities to job for squeue to display.
-- cgroup/v2 - Require dbus-1 version >= 1.11.16.
-- Add RestrictedCoresPerGPU configuration option.
-- Fix how ntasks is inferred from --cpus-per-task when using --nodes,
--threads-per-core, or --hint=nomultithread.
-- For PreemptMode=CANCEL and PreemptMode=REQUEUE assume that job signalled
for GraceTime was preempted.
-- slurmd - Retry fetching configs indefinitely during startup.
-- Fix SPANK options not bing sent to remote context when --export was used.
-- slurmrestd - Attempt to automatically convert enumerated string arrays with
incoming non-string values into strings. Add warning when incoming value for
enumerated string arrays can not be converted to string and silently ignore
instead of rejecting entire request.
-- slurmrestd - Require `user` and `association_condition` fields to be
populated for requests to 'POST /slurmdb/v0.0.41/users_association'.
-- Allow NodeSet names to be used in SuspendExcNodes.
-- SuspendExcNodes=<nodes>:N now counts allocated nodes in N. The first N
powered up nodes in <nodes> are protected from being suspended.
-- Add SlurmctldParameters=max_powered_nodes=N, which prevents powering up
nodes after the max is reached.
-- Store output, error and input paths in the database and make them available
in accounting tools.
-- slurmrestd - Add 'POST /slurm/v0.0.41/job/allocate' endpoint.
-- Fix issues related to the extern step getting killed before other steps.
This includes the job_containter/tmpfs plugin not cleaning up.
-- Add USER_DELETE reservation flag to allow users with access to a reservation
to delete it.
-- Add CgroupPlugin=disabled to disable any interaction with Cgroups.
-- slurmrestd - Add "STEPMGR_ENABLED" as possible flag value to the following:
'GET /slurm/v0.0.41/jobs' flags field.
'GET /slurm/v0.0.41/job/{job_id}' flags query parameter.
-- scontrol,squeue - Added possible flags "STEPMGR_ENABLED" to '.jobs[].flags' for
'scontrol show jobs --{json|yaml}' and 'squeue --{json|yaml}' responses.
-- Add SlurmctldParameters=enable_stepmgr to enable step management through
the slurmstepd instead of the controller.
-- Avoid slurmstepd infinite loop waiting for tasks termination.
-- Fix logging of JSON/YAML values in some messages where nothing would be
printed as the value instead of the actual JSONified version of the parsed
string.
-- slurmrestd,sackd,scrun - Improve logic around handling kernel provided
buffer size of incoming data in files/sockets/pipes to avoid crashes.
-- Add --segment to job allocation to be used in topology/block.
-- Add --exclusive=topo for use with topology/block.
-- Add ExclusiveTopo to a partition definition in slurm.conf.
-- Add new 'BLOCKED' state to a node.
-- Account coordinators may not increase association job limits above
parent ones
-- Account coordinators can now suspend/resume jobs owned by member users.
-- Add DisableCoordDBD slurmdbd configuration parameter to disable the
coordinator status in all slurmdbd interactions.
-- slurmrestd - Added possible flags "WithAssociations" and "WithCoordinators"
to `.accounts[].flags` for "GET /slurmdb/v0.0.41/accounts/" and
"POST /slurmdb/v0.0.41/accounts/" endpoints.
-- sacctmgr - Added possible flags "WithAssociations" and "WithCoordinators"
to `.accounts[].flags` for `sacctmgr show accounts --{json|yaml}` response.
-- slurmrestd - Rename URL query parameter "with_assocs" to "WithAssociations"
for "GET /slurmdb/v0.0.41/accounts?WithAssociations".
-- slurmrestd - Rename URL query parameter "with_coords" to "WithCoordinators"
for "GET /slurmdb/v0.0.41/accounts?WithCoordinators".
-- slurmrestd - Rename URL query parameter "with_deleted" to "deleted"
for "GET /slurmdb/v0.0.41/accounts?deleted".
-- slurmrestd - Added possible flags "RemoveUsersAreCoords" and
"UsersAreCoords" to `.accounts[].flags` for "GET /slurmdb/v0.0.41/accounts/"
and "POST /slurmdb/v0.0.41/accounts/" endpoints.
-- sacctmgr - Added possible flags "RemoveUsersAreCoords" and "UsersAreCoords"
to `.accounts[].flags` for `sacctmgr show accounts --{json|yaml}` response.
-- slurmrestd - Add URL query parameter "UsersAreCoords" and
"RemoveUsersAreCoords" for
"GET /slurmdb/v0.0.41/accounts?UsersAreCoords&RemoveUsersAreCoords".
-- sacctmgr - Add new possible new flags "NoUpdate" and "Exact" to
'.associations[].flags' response from 'sacctmgr show assocs --{json|yaml}'.
-- slurmrestd - Added possible flags "NoUpdate" and "Exact" to
`.associations[].flags` for "GET /slurmdb/v0.0.41/associations/"
and "POST /slurmdb/v0.0.41/associations/" endpoints.
-- Fix false success of REQUEST_FORWARD_DATA RPC that made pmix to get out
of sync during initialization.
-- slurmrestd - Allow startup when slurmdbd is not configured and avoid loading
slurmdbd specific plugins.
-- Added PrologFlags=RunInJob to make prolog and epilog run inside the job
extern step to include it in the job's cgroup.
-- Return '*' for the password field for nss_slurm instead of "x".
-- slurmrestd - Add "topo" as possible value to the following:
'GET /slurm/v0.0.41/jobs' in '.jobs[].shared' field
'GET /slurm/v0.0.41/job/{job_id}' in '.jobs[].shared' field
'POST /slurm/v0.0.41/job/submit' in '.job.shared' and '.jobs[].shared'
'POST /slurm/v0.0.41/job/allocate' in '.job.shared' and '.jobs[].shared'
-- sacctmgr - Added possible flags "NoUsersAreCoords" and "UsersAreCoords"
to `.accounts[].flags` for `sacctmgr show accounts --{json|yaml}` response.
-- sacct - Add "topo" as possible value to output of 'sacct --{json|yaml}' to
'.jobs[].shared' field.
-- squeue - Add "topo" as possible value to output of 'squeue --{json|yaml}' to
'.jobs[].shared' field.
-- scontrol - Add "topo" as possible value to output of
'scontrol show jobs --{json|yaml}' to '.jobs[].shared' field.
-- slurmrestd - Add "topo" as possible value to the following:
'GET /slurm/v0.0.41/jobs' in '.jobs[].exclusive' field
'GET /slurm/v0.0.41/job/{job_id}' in '.jobs[].exclusive' field
'POST /slurm/v0.0.41/job/submit' in '.job.exclusive' and
'.jobs[].exclusive'
'POST /slurm/v0.0.41/job/allocate' in '.job.exclusive' and
'.jobs[].exclusive'
-- sacctmgr - Added possible flags "RemoveUsersAreCoords" and "UsersAreCoords"
to `.accounts[].flags` for `sacctmgr show accounts --{json|yaml}` response.
-- sacct - Add "topo" as possible value to output of 'sacct --{json|yaml}' to
'.jobs[].exclusive' field.
-- squeue - Add "topo" as possible value to output of 'squeue --{json|yaml}' to
'.jobs[].exclusive' field.
-- scontrol - Add "topo" as possible value to output of
'scontrol show jobs --{json|yaml}' to '.jobs[].exclusive' field.
-- slurmrestd - Add fields '.job.segment_size' and '.jobs[].segment_size' to
'POST /slurm/v0.0.41/job/submit' and 'POST /slurm/v0.0.41/job/allocate'.
-- sacctmgr - Added possible flags "NoUsersAreCoords" and "UsersAreCoords"
to `.associations[].flags` for `sacctmgr show assocs --{json|yaml}`
response.
-- slurmrestd - Added possible flags "NoUsersAreCoords" and "UsersAreCoords" to
`.associations[].flags` for "GET /slurmdb/v0.0.41/associations/"
and "POST /slurmdb/v0.0.41/associations/" endpoints.
-- Add ability to reserve MPI ports at the job level for stepmgr jobs and
subdivide them at the step level.
-- slurmrestd - Fix possible memory leak from failed job submissions to
'POST /slurm/v0.0.{39,40,41}/job/submit'.
-- slurmrestd - Fix possible memory leak from failed job allocation requests to
'POST /slurm/v0.0.{39,40,41}/job/allocate'.
* Changes in Slurm 23.11.9
==========================
-- Fix many commands possibly reporting an "Unexpected Message Received" when
in reality the connection timed out.
-- Fix heterogeneous job components not being signaled with scancel --ctld and
'DELETE slurm/v0.0.40/jobs' if the job ids are not explicitly given,
the heterogeneous job components match the given filters, and the
heterogeneous job leader does not match the given filters.
-- Fix regression from 23.02 impeding job licenses from being cleared.
-- Move error to log_flag which made _get_joules_task error to be logged to the
user when too many rpcs were queued in slurmd for gathering energy.
-- slurmrestd - Prevent a slurmrestd segfault when modifying an association
without specifying max TRES limits in the request if those TRES
limits are currently defined in the association. This affects the following
fields of endpoint 'POST /slurmdb/v0.0.38/associations/':
'associations/max/tres/per/job'
'associations/max/tres/per/node'
'associations/max/tres/total'
'associations/max/tres/minutes/per/job'
'associations/max/tres/minutes/total'
-- Fix power_save operation after recovering from a failed reconfigure.
-- scrun - Delay shutdown until after start requested. This caused scrun
to never start or shutdown and hung forever when using --tty.
-- Fix backup slurmctld potentially not running the agent when taking over as
the primary controller.
-- Fix primary controller not running the agent when a reconfigure of the
slurmctld fails.
-- jobcomp/{elasticsearch,kafka} - Avoid sending fields with invalid date/time.
-- Fix energy gathering rpc counter underflow in _rpc_acct_gather_energy when
more than 10 threads try to get energy at the same time. This prevented
the possibility to get energy from slurmd by any step until slurmd was
restarted, so losing energy accounting metrics in the node.
* Changes in Slurm 23.11.8
==========================
-- Fix slurmctld crash when reconfiguring with a PrologSlurmctld is running.
-- Fix slurmctld crash after a job has been resized.
-- Fix slurmctld and slurmdbd potentially stopping instead of performing a
logrotate when recieving SIGUSR2 when using auth/slurm.
-- Fix not having a disabled value for keepalive CommunicationParameters in
slurm.conf when these parameters are not set. This can log an error when
setting a socket, for example during slurmdbd registration with ctld.
-- switch/hpe_slingshot - Fix slurmctld crash when upgrading from 23.02.
-- Fix "Could not find group" errors from validate_group() when using
AllowGroups with large /etc/group files.
-- slurmrestd - Prevent a slurmrestd segfault when parsing the crontab field,
which was never usable. Now it explicitly ignores the value and emits a
warning if it is used for the following endpoints:
'POST /slurm/v0.0.39/job/{job_id}'
'POST /slurm/v0.0.39/job/submit'
'POST /slurm/v0.0.40/job/{job_id}'
'POST /slurm/v0.0.40/job/submit'
-- Fix getting user environment when using sbatch with "--get-user-env" or
"--export=" when there is a user profile script that reads /proc.
-- Prevent slurmd from crashing if acct_gather_energy/gpu is configured but
GresTypes is not configured.
-- Do not log the following errors when AcctGatherEnergyType plugins are used
but a node does not have or cannot find sensors:
"error: _get_joules_task: can't get info from slurmd"
"error: slurm_get_node_energy: Zero Bytes were transmitted or received"
However, the following error will continue to be logged:
"error: Can't get energy data. No power sensors are available. Try later"
-- Fix cloud nodes not being able to forward to nodes that restarted with new
IP addresses.
-- sacct - Fix printing of job group for job steps.
-- Fix error in scrontab jobs when using slurm.conf:PropagatePrioProcess=1.
-- Fix slurmctld crash on a batch job submission with "--nodes 0,...".
-- Fix dynamic IP address fanout forwarding when using auth/slurm.
* Changes in Slurm 23.11.7
==========================
-- slurmrestd - Correct OpenAPI specification for
'GET /slurm/v0.0.40/jobs/state' having response as null.
-- Allow running jobs on overlapping partitions if jobs don't specify -s.
-- Fix segfault when requesting a shared gres along with an exclusive
allocation.
-- Fix regression in 23.02 where afternotok and afterok dependencies were
rejected for federated jobs not running on the origin cluster of the
submitting job.
-- slurmctld - Disable job table locking while job state cache is active when
replying to `squeue --only-job-state` or `GET /slurm/v0.0.40/jobs/state`.
-- Fix sanity check when setting tres-per-task on the job allocation as well as
the step.
-- slurmrestd - Fix compatiblity with auth/slurm.
-- Fix issue where TRESRunMins gets off correct value if using
QOS UsageFactor != 1.
-- slurmrestd - Require `user` and `association_condition` fields to be
populated for requests to 'POST /slurmdb/v0.0.40/users_association'.
-- Avoid a slurmctld crash with extra_constraints enabled when a job requests
certain invalid --extra values.
-- `scancel --ctld` and `DELETE /slurm/v0.0/40/jobs` - Fix support for job
array expressions (e.g. 1_[3-5]). Also fix signaling a single pending array
task (e.g. 1_10), which previously signaled the whole array job instead.
-- Fix a possible slurmctld segfault when at some point we failed to create an
external launcher step.
-- Allow the slurmctld to open a connection to the slurmdbd if the first
attempt fails due to a protocol error.
-- mpi/cray_shasta - Fix launch for non-het-steps within a hetjob.
-- sacct - Fix "gpuutil" TRES usage output being incorrect when using --units.
-- Fix a rare deadlock on slurmctld shutdown or reconfigure.
-- Fix issue that only left one thread on each core available when "CPUs=" is
configured to total thread count on multi-threaded hardware and no other
topology info ("Sockets=", "CoresPerSocket", etc.) is configured.
-- Fix the external launcher step not being allocated a VNI when requested.
-- jobcomp/kafka - Fix payload length when producing and sending a message.
-- scrun - Avoid a crash if RunTimeDelete is called before the container
finishes.
-- Save the slurmd's cred_state while reconfiguring to prevent the loss job
credentials.
* Changes in Slurm 23.11.6
==========================
-- Avoid limiting sockets per node to one when using gres enforce-binding.
-- slurmrestd - Avoid permission denied errors when attempting to listen on
the same port multiple times.
-- Fix GRES reservations where the GRES has no topology
(no cores= in gres.conf).
-- Ensure that thread_id_rpc is gone before priority_g_fini().
-- Fix scontrol reboot timeout removing drain state from nodes.
-- squeue - Print header on empty reponse to `--only-job-state`.
-- Fix slurmrestd not ending job properly when xauth is not present and a x11
job is sent.
-- Add experimental job state caching with
SchedulerParameters=enable_job_state_cache to speed up querying job states
with squeue --only-job-state.
-- slurmrestd - Correct dumping of invalid ArrayJobIds returned from
'GET /slurm/v0.0.40/jobs/state'.
-- squeue - Correct dumping of invalid ArrayJobIds returned from
`squeue --only-job-state --{json|yaml}`.
-- If scancel --ctld is not used with --interactive, --sibling, or specific
step ids, then this option issues a single request to the slurmctld to
signal all jobs matching the specified filters. This greatly improves
the performance of slurmctld and scancel. The updated --ctld option also
fixes issues with the --partition or --reservation scancel options for jobs
that requested multiple partitions or reservations.
-- slurmrestd - Give EINVAL error when failing to parse signal name to numeric
signal.
-- slurmrestd - Allow ContentBody for all methods per RFC7230 even if ignored.
-- slurmrestd - Add 'DELETE /slurm/v0.0.40/jobs' endpoint to allow bulk job
signaling via slurmctld.
-- Fix combination of --nodelist and --exclude not always respecting the
excluded node list.
-- Fix jobs incorrectly allocating nodes exclusively when started on a
partition that doesn't enforce it. This could happen if a multi-partition
job doesn't specify --exclusive and is evaluated first on a partition
configured with OverSubscribe=EXCLUSIVE but ends up starting in a partition
configured with OverSubscribe!=EXCLUSIVE evaluated afterwards.
-- Setting GLOB_SILENCE flag no longer exposes old bugged behavior.
-- Fix associations AssocGrpCPURunMinutes being incorrectly computed for
running jobs after a controller reconfiguration/restart.
-- Fix scheduling jobs that request --gpus and nodes have different node
weights and different numbers of gpus.
-- slurmrestd - Add "NO_CRON_JOBS" as possible flag value to the following:
'DELETE /slurm/v0.0.40/jobs' flags field.
'DELETE /slurm/v0.0.40/job/{job_id}?flags=' flags query parameter.
-- Fix scontrol segfault/assert failure if the TRESPerNode parameter is used
when creating reservations.
-- Avoid checking for wsrep_on when restoring streaming replication settings.
-- Clarify in the logs that error "1193 Unknown system variable 'wsrep_on'" is
innocuous.
-- accounting_storage/mysql - Fix problem when loading reservations from an
archive dump.
-- slurmdbd - Fix minor race condition when sending updates to a shutdown
slurmctld.
-- slurmctld - Fix invalid refusal of a reservation update.
-- openapi - Fix memory leak of /meta/slurm/cluster response field.
-- Fix memory leak when using auth/slurm and AuthInfo=use_client_ids.
* Changes in Slurm 23.11.5
==========================
-- Fix Debian package build on systems that are not able to query the systemd
package.
-- data_parser/v0.0.40 - Emit a warning instead of an error if a disabled
parser is invoked.
-- slurmrestd - Improve handling when content plugins rely on parsers
that haven't been loaded.
-- Fix old pending jobs dying (Slurm version 21.08.x and older) when upgrading
Slurm due to "Invalid message version" errors.
-- Have client commands sleep for progressively longer periods when backed off
by the RPC rate limiting system.
-- slurmctld - Ensure agent queue is flushed correctly at shutdown time.
-- slurmdbd - correct lineage construction during assoc table conversion for
partition based associations.
-- Add new RPCs and API call for faster querying of job states from slurmctld.
-- slurmrestd - Add endpoint '/slurm/{data_parser}/jobs/state'.
-- squeue - Add `--only-job-state` argument to use faster query of job states.
-- Make a job requesting --no-requeue, or JobRequeue=0 in the slurm.conf,
supersede RequeueExit[Hold].
-- Add sackd man page to the Debian package.
-- Fix issues with tasks when a job was shrinked more than once.
-- Fix reservation update validation that resulted in reject of correct
updates of reservation when the reservation was running jobs.
-- Fix possible segfault when the backup slurmctld is asserting control.
-- Fix regression introduced in 23.02.4 where slurmctld was not properly
tracking the total GRES selected for exclusive multi-node jobs, potentially
and incorrectly bypassing limits.
-- Fix tracking of jobs typeless GRES count when multiple typed GRES with the
same name are also present in the job allocation. Otherwise, the job could
bypass limits configured for the typeless GRES.
-- Fix tracking of jobs typeless GRES count when request specification has a
typeless GRES name first and then typed GRES of different names (i.e.
--gres=gpu:1,tmpfs:foo:2,tmpfs:bar:7). Otherwise, the job could bypass
limits configured for the generic of the typed one (tmpfs in the example).
-- Fix batch step not having SLURM_CLUSTER_NAME filled in.
-- slurmstepd - Avoid error during `--container` job cleanup about
RunTimeQuery never being configured. Results in cleanup where job steps not
fully started.
-- Fix nodes not being rebooted when using salloc/sbatch/srun "--reboot" flag.
-- Send scrun.lua in configless mode.
-- Fix rejecting an interactive job whose extra constraint request cannot
immediately be satisfied.
-- Fix regression in 23.11.0 when parsing LogTimeFormat=iso8601_ms that
prevented milliseconds from being printed.
-- Fix issue where you could have a gpu allocated as well as a shard on that
gpu allocated at the same time.
-- Fix slurmctld crashes when using extra constraints with job arrays.
-- sackd/slurmrestd/scrun - Avoid memory leak on new unix socket connection.
-- The failed node field is filled when a node fails but does not time out.
-- topology/block - Implement topology_p_generate_node_ranking().
-- slurmrestd - Remove requiring job script field and job component script
fields to both be populated in the `POST /slurm/v0.0.40/job/submit`
endpoint as there can only be one batch step script for a job.
-- slurmrestd - When job script is provided in '.jobs[].script' and '.script'
fields, the '.script' field's value will be used in the
`POST /slurm/v0.0.40/job/submit` endpoint.
-- slurmrestd - Reject HetJob submission missing or empty batch script for
first Het component in the `POST /slurm/v0.0.40/job/submit` endpoint.
-- slurmrestd - Reject job when empty batch script submitted to the
POST /slurm/v0.0.40/job/submit` endpoint.
-- Fix pam_slurm and pam_slurm_adopt when using auth/slurm.
-- slurmrestd - Add 'cores_per_socket' field to
`POST /slurm/v0.0.40/job/submit` endpoint.
-- Fix srun and other Slurm commands running within a "configless" salloc when
salloc itself fetched the config.
-- Enforce binding with shared gres selection if requested.
-- Fix job allocation failures when the requested tres type or name ends in
"gres" or "license".
-- accounting_storage/mysql - Fix lineage string construction when adding a
user association with a partition.
-- Fix sattach command.
-- Fix ReconfigFlags. Due how reconfig was changed in 23.11, they will also
be used to influence the slurmctld startup as well.
-- Fix starting slurmd in configless mode if MUNGE support was disabled.
* Changes in Slurm 23.11.4
==========================
-- Fix a memory leak when updating partition nodes.
-- Don't leave a partition around if it fails to create with scontrol.
-- Fix segfault when creating partition with bad node list from scontrol.
-- Fix preserving partition nodes on bad node list update from scontrol.
-- Fix assertion in developer mode on a failed message unpack.
-- Fix repeat POWER_DOWN requests making the nodes available for ping.
-- Fix rebuilding job alias_list on restart when nodes are still powering up.
-- Fix INVALID nodes running health check.
-- Fix cloud/future nodes not setting addresses on invalid registration.
-- scrun - Remove the requirement to set the SCRUN_WORKING_DIR environment
variable. This was a regression in 23.11.
-- Add warning for using select/linear with topology/tree.
This combination will not be supported in the next major version.
-- Fix health check program not being run after first pass of all nodes when
using MaxNodeCount.
-- sacct - Set process exit code to one for all errors.
-- Add SlurmctldParameters=disable_triggers option.
-- Fix issue running steps when the allocation requested an exclusive
allocation shards along with shards.
-- Fix cleaning up the sleep process and the cgroup of the extern step if
slurm_spank_task_post_fork returns an error.
-- slurm_completion - Add missing --gres-flags= options
multiple-tasks-per-sharing and one-task-per-sharing.
-- scrun - Avoid race condition that could cause outbound network
communications to incorrectly rejected with an incomplete packet error.
-- scrun - Gracefully handle kernel giving invalid expected number of incoming
bytes for a connection causing incoming packet corruption resulting in
connection getting closed.
-- srun - return 1 when a step lauch fails
-- scrun - Avoid race condition that could cause deadlock during shutdown.
-- Fix scontrol listpids to work under dynamic node scenarios.
-- Add --tres-bind to --help and --usage output.
-- Add --gres-flags=allow-task-sharing to allow GPUs to still be accessible
among all tasks when binding GPUs to specific tasks.
-- Fix issue with CUDA_VISIBLE_DEVICES showing the same MIG device for all
tasks when using MIGs with --tres-per-task or --gpus-per-task.
-- slurmctld - Prevent a potential hang during shutdown/reconfigure if the
association cache thread was previously shut down.
-- scrun - Avoid race condition that could cause scrun to hang during
shutdown when connections have pending events.
-- scrun - Avoid excessive polling of connections during shutdown that could
needlessly cause 100% CPU usage on a thread.
-- sbcast - Use user identity from broadcast credential instead of looking it
up locally on the node.
-- scontrol - Remove "abort" option handling.
-- Fix an error message referring to the wrong RPC.
-- Fix memory leak on error when creating dynamic nodes.
-- Fix a slurmctld segfault when a cloud/dynamic node changes hostname on
registration.
-- Prevent a slurmctld deadlock if the gpu plugin fails to load when
creating a node.
-- Change a slurmctld fatal() to an error() when attempting to create a
dynamic node with a global autodetect set in gres.conf.
-- Fix leaving node records on error when creating nodes with scontrol.
-- scrun/sackd - Avoid race condition where shutdown could deadlock.
-- Fix a regression in 23.02.5 that caused pam_slurm_adopt to fail when
the user has multiple jobs on a node.
-- Add GLOB_SILENCE flag that silences the error message which will display if
an include directive attempts to use the "*" wildcard.
-- Fix jobs getting rejected when submitting with --gpus option from older
versions of job submission commands (23.02 and older).
-- cgroup/v2 - Return 0 for VSZ. Kernel cgroups do not provide this metric.
-- scrun - Avoid race condition where outbound RPCs could be corrupted.
-- scrun - Avoid race condition that could cause a crash while compiled in
debug mode.
-- gpu/rsmi - Disable gpu usage statistics when not using ROCM 6.0.0+
-- Fix stuck processes and incorrect environment when using --get-user-env.
-- Avoid segfault in the slurmdbd when TrackWCKey=no but you are still using
use WCKeys.
-- Fix ctld segfault with TopologyParam=RoutePart and no partition defined.
-- slurmctld - Fix missing --deadline handling for jobs not evaluated by the
schedulers (i.e. non-runnable, skipped for other reasons, etc.).
-- Demote some eio related logs from error to verbose in user commands. These
are not generally actionable by the user and are easilly generated by port
scanning a machine running srun.
-- Make sprio correctly print array tasks that have not yet been split out.
-- topology/block - Restrict the number of last-level blocks in any allocation.
-- slurmrestd - Treat multiple repeat URL query values as list.
-- slurmrestd - Treat all URL query values as string by default to avoid
parser warnings.
* Changes in Slurm 23.11.3
==========================
-- Fix debian/changelog file to reflect the correct version so the packages
are generated correctly.
* Changes in Slurm 23.11.2
==========================
-- slurmrestd - Reject single http query with multiple path requests.
-- Fix launching Singularity v4.x containers with srun --container by setting
.process.terminal to true in generated config.json when step has
pseudoterminal (--pty) requested.
-- Fix loading in dyanmic/cloud node jobs after net_cred expired.
-- Fix cgroup null path error on slurmd/slurmstepd tear down.
-- data_parser/v0.0.40 - Prevent failure if accounting is disabled, instead
issue a warning if needed data from the database can not be retrieved.
-- openapi/slurmctld - Prevent failure if accounting is disabled.
-- Prevent slurmscriptd processing delays from blocking other threads in
slurmctld while trying to launch various scripts. This is additional work
for a fix in 23.02.6.
-- Fix memory leak when receiving alias addrs from controller.
-- scontrol - Accept `scontrol token lifespan=infinite` to create tokens that
effectively do not expire.
-- Avoid errors when Slurmdb accounting disabled when '--json' or '--yaml' is
invoked with CLI commands and slurmrestd. Add warnings when query would
have populated data from Slurmdb instead of errors.
-- Fix slurmctld memory leak when running job with --tres-per-task=gres:shard:#
-- Fix backfill trying to start jobs outside of backfill window.
-- Fix oversubscription on partitions with PreemptMode=OFF.
-- Preserve node reason on power up if the node is downed or drained.
-- data_parser/v0.0.40 - Avoid aborting when invoking a not implemented
parser.
-- data_parser/v0.0.40 - Fix how nice values are parsed for job submissions.
-- data_parser/v0.0.40 - Fix regression where parsing error did not result in
invalid request being rejected.
-- Fix segfault in front-end node registration.
-- Prevent jobs using none typed gpus from being killed by the controller after
a reconfig or restart.
-- Fix deadlock situation in the dbd when adding associations.
-- Update default values of text/blob columns when updating from old mysql
versions in more situations. This improves a previous fix to handle an
uncommon case when upgrading mysql/mariadb.
-- Fix rpmbuild in openSUSE/SLES due to incorrect mariadb dependency.
-- Fix compilation on RHEL 7.
-- When upgrading the slurmdbd to 23.11, avoid generating a query to update
the association table that is larger than max_allowed_packet which would
result in an upgrade failure.
-- Fix rare deadlock when a dynamic node registers at the same time that a
once per minute background task occurs.
-- Fix build issue on 32-bit systems.
-- data_parser/v0.0.40 - Fix enumerated strings in OpenAPI specification not
have type field specified.
-- Improve scontrol show job -d information of used shared gres (shard/mps)
topology.
-- Allow Slurm to compile without MUNGE if --without-munge is used as an
argument to configure.
-- accounting_storage/mysql - Fix usage query to use new lineage column
instead of lft/rgt.
-- slurmrestd - Improve handling of missing parsers when content plugins
expect parsers not loaded.
-- slurmrestd - Correct parsing of StepIds when querying jobs.
-- slurmrestd - Improve error from parsing failures of lists.
-- slurmrestd - Improve parsing of singular values for lists.
-- accounting_storage/mysql - Fix PrivateData=User when listing associations.
-- Disable sorting of dynamic nodes to avoid issues when restarting with
heterogenous jobs that cause jobs to abort on restart.
-- Don't allow deletion of non-dynamic nodes.
-- accounting_storage/mysql - Fix issue adding partition based associations.
-- Respect non-"slurm" settings for I_MPI_HYDRA_BOOTSTRAP and HYDRA_BOOTSTRAP
and avoid injecting the --external-launcher option which will cause
mpirun/mpiexec to fail with an unexpected argument error.
-- Fix bug where scontrol hold would change node count for jobs with
implicitly defined node counts.
-- data_parser/v0.0.40 - Fix regression of support for "hold" in
job description.
-- Avoid sending KILL RPCs to unresolvable POWERING_UP and POWERED_DOWN nodes.
-- data_parser/v0.0.38 - Fix several potential NULL dereferences that could
cause slurmrestd to crash.
-- Add --gres-flags=one-task-per-sharing. Do not allow different tasks in to be
allocated shared gres from the same sharing gres.
-- Add SelectTypeParameters=ENFORCE_BINDING_GRES and ONE_TASK_PER_SHARING_GRES.
This gives default behavior for a job's --gres-flags.
-- Alter the networking code to try connecting to the backup controllers
if the DNS lookup for the primary SlurmctldHost fails.
-- Alter the name resolution to only log at verbose() in client commands
on failures. This allows for HA setups where the DNS entries are withdrawn
for some SlurmctldHost entries without flooding the user with errors.
-- Prevent slurmscriptd PID leaks when running slurmctld in foreground mode.
-- Open all slurmctld listening ports at startup, and persist throughout.
This also changes the backup slurmctld process to open the SlurmctldPort
range, instead of only the first.
-- Fix backup slurmctld shutting down instead of resuming standby duty if it
took control.
-- Fix race condition that delayed the primary slurmctld resuming when taking
control from a backup controller.
-- srun - Ensure processed messages are meant for this job in case of a
rapidly-reused TCP port.
-- srun - Prevent step launch failure while waiting for step allocation if
a stray message is received.
-- Fix backup slurmctld to be able to respond to configless config file
requests correctly.
-- Fix slurmctld crashing when recovering from a failed reconfigure.
-- Fix slurmscriptd operation after recovering from a failed reconfigure.
* Changes in Slurm 23.11.1
==========================
-- Fix scontrol update job=... TimeLimit+=/-= when used with a raw JobId of job
array element.
-- Reject TimeLimit increment/decrement when called on job with
TimeLimit=UNLIMITED.
-- Fix slurmctld segfault when reconfiguring after a job resize.
-- Fix compilation on FreeBSD.
-- Fix issue with requesting a job with --licenses as well as
--tres-per-task=license.
-- slurmctld - Prevent segfault in getopt_long() with an invalid long option.
-- Switch to man2html-base in Build-Depends for Debian package.
-- slurmrestd - Added /meta/slurm/cluster field to responses.
-- Adjust systemd service files to start daemons after remote-fs.target.
-- Add "--with selinux" option to slurm.spec.
-- Fix task/cgroup indexing tasks in cgroup plugins, which caused
jobacct/gather to match the gathered stats with the wrong task id.
-- select/linear - Fix regression in 23.11 in which jobs that requested
--cpus-per-task were rejected.
-- Fix crash in slurmstepd that can occur when launching tasks via mpi using
the pmi2 plugin and using the route/topology plugin.
-- Fix sgather not gathering from all nodes when using CR_PACK_NODES/--m pack.
-- Fix mysql query syntax error when getting jobs with private data.
-- Fix sanity check to prevent deleting default account of users.
-- data_parser/v0.0.40 - Fix the parsing for /slurmdb/v0.0.40/jobs exit_code
query parameter.
-- Fix issue where TRES for energy wasn't always set before sending it to the
jobcomp plugin.
-- jobcomp/[kafka|elastisearch] Print raw TRES values along with the
formatted versions as tres_[req|alloc]_raw.
-- Fix inconsistencies with --cpu-bind/SLURM_CPU_BIND and --hint/SLURM_HINT.
-- Fix ignoring invalid json in various subsystems.
-- Remove shebang from bash completion script.
-- Fix elapsed time in JobComp being set from invalid start and end times.
-- Update service files to start slurmd, slurmctld, and slurmdbd after sssd.
-- data_parser/v0.0.40 - Fix output of DefMemPerCpu, MaxMemPerCpu, and
max_shares.
-- When determining a jobs index in the database don't wait if there are more
jobs waiting.
-- If a job requests more shards which would allocate more than one sharing
GRES (gpu) per node refuse it unless SelectTypeparameters has
MULTIPLE_SHARING_GRES_PJ.