forked from openucx/ucx
-
Notifications
You must be signed in to change notification settings - Fork 2
/
NEWS
1372 lines (1315 loc) · 57 KB
/
NEWS
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
#
## Copyright (c) NVIDIA CORPORATION & AFFILIATES, 2001-2021. ALL RIGHTS RESERVED.
## Copyright (C) UT-Battelle, LLC. 2014-2019. ALL RIGHTS RESERVED.
## Copyright (C) ARM Ltd. 2017-2021. ALL RIGHTS RESERVED.
##
## See file LICENSE for terms.
##
#
## Current
### Features:
### Bugfixes:
## 1.15.0 (September 28, 2023)
### Features:
#### UCP
* Added 2-stage pipeline protocol in the new protocol infrastructure
* Added reset and abort functionality of rendezvous protocols in the new infrastructure
* Added zero-copy rendezvous data send protocol in the new infrastructure
* Added support for user memory handle in the new protocol infrastructure
* Added option to force ODP registration for certain memory types
* Enabled lock free memory region deregistration
* Updated allow/deny transport list feature to control auxiliary transport selection
* Multiple performance improvements of the new protocol infrastructure
* Multiple improvements in error and debug messages
#### UCT
* Split UCT_MD_MKEY_PACK_FLAG_INVALIDATE into two flags for RMA and AMO
* Added put_zcopy and get_zcopy scheme support for self transport
* Added base implementation of is_reachable_v2 API using intra/inter flag
* Introduced MD capability for non-blocking registration memory types
#### RDMA CORE (IB, ROCE, etc.)
* Added implementation of is_reachable_v2 routine to IB interface
* Added option to control CQE zipping per CQ RX/TX direction
* Added option to specify how DCI selects port under RoCE LAG
* Added hw_dcs to the list of policies to select DCI by an endpoint
* Removed implicit on-demand paging
* Added option to set RoCE lag dct port for response under queue affinity mode
* Improved IB memlock limit logging
#### UCS
* Added ucs_string_buffer_rbrk() to split token
#### GPU (CUDA, ROCM)
* Added support for atomic reply_buffer on GPU memory
* Added system device information for AMD GPUs
* Improved performance estimation of gdr_copy transport
* Added a simplistic implementation of performance estimation of cuda_ipc transport
* Improved performance estimation of cuda_ipc on Hopper architecture
* Added rcache parameters for rocm transports
* Introduced dmabuf support for rocm transports
* Implemented asynchronous progress for the zcopy operations in the rocm_copy transport
* Added option to enable using cross-device dmabuf file descriptor for rocm
#### Java
* Added Java bindings for exported memh feature
#### Tests
* Added a rocm docker container for testing
* Added option to send client_id in iodemo test
* Added support for multiple connections to the same server in iodemo test
* Added synchronization before exit to hello world examples
#### Tools
* Added user-side memcpy option for AM benchmarks in ucx_perftest
* Added wireshark LUA dissectors for some UCX protocols
#### Build
* Added support for binutils 2.40
* Added versioned dependency to switch between packages with the same names
* Added a separate xpmem deb subpackage
* Added aarch64 support to the binary distribution pipeline
* Removed dependency on libnuma
### Bugfixes:
#### UCP
* Fixed assertion when sending from non-contiguous GPU buffer to managed buffer
* Fixed the race condition on endpoint configurations
* Fixed endpoint reconfiguration issues due to asymmetrical selection
* Fixed endpoint reconfiguration error due to wrong locality detection
* Fixed crash during connection manager cleanup
* Fixed rkey index calculation for rendezvous protocol
* Fixed rcache dump function
* Removed logging from rkey unpack in release mode
* Fixed dobule free of rkey in rendezvous protocol
* Fixed rendezvous pipeline protocol error flow
* Fixed error handling in rendezvous get zcopy protocol
* Replay pending requests of wireup EP CM during connection establishment to prevent potential ordering issues and wrong configuration
* Pass user-provided memory type to the function that checks whether the buffer can be sent inline or not
* Avoid memory registration during UCP context initialization
* Fixed CPU/device atomics selection in the new protocol infrastructure
* Multiple fixes in the new protocol infrastructure information output
#### UCT
* Added check for dmabuf kernel support in ROCm memory domain
* Fixed exported memh packing
* Fixed an error in checking return status of multi-threaded memory registration function
#### RDMA CORE (IB, ROCE, etc.)
* Fixed dma-buf based memory region registration
* Fixed memory handle data corruption when PCIe relaxed ordering is enabled
* Fixed performance degradation when indirect atomic key is not supported by the hardware
* Fixed remote access error to strict-order keys because of wrong offset
* Added check for UAR support to memory domain opening
* Fixed updating port counters for devx qp
* Fixed ibv_create_cq error message on node without Infiniband
* Fixed performance degradation due to using 2 paths on NDR400 by default
* Removed unnecessary async lock which otherwise would block UD progress
#### GPU (CUDA, ROCM)
* Fixed CUDA IPC performance degradation due to libnuma removal
#### UCS
* Fixed lane selection and added bandwidth estimation for Sapphire Rapids family
* Fixed displaying wrong environment variable suggestions
* Fixed VFS warning output
* Fixed SEGV in ucs_debug_backtrace_next(), upon previous SEGV handling, due to ENOMEM situation
* Fixed memory corruption when using UCX_MPOOL_FIFO=y
#### UCM
* Fixed conditional jump patching
* Fixed mremap() override
#### GPU (CUDA, ROCM)
* Fixed usage of dmabuf when the buffer is not page-aligned
* Removed async_cb from cuda_copy to avoid the issue with UCP worker async lock
#### Java
* Fixed leakage of jucx_request global references
#### Documentation
* Updated ucp_worker_release_address description
#### Tests
* Fixed wrong usage of ep_close in examples
#### Tools
* Fixed memory access flags in perftest
* Removed support for librte from perf
* Fixed worker flush deadlock when using multiple workers in ucx_perftest
#### Build
* Changed 'unsupported option' ICC command line warning to error
* Removed never used fault-injection configuration option
* Fixed obsolete macro warnings in new autoconf/libtool
* Fixed building UCX with GCC 13
* Fixed UCX RPM build on machines that have libxpmem-devel rpm from MLNX_OFED installation
* Fixed ucx-rdmacm package requirements
* Fixed compilation errors with armcc-22.1
* Fixed passing port number to goperftest
## 1.14.1 (May 22, 2023)
### Bugfixes:
* Fixed ROCm to prevent the locking of host pinned memory
* Added CUDA 12 based UCX builds to the release flow
* Increased the maximal number of endpoint configurations
* Fixed filter for a slow-lanes in selection logic
* Fixed TCP transport bandwidth calculation
* Fixed device detection for ROCM
* Fixed compatibility with CUDA 12
* Fixed rendezvous threshold for multi-path configurations
* Fixed error message in case of static link
* Fixed BlueField-3 detection
* Multiple fixes for Azure CI pipeline
## 1.14.0 (March 13, 2023)
### Features:
#### Core
#### UCP
* Added API for querying transport and device names on endpoint
* Added API for querying datatype object
* Added API for exporting and importing memory keys (no implementation yet)
* Added support for non-persistent active message header
* Added infrastructure to print protocols v2 performance
* Multiple performance improvements for protocols v2
* Added support for non-contiguous datatypes for rendezvous protocols v2
* Added support for reset and abort request in protocols v2
* Added support for user memory handles in RMA API
* Added multi-rail support for RMA API in protocols v2
* Added support for up to 16 different lanes per endpoint
* Added support for dmabuf memory registration in protocols v2
* Added strong fence mode for ucp_worker_fence() API
#### UCT
* Added new uct_md_mem_attach() API to support exported memory handles
* Added remote completion mode for endpoint flush (via new flag)
* Added support for dmabuf registration
* Added new uct_ep_connect_to_ep_v2() API
* Added new uct_mem_reg_v2() API
* Added new uct_md_query_v2() API
* Added support for IPv6 loopback address in TCP transport
#### RDMA CORE (IB, ROCE, etc.)
* Added ECE (enhanced connection establishment) support for RC and DC transports
* Added support for hardware DCS in DC transport
* Added UD interface and endpoint resource information to VFS
* Added CQ creation via DEVX API
* Removed support for accelerated IB transports over legacy experimental verbs
#### UCS
* Added support for auto-correction of user environment variables
#### UCM
* Implemented CUDA bistro hooks for aarch64 (to enable memory cache on this platform)
* Added support for CUDA virtual/stream-ordered memory with cudaMallocAsync
#### GPU (CUDA, ROCM)
* Implemented uct_iface_estimate_perf() function for ROCM
* Removed obsoleted ROCM gdr transport
* Added support for hsa async_copy for short operations in ROCM
* Added memory allocation functions in ROCM
#### Java
* Added methods for ucp_worker_arm() and ucp_worker_get_efd()
#### Documentation
* Added FAQ for using pkg-config tool to build applications with UCX
#### Tests
* Added prints of latency per connection in io_demo
#### Tools
* Added runtime library version to the 'ucx_info -v' output
* Added support for memory types in ucx_info
### Bugfixes
#### UCP
* Multiple fixes in keepalive protocol
* Multiple fixes and improvements in UCP rcache flows
* Fixed endpoints leak by disabling resolving remote endpoints in certain cases
* Multiple fixes and cleanups in wireup protocol and lanes selection flows
* Multiple fixes in protocols v2 infrastructure
* Fixed worker interface initialization taking atomic caps into account
* Fixed UCP AM max payload value calculation for protocols v2
* Fixed deadlock in rcache when UCX_LOG_LEVEL set to debug
* Fixed lanes weight calculation in rendezvous protocol v2
* Fixed user memory handle support in rendezvous protocol
* Fixed message split in rendezvous protocol to avoid having very small chunks
* Improved performance estimations for protocols v2
* Fixed receive descriptors leak in UCP AM rendezvous
#### UCT
* Fixed double free of server endpoint in TCP sockcm
* Updated KNEM bandwidth to be dedicated resource rather than shared
* Fixed race in CM when listener is destroyed during conn_req_cb invocation
* Updated default bandwidth value for memory mapper transports
* Disqualify posix transport if /dev/shm size is too small
* Disqualify KNEM transport if memory registration fails with it
* Fixed cuda detection (when cuda headers are not present, but nvml headers are)
#### RDMA CORE (IB, ROCE, etc.)
* Fixed device error handling (prevent coredump when iface is down/up)
* Multiple fixes in DC transport (error flows, flow control, etc)
* Multiple fixes and cleanups in UD transport
* Fixed MR registration (avoid atomic offset breaking region alignment)
* Fixed indirect key registration (avoid creating atomic KSM on top of relaxed-order key)
* Fixed thread domain usage for accelerated verbs transports
* Added print of a particular syndrome on DEVX function failures
* Fixed DEVX QP creation by setting proper ts_format attribute
* Decreased size of DC endpoint
* Fixed bandwidth calculation for RoCE LAGs
* Fixed port counters setting for DEVX QPs
* Fixed compile errors on SLES sp3
* Removed errors during md open in case of strict memlock limit
#### UCS
* Removed async_max_events limit (e.g. to support many concurrent TCP connections)
* Updated memory wc flush using DGH hint for ARM platform
* Fixed deprecation warnings because of <sys/fcntl.h> includes
* Added default bandwidth value for ZHAOXIN CPU
#### UCM
* Fixed segfault in malloc when compiled with -flto
#### GPU (CUDA, ROCM)
* Updated cuda_copy transport to use event fd instead of async callback
* Fixed ROCM IPC transport (use remote agent if available)
* Fixed clang compilation errors in CUDA copy transport
* Fixed ROCM memtype detection
* Improved performance estimation of CUDA copy transport
* Fixed send to self flows in ROCM
#### Documentation
* Updated GPU memory support section in FAQ
#### Tests
* Multiple fixes and improvements in unit tests
#### Tools
* Fixed MPI RTE send deadlock in ucx_perftest
#### Build
* Build Debian package with multi-thread support
* Fixed configure warning by using POSIX compliant sh syntax
* Multiple fixes for Debian package build
* Dropped support for Ubuntu16
## 1.13.1 (September 7, 2022)
#### Bugfixes
* Fixed flow control protocol in DC transport
* Fixed reordering of pending operations in DC transport
* Fixed relaxed order detection in IB transports
* Fixed build configuration and IB ops references
* Fixed bandwidth calculation during wireup phase
* Fixed TCP transport server port selection
* Minor fixes in CI testing
## 1.13.0 (July 7, 2022)
#### Features
##### Core
* Added new objects to VFS: local and remote address of endpoint, statistics of ucp_ep_create success/failure, failed/destroyed endpoints
* Added support for UCX static libraries
* Added profiling for rkey management routines
* PCIe relaxed order enabled by default for AMD CPUs
#### UCP
* Added API to pass pre-registered memory handle to UCP operations
* Added implementation of AM rendezvous protocol
* Added 2-stage pipeline rendezvous protocol for GPU
* Added support for fragment mem_type for v1 pipeline proto, disabled by default
* Added active message support for proto v2
* Added UCP memory registration cache
* Improved adaptive progress - deactivate iface when all p2p lanes are destroyed
* Added support for user memh in proto_v1
* Added support for selecting local address when creating a client endpoint
* Added option to limit GPUDirectRDMA size in rendezvous protocol, UCX_RNDV_MEMTYPE_DIRECT_SIZE
* Deprecated UCX_SOCKADDR_AUX_TLS configuration parameter
#### UCT
* Introduced API uct_md_mkey_pack_v2
* Introduced UCT iface features API
* Introduced max_inflight_eps parameter in perf_attr API
* Introduced UCT_SEND_FLAG_PEER_CHECK flag that forces checking connectivity to a peer
* Introduced UCX_RCACHE_PURGE_ON_FORK to enable/disable cleaning regions when application is forking
#### RDMA CORE (IB, ROCE, etc.)
* Introduced NDR autorecognition
* Introduced CQE zipping support
* Set the default MAX_RD_ATOMIC to maximum value supported by the hardware
#### ROCM
* Increased maximum number of HSA agents
#### UCS
* Added topo module infrastructure
* Added memtrack and rcache information to VFS
#### Tools
* Added support for pre-registered memory in ucx_perftest
* Added loopback transport support for UCT perf tests
### Bugfixes
#### Core
* Fixed not deallocating memory from ucp_mem_unmap if no rcache
* Fixed versioning infrastructure
* Multiple code improvements: refactoring, debug prints and assertions, etc.
* Multiple improvements in build, test and docs infrastructure
#### UCP
* Resolving remote EP ID when creating local EP disabled by default
* Multiple fixes in keepalive protocol
* Fixed initialization request send state if software RMA/AMO in use
* Fixed error handling in RMA and BW lanes selection logic
* Fixed CM wireup fallback
* Fixed occasional crash in finalize
* Fixed AM proto flags
* Fixed single zcopy proto initialization for AM
* Fixed proto v2 selection, take into account user header length
* Fixed selecting auxiliary transports when creating EP for sending EP_REMOVED
* Fixed printing invalid configuration
* Fixed allocation of indirect remote ID for internal EP if connected EP supports PEER_FAILURE
* Fixed memh allocation when no rcache
* Fixed protocol selection logic for UCP AM send
* Fixed error handling flow for EP discard requests from pending queue
* Fixed EP destroy flow
* Fixed rsc_index for prereg_md_map
* Fixed wireup error handling flow Create EP which send WIREUP_MSG/EP_REMOVED with AM lane only
* Fixed probe for multi-fragment eager
* Fixed alignment for AM rdesc init
* Fixed perf estimation for proto v2
* Fixed CM wireup with proto v2
* Fixed EP discard flow during fast-forward
* Fixed datatype issue in TAG send
* Fixed EP refcount overflow
* Fixed EP error handling flow
* Fixed wire compatibility in address unpacking
* Fixed ucp_ep_close_nb for failed endpoint when related requests have registered memory that should be invalidated
* Fixed fragmented proto v2
* Fixed UCP address v2 packing/unpacking and usage of seg_size
* Fixed purge requests on failed endpoint
* Fixed error handling of connecting p2p lanes during WIREUP phase
* Fixed UCP endpoint use after free
#### UCT
* Fixed ABI break of uct_ep_params_t
* Fixed common intra-node keepalive protocol
* Fixed a typo UCT_PERF_ATTR_FIELD_REMOTE_SYS_DEIVCE -> UCT_PERF_ATTR_FIELD_REMOTE_SYS_DEVICE
* Fixed potential crash on MD mem alloc
* Disabled PEER_FAILURE capability for XPMEM
* Updated TCP iface bandwidth calculation taking into account PCI bandwidth
#### RDMA CORE (IB, ROCE, etc.)
* Fixed 2G aligned MR registration
* Fixed FC_HARD_REQ resending
* Fixed remote access to invalidated MR
* Fixed max_rd_atomic_dc value for DV
* Fixed DC handshake logic
* Fixed error handling flows
* Fixed flush(CANCEL) with UD and DC transports
* Fixed multi-path handling for passive endpoint with UD transport
* Fixed attributes for DV QP creation
* Fixed device query
* Fixed memory leak in case of disabling RDMA transport
* Fixed dci->pool_index initialization
* Fixed fallback if port speed not detected
* Fixed tag offload recv for inlined data
* Fixed PKEY index initialization
* Disabled mlx5 ifaces on verbs MD
#### TCP
* Fixed flush(CANCEL)
* Fixed close protocol when UCT EP pairs have only RX capability
* Fixed query local/remote saddr
#### GPU (CUDA, ROCM)
* Fixed a bug in invalidating address range in CUDA_IPC
* Fixed CUDA context caching and cleanup
* Fixed ROCM initialization
* Fixed ROCM components compilation
* Fixed IPC tls reachability check
* Fixed ROCM memory type detection
* Use ROCM remote_agent if available
* Fixed CUDA module compilation with clang 13
* Fixes in ROCm memory detection and performance estimation
#### KNEM
* Fixed memory registration cost
#### UCM
* Fixed potential hang on init
#### UCS
* Fixed name shadow problem in CentOS6.x
#### Tools
* Print stream API limits and handle stream feature in ucx_info
* Replaced ucp_ep_close_nb by ucp_ep_close_nbx in examples
* Replaced completed field by checking UCS status in io_demo
#### JAVA
* Throw exception if ucp_mem_query failed
#### GO
* Disabled go bindings in rpmbuild
* Fixed configure behavior if can't find go compiler
* Standalone performance benchmark
* Increased port range + make it dependent on agent_id
* Check compiler minimum version
* Set GOCACHE to a local directory that is cleared for each job in CI
* Disabled module for goperftest
* Fixed OOS build
## 1.12.1 (March 21, 2022)
#### Bugfixes
* Fixed memory hooks for Cuda 11.5
* Fixed memory type cache merge
* Fixed continuously triggering wakeup fd when keepalive is used
* Fixed memtype cache fallback when memory hooks are not installed
* Fixed parsing header flags of worker address
* Fixed pipeline protocol when sending from host memory to GPU memory
* Fixed transport progress not deactivated when all transport's connections are closed
* Fixed progress loop in io_demo application
* Fixed ROCm segfault when using internal_ops functions
* Fixed ROCm memory hooks
* Fixed performance regression on A64FX
* Fixed DCT create failure with rdma-core v22
* Fixed golang bindings build
* Fixed .deb package build on Ubuntu 22.04
* Fixed build on archlinux
#### Important changes
* If Cuda memory hooks on driver API cannot be installed, memory type cache and
memory registration cache will be disabled. This may lead to lower performance
of some applications on setups with NVIDIA GPUs, even if Cuda memory is not
being used. Prior to this change, failing to install driver API hooks could
lead to runtime errors or data corruption when Cuda memory is used and linked
statically with cuda runtime.
In order to revert to previous behavior (when the application is linked
dynamically with cuda runtime), the user can set UCX_MEM_CUDA_HOOK_MODE=reloc.
See more info in https://github.com/openucx/ucx/pull/7865.
## 1.12.0 (January 12, 2022)
### Features:
#### Core
* Added beta-level support for Go language bindings
* Added new objects to VFS (md, component, log_level, etc.)
* Added configuration variable to specify which loadable modules are allowed
* Added build-time configuration to disable sigaction overriding
#### UCP
* Added client_id to ucp_worker_create() and ucp_conn_request_query() APIs
* Added ucp_worker_address_query() API
* Updated ucp_ep_query() API for getting local and remote addresses
* Added address versioning to correctly preserve wire compatibility starting from version 1.11.0
* Added new client/server connection establishment packet header format
* Enabled rendezvous and tag sync protocols when error handling is enabled on the endpoint
* Added iov zcopy support to RMA operations
* Reduced memory usage of unexpected messages by fitting receive buffer size to packet size
* Added support for modifying UCT and UCS configs by ucp_config_modify() API
* Optimized unpacked rkeys memory consumption
* Added request flag to influence latency vs. bandwidth protocol
* Reduced memory management overhead with new protocols
* Improved performance calculations for new protocols
* Added AMO support with GPU memory target using new protocols
* Added put_zcopy, get_zcopy and pipeline based rendezvous in new protocols
* Added support for user-defined alignment in Active Messages
* Added support for offload tag sync in new protocols
* Updated ucp_atomic_post() to use NBX flow
#### UCT
* Added API - uct_iface_is_reachable_v2()
* Added IPv6 address support in TCP
* Added latency estimation to uct_iface_estimate_perf()
* Adjusted knem and cma overhead cost
* Increased built-in TCP keep-alive interval to 2 seconds
#### RDMA CORE (IB, ROCE, etc.)
* Added detection of IB NDR devices
* Added check for CQ overrun in assert mode
* Added bitmap usage for releasing detached DCIs
* Added configuration for requests ack frequency with DevX
* Added remote QP info to tx error CQE traces
#### UCS
* Added API for a per-process aggregate-sum statistics report
* Added memory pool set data structure
* Added new ptr_array API for bulk allocation
* Added ucs_string_buffer_append_flags() for string buffer
* Added ucs_ffs32()
* Added ucs_vsnprintf_safe() which always adds '\0'
* Added thread-safe put to ptr_map
* Improved accuracy of the topology distance estimation
* Added prints of leaked callbacks from the callback queue
* Removed a diagnostic message when fuse thread is stopped
* Added configurable limit for the memory consumed by rcache
* Added configuration for VFS(FUSE) thread affinity
* Added memory limit support to memtrack
#### CUDA
* Added global memtype cache to allow UCT transports to query memory attributes
* Auto-register CUDA whole allocations to avoid repeated registration costs
* Added capability to select CUDA stream based on source and destination memory type
(required for device memory based pipelining)
* Added selection of CUDA-IPC capabilities based on NVLINK topology
(to prefer writes vs. reads for specific platforms using NVML)
* Added option to set cuda_copy bandwidth
* Added profiling of CUDA runtime function calls
* Added option to limit GPUDirectRDMA size in rendezvous protocol
#### Java
* Added ucp_listener_reject functionality
* Added support for setting worker id and querying it from the connection request
* Added support to bind on a free port in UcpListener
#### Packaging
* Added cmake config files for better integration with external cmake based projects
#### Tests
* Removed memcpy from AM eager flow in io_demo
* Added check_qps.sh script to detected stuck QPs
* Improved diagnostic in test_init_mt
* Added iov support in ucp_client_server
* Added option to use epoll in io_demo
* Added registration of memory allocated by io_demo in memtrack
* Extended statistics in io_demo
* Improved logging in io_demo
* Replaced rand by urand in io_demo
* More improvements in io_demo
* Generalized median calculation to support any percentile in ucx_perftest
#### Tools
* Added loop-back transport support in ucx_perftest
* Split ucx_perftest into separate modules
* Added process placement option for ucx_info
* Extended parameters correctness check in ucx_perftest
* Added support for GPU memory RMA and atomics in ucx_perftest
#### CI
* Updated gtest 1.7 to 1.10
* Increased uptime in network corrupter (used for io_demo)
* Enabled set of gtests for new protocols
* Added running CI in docker containers
* Increased thresholds for test_ucp_wait_mem
* Added test for ucx binary compatibility between OS versions
* Increased test job timeout to 6 hours
* Reduced testing time under valgrind
* Added suppressions for glibc and libnl leaks
* Relaxed performance requirements in perf test
### Bugfixes
#### Core
* Fixed invalid remote memory access after connection error
* Fixed creating more than 64K endpoints between the same peers
* Fixed simultaneous endpoint close with ucp_hello_world
#### UCP
* Fixes and improvements in new protocols infrastructure
* Fixes in AM flows
* Fixed tag short threshold selection
* Multiple fixes in keep-alive protocol
* Multiple fixes in wire-up protocol
* Fixes in error flow during rendezvous protocol
* Multiple fixes in general error flow
* Fixed fallback to PUT pipeline in rendezvous protocol
* Reduced default value of keep-alive interval to 20 seconds
* Fixes in tag_send datatype processing
#### UCT
* Fixed keep-alive protocol for intra-node transports (sm, cuda)
* Fixed deadlock in TCP
* Suppressed EHOSTUNREACH error in TCP sockcm
* Restricted connecting loop-back to other devices in TCP
#### RDMA CORE (IB, ROCE, etc.)
* Fixed pkey_index initialization when creating RC QP with DEVX
* Disabled MP_SRQ by default
* Fixed TX WQ overflow check
* Fixed dci->pool_index initialization when HAVE_DC_DV is false
* Fixed syndrome value for creating rdmacm reserved qpn
* Fixed error code on rdma_establish failure
* Fixed uct_ep_am_short_iov for UD verbs
* Fixed handling of error CQE after rc_ep is destroyed
* Fixes in flow control when error CQE is polled
* Multiple fixes in RC and DC error flows
* Fixed deadlock between DCIs and RDMA_READ credits
* Removed AM handler invocation for PURE_GRANT messages
* Fixed endpoint arbiter_group leak in DC
* Fixed resource check in flush for DC
#### UCS
* Fixed segmentation fault for ucs_stats_parser
* Fixed potential crash on cleanup when use UCX profiling
* Fixed read_profile print of new request
* Fixed uninitialized variable access in VFS
* Changed log level of inotify_init failure to diag
* Fixed integer overflow in mpool chunk allocation
#### Packaging
* Fixed with-fuse arg for RPM build
#### Documentation
* Fixes in UCP, UCT, UCS, FAQ and README documentation
#### Tests
* Multiple fixes in io_demo
#### CI
* Fixed snapshot docker name
* Fixed hipMallocManaged hook gtest
* Fixes in Azure release pipeline
* Fixes in Coverity CI
* Fixed test_uct_query gtest for ROCm
* Fixes in jenkins test script
* Fixed release commit title check
## 1.11.2 (September 30, 2021)
### Bugfixes
* Fixes in Java release pipeline
* Fixes in handling large number of devices
* Fixes in UD out-of-order processing
* Fixes in switching transports during client/server connection setup
* Fixes in transport-level error reporting
## 1.11.1 (August 31, 2021)
### Features:
#### UCS
* Added API to read boot ID value or use machine_guid
### Bugfixes:
* Fixes in Cuda memory hooks
* Fixes in setting traffic class for DCT RoCE transport
* Fixes in TCP endpoint flush
* Fixes in TCP pending operations progress
* Fixes in release pipelines
* Fixes in error handling flow
* Fixes in multi-threaded tag probe
* Fixes in TCP disconnect flow
* Fixes in RPM post-install script
* Fixes in UCT common keepalive
## 1.11.0 (July 26, 2021)
### Features:
#### Core
* Added support for UCX monitoring using virtual file system (VFS)/FUSE
* Added support for applications with static CUDA runtime linking
* Added support for a configuration file
* Updated clang format configuration
#### UCP
* Added rendezvous API for active messages
* Added user-defined name to context, worker, and endpoint objects
* Added flag to silence request leak check
* Added API for endpoint performance evaluation
* Added API - ucp_request_query
* Added API - ucp_lib_query
* Ported connection manager to a new UCT API
* Added bandwidth optimizations for new protocols multi-lane
* Added support for multi-rail over lanes with BW ratio >= 1/4
* Added support for tracking outstanding requests and aborting those in case of connection failure
* Refactored keep-alive protocol
* Added device id to wireup protocol
* Added support up to 128 transport layer resources in UCP context
* Added support CUDA memory allocations with ucp_mem_map
* Increased UCP_WORKER_MAX_EP_CONFIG to 64
* Adjusted memory type zcopy threshold when UCX_ZCOPY_THRESH set
* Refactored wireup protocols, rendezvous, get, zcopy protocols
* Added put zcopy multi-rail
* Improved logging for new protocols
* Added system topology information
* Added new protocols for eager offload protocols
#### UCT
* Extended connection establishment API
* Added active message AM alignment in iface params
* Added active message short IOV API.
* Added support for interface query by operation and memory type
* Added API to get allocation base address and length
* Added md_dereg_v2 API
#### UCS
* Added log filter by source file name.
* Added checking for last element in fraglist queue
* Added a method to get IP address from sockaddr.
* Added memory usage limits to registration cache
#### UCM
* Improved x86 parser to recognize some mov flavors
#### CUDA
* Added registration for whole CUDA allocations
* Added CUDA-IPC keepalive
* Adjusted performance estimations
* Added Improve logging
* Added allocation methods for CUDA pinned/managed memory
* Added support for a global cuda_ipc cache
#### RDMA CORE (IB, ROCE, etc.)
* Added report of QP info in case of completion with error
* Refactored of FC send operations
* Added support for DevX unique QPN allocation
* Optimized endpoint lookup for DCI
* Added support for RDMA sub-function (SF)
* Added support for DCI via DEVX
* Added DCI pool per LAG port
* Added support for RoCE IP reachability check using a subnet mask
* Added active message short IOV for UD/DC/RC mlx, UD/RC verbs
* Added endpoint keep alive check for UD
* Suppressed warning if device can't be opened
* Added support for multiple flush cancel without completion
* Added ignore for devices with invalid GID
* Added support for SRQ linked list reordering
* Added flush by flow control on old devices
* Added support for configurable rdma_resolve_addr/route timeout
#### Shared memory
* Added active message short IOV support for posix, sysv, and self transports
#### TCP
* Added support for peer failure in case of CONNECT_TO_EP
* Added support for active message short IOV
#### Java
* Added full support for UCP Java API
#### Tests
* Added length/mem_type for UCP client server example
* Added port sockaddr tests for a new API
* Added test send-recv between client/server with diff UCX_IB_NUM_PATHS
* Added support for CUDA and CUDA managed memory in io_demoo
* Added support for a custom watchdog timeout from command line
* Extended memtype hook tests
#### Tools
* Added UCP active message support to perftest
* Added error handling option to perftest
* Added wakeup option
* Added performance tests for am short iov
#### CI
* Added RHEL 7.6 with MOFED 4.7
* Added Fedora 34, RHEL 7.2, 7.4
* Added PGI support from HPC-SDK module
* Added docker image with CUDA 11.2
* Added IODEMO test
* Added Ubuntu 20.4
* Added test for connection manager fallback in client-server testing
* Added loopback interface for tcp testing
### Bugfixes:
#### Build
* Fixes in libnuma detection macro
* Fixes for cross compilation support
* Fixes for --without-dc compilation
#### Continues Integration
* Fixes in Azure pipeline build system
* Fixes in Coverity CI
* Fixes in Azure release pipeline
#### Packaging
* Fixed in DEB package - added essential system dependencies
#### Documentation
* Fixes in UCP, UCT, Readme, FAQ, and Read-the-docs documentation
#### Tests
* Fixes in CMA peer failure test
* Fixes in SRQ tests
* Fixes in the usage requests_wait
* Fixes in test_uct_query
* Fixes addressing race conditions on client user data in test_uct_sockaddr
* Fixes in IODEMO app
* Fixes in error handling flow for perftest
* Fixes in perftest batch tests
* Fixes addressing hang issues for rendezvous protocol in UCP client server example
#### UCP
* Fixes in endpoint error handling
* Fixes in error reporting failed CM lanes
* Fixes in progress worker flush
* Fixes in rendezvous pipeline flow
* Fixes in recursive protocol selection
* Fixes in error handling for AM_ZCOPY
* Fixes in length check condition in RMA PUT short
* Fixes in failure handling rendezvous offload send
* Fixes in offload completion with inlined data
* Fixes in statistics calculations for rendezvous protocol
* Fixes in ucp_worker_query() thread mode for SERIALIZED
* Fixes preventing leaks of UCP requests
#### ROCM
* Fixes in device memory registration and de-registration
* Fixes in missing mem_query definition for rocm_copy
* Fixes addressing build failure due to const violation
* Fixes in sockaddr_accessibility test for rocm_copy and rocm_ipc
* Fixes in bandwidth estimation for rocm_ipc
#### RDMA CORE (IB, ROCE, etc.)
* Fixes addressing deadlock between DCI resources and RDMA_READ credits
* Fixes in DSCP for RoCE DCT
* Fixes in flush(cancel) flow
* Fixes preventing segfault in uct_rdmacm_cm_ep_str
* Fixes in scatter-gather entries logging
* Fixes for compilation with experimental verbs
* Fixes in UD dgid filtering
* Fixes in domain resources destroying
* Fixes in PCIe bandwidth calculation
* Fixes addressing CQ creation failure using legacy ibv API
* Fixes in iov2sge converter
* Fixes in port width check on HDR100
* Fixes in SL selection
* Fixes in hardware tag matching compilation
* Fixes in uct_rdmacm_cm_cqs hash key
* Fixes for compilation with rdma-core 20
#### Java
* Fixes in tag sender mask
#### UCT
* Fixes in reachability of loopback ifaces
* Fixes addressing possible uninitialized memory accesses
* Fixes in error flow for endpoints created upon receiving connection request
* Fixes in TCP keepalive to avoid false-positive error detection
#### UCM
* Fixes addressing heap corruption caused by ucp_set_event_handler()
* Fixes in mmap events test
## 1.10.1 (May 12, 2021)
### Bugfixes:
* Fixes in Infiniband port speed detection for HDR100
* Fixes in building gtest-all.cc and sock.c with GCC11
* Fixes addressing performance degradation with cuda memory on a self endpoint
* Fixes in JUCX listener connection handler
* Fixed in configuration of loopback TCP transport (disable by default)
* Fixes in RPM dependency on libibverbs
* Fixes in ABI backward compatibility for active message protocol
* Fixes in the DC transport - adding support for full-handshake mode (off by default)
* Fixes in Active Messages short reply protocol
* Fixes for segmentation fault while listening for connections
## 1.10.0 (March 9, 2021)
### Features:
#### Core
* Added support for Nvidia HPC SDK
* Added support for latest PGI and Clang
* Added support for ROCM-3.7+ (warning generated if older version detected)
* Added support for GCC11
#### Architecture
* Added Arm SVE memcpy()
* Redesigned Arm WFE support
* Improved clear_cache performance for Arm
* Added architecture detection for Zhaoxin CPU
#### CI
* Added release builds on CUDA 11
* Enabled performance validation in gtest
* Added new OS for release CI
#### UCP
* Added locality awareness to the transport selection logic for GPU devices
* Added put/offload/short and put/offload/zcopy protocols
* Added receive message nbx routine
* Reworked AM implementation and API, which adds support for RNDV semantics
* Added support for multi-lane connection manager over TCP
* Added support for printing AM tls with info log level
* Implement flush and destroy for UCT EPs on UCP worker
* Reduced UCP request size
* Added support for keepalive protocol
* Added support for multi-fragment protocol
* Added implementation for protocol progress for eager, bcopy, and multicopy
* Improved selection logic for protocol selection
* Added new protocols for UCP get operation
* Added bcopy protocols with support for GPU memory
* Added RNDV protocol implementation for GPU devices (CUDA, ROCm)
* Set SOCKADDR_CM_ENABLE=y by default
* Added support for fast-path short with new tag protocols
* Added a new parameter to control the CM listener's backlog
* Added support sending AM RTS over short message protocol
* Added support for shared memory multi-lane when CM is used
* Added missing async locks
#### UCT
* Added API for keepalive_timeout value
* Added add uct_completion.status
* Allowed transports to access multiple mem_types
* Removed status arg from uct_completion_callback_t
* Restructured uct_mem_alloc/uct_md_mem_alloc to use mem_type
* Updated documentation for uct_listener_params
* Lowered the log level for certain network errors
* Added cuda_copy wakeup feature
* Added wakeup support for shared memory
#### UCS
* Added "inf" and "auto" values to time units
* Added on-stack constructors for array and string buffer
* Added ucs_ptr_map_t data structure
* Added bool CSWAP
* Improved logging
* Added optimization for namespace processing
* Fixes for connection matching functionality
#### CUDA
* Added support for global IPC cache
#### RDMA CORE (IB, ROCE, etc.)
* Added support for auto detection of adapative routing settings
* Added an option to poll TX CQ every progress iteration
* Added local and remote addresses to the reject error message
* Added support for UAR allocation with non-cacheable memory type
* Added support for multiple flush cancel without completion
* Added async events callback support
* Added detection for ConnectX-6, ConnectX-7 and BlueField-1/2 devices
* Added support for connection matching for UD
* Added a check for AM ordering
* Added better support for non-4K MTU values
#### Java (preview)
* Added support for a different javadoc executable path for different java versions
* Added UCS memory type constants
* Added support build on Java10+
* Added support for io-vector datatype.
* Removed libjucx from packages.
#### Tests
* Added CI for CUDA 11
* Added test_ucp_sockaddr_protocols.stream_short
* Reimplemented tests using NBX API
* Added flush(cancel) test
* Added memory_wait mode to perftest
* Added support for clang 10
* Refactored RMA and atomic tests, add memtype support
* Added test for uct_md_mem_query()
* Added request interrupt support
* Added support for connection manager fallbacks
* Added new ucp request test checking for leaks from the ptr_map
#### Documentation
* Added glossaries
### Bugfixes:
#### Portability
* Fixes in print functions to use format string like PRIx64, etc.
* Fixes for Arm v8 cross compilation support
#### Continues Integration:
* Fixes in Github release flow
* Fixes in docker image
#### Packaging
* Removed deb package dependencies
* Fixes in SPEC to make the RPM relocatable
#### Documentation
* Fixes in documentation for ucp_am_recv_data_nbx
* Fixes in quick start example
* Fixes in installation instruction
* Fixes in updates in author list
#### Tests
* Fixes for failures under valgrind runtime
* Fixes in mmap tests for 0-length RMA
* Fixes in definition of LAST_WQE wait timeout
* Fixes in ROCm for mem_buffer test
* Fixes in test name printing format
* Fixes in tcp_sockcm test
#### UCP
* Fixes in worker cleanup flow
* Fixes in RNDV RTS flow
* Fix in length check condition for RMA PUT short
* Fixes in handling failures from AM Bcopy
* Fix in a release flow of deferred data
* Fixes for invalid ID and handling of status in RNDV
* Fixes in short active message reply protocol
#### CUDA
* Fixes in managed memory support
* Fixes in topology detection
#### RDMA CORE (IB, ROCE, etc.)
* Fixes in assert definitions
* Fixes in printing an error about invalid AM Bcopy length for UD
* Fixes for thread safety support
* Fixes to get ROCE device name according to GID
* Fixes for SL selection
* Fixes in create STRICT_ORDER key
* Fixes addressing performance degradation in UD transport due to excess async events
* Fixes in QP destroy
* Fixes for CQ creation failure using old Verbs API
#### UGNI
* Fixing disable logic in config
* Fixing clang 11 warnings
#### Java
* Fixes in build dependencies
* Fixes in constructing UcpRequest object on error
* Fixes in exception handling on endpoint closure request
* Fixes for segfault in UcpErrorHandler
#### UCP
* Fixes in datatype support for get_zcopy RNDV
* Fixes in connection manager disconnect
* Fixes in assert definitions
* Fixes in completion flow for failed EP
* Fixes in flush error handling flow
* Fixes in latency calculations for wireup protocol
* Fixes in offload completion with inlined data
* Fixes in unpacking flow
* Fixes in error handling for various protocols
#### UCT
* Fixes in flush TX
* Fixes in checks for enabling GPU Direct RDMA
#### UCS
* Fixes for crashes on incorrect value set in config
* Fixes in ptr_array
* Fixes in maximal size for ucs_snprintf_safe()
* Fixes in compilation warning
* Fixes in ucs_aarch64_dsb(_op) definition
#### TCP
* Fixes in default route interface confirmation flow
* Fixes in PUT protocol
* Fixes in max connection limit and improved error reporting
#### UCM
* Fixing crash on prevent unload
* Fixes in libucm_rocm
* Fixes for few racing conditions
## 1.9.0 (September 19, 2020)
### Features:
#### UCX Core
- Added a new class of communication routines '*_nbx' that enable API extendability while
preserving ABI backward compatibility
- Added asynchronous event support to UCT/IB/DEVX
- Added support for latest CUDA library version
- Added NAK-based reliability protocol for UCT/IB/UD to optimize resends
- Added new tests for ROCm
- Added new configuration parameters for protocol selection
- Added performance optimization for Fujitsu A64FX with InfiniBand
- Added performance optimization for clear cache code aarch64
- Added support for relaxed-order PCIe access in IB RDMA transports
- Added new TCP connection manager
- Added support for UCT/IB PKey with partial membership in IB transports
- Added support for RoCE LAG
- Added support for ROCm 3.7 and above
- Added flow control for RDMA read operations
- Improved endpoint flush implementation for UCT/IB
- Improved UD timer to avoid interrupting the main thread when not in use
- Improved latency estimation for network path with CUDA
- Improved error reporting messages
- Improved performance in active message flow (removed malloc call)
- Improved performance in ptr_array flow
- Improved performance in UCT/SM progress engine flow
- Improved I/O demo code
- Improved rendezvous protocol for CUDA
- Updated examples code
#### UCX Java (API Preview)
- Added support for UCX shared library loading from both classpath and LD_LIBRARY_PATH
- Added configuration map to ucp_params to be able to set UCX properties programmatically
### Bugfixes:
- Fixes for most recent versions of GCC, CLANG, ARMCLANG, PGI