-
Notifications
You must be signed in to change notification settings - Fork 102
/
README.IB
66 lines (56 loc) · 1.89 KB
/
README.IB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
To use RDMA InfiniBand connections the following is needed:
1) The rdma_ucm kernel module must be loaded
2) The application must have read/write access to the IB device nodes
(typically /dev/infiniband/[rdma_cm|uverbs*])
3) The IP address used for the connection must be an address assigned to an
RDMA-capable device (e.g. IPoIB)
4) Shared memory must be sufficient for all RDMA connections,
2 * Global::IATTR_RDMA_RING_BUFFER_SIZE_MB are needed for each connection
set /dev/shm, kernel.shm[min|max|all] accordingly
5) The user must be able to lock the memory registered with verbs, such
that the locked memory limit needs to be sufficient ("ulimit -l" for
bash, "limit memorylocked" for csh). Updating /etc/security/limits.conf
with entries like this is usually adequate (e.g. to raise the limit
to 2GB for all users):
* soft memlock 2048000
* hard memlock 2048000
Ubuntu 11.04 Setup Notes
http://davidhunt.ie/wp/?p=375
Ubuntu 11.04:
blue:~/Software/Equalizer# cat /etc/network/interfaces
auto lo
iface lo inet loopback
auto ib0
iface ib0 inet static
address 192.168.1.1
netmask 255.255.255.0
auto ib1
iface ib1 inet static
address 192.168.1.2
netmask 255.255.255.0
blue:~/Software/Equalizer# cat /etc/udev/rules.d/99-infiniband.rules
KERNEL=="umad*", NAME="infiniband/%k"
KERNEL=="issm*", NAME="infiniband/%k"
KERNEL=="ucm*", NAME="infiniband/%k", MODE="0666"
KERNEL=="uverbs*", NAME="infiniband/%k", MODE="0666"
KERNEL=="ucma", NAME="infiniband/%k", MODE="0666"
KERNEL=="rdma_cm", NAME="infiniband/%k", MODE="0666"
blue:~/Software/Equalizer# cat /etc/modules
# /etc/modules: kernel modules to load at boot time.
#
# This file contains the names of kernel modules that should be loaded
# at boot time, one per line. Lines beginning with "#" are ignored.
lp
rtc
rdma_ucm
rdma_cm
ib_sa
ib_cm
ib_umad
ib_addr
ib_uverbs
ib_ipoib
ib_ipath
ib_qib
mlx4_ib
mlx4_core