Skip to content
This repository
Newer
Older
100644 168 lines (134 sloc) 6.716 kb
0b8aa412 » BuzzTroll
2010-09-21 merged virga1 and master
1 ==========
2 Lantorrent
3 ==========
4
5 Lantorrent is a file distribution protocol integrated into the Nimbus
6 IaaS toolkit. It works as a means to multi-cast virtual machine images
7 to many backend nodes. The protocol is optimized for propagating
8 virtual machine images (typically large files) from a central repository
9 across a LAN to many virtual machine monitor nodes.
10
11 Lantorrent works best for the following scenarios:
12
13 1) large file transfers (VM images are typically measured in gigabytes)
14 2) local area switched network (typical for data center computer racks)
15 3) file recipients are willing peers.
16 4) many endpoints request the same file at roughly the same time
17
18 -----------------
19 Protocol overview
20 -----------------
21
22 When an endpoint wants a file it submits a request to a central agent.
23 This agent aggregates request for files so that they can be sent out in
24 an efficient single multi-cast session. Each request for a source file
25 is stored until either N request on that file have been made or N'
26 seconds have passed since the last request on that source file has been
27 made. This allows for a user to request a single file in several
28 unrelated session yet still have the file transfered in an efficient
29 multi-cast session.
30
31 Once N requests for a given source file have been made or N' seconds
32 have passed the destination set for the source file is determined. A
33 chain of destination endpoints is formed such that each node receives
34 from and sends to one other node. The first node receives from the
35 repository and send to a peer node, that peer node sends to another,
36 and so on until all receive the file. In this way all links of the
37 switch are utilized to send directly to another endpoint in the switch.
38 This results in the most efficient transfer on a LAN switched network.
39
40 Often times in a IaaS system a single network endpoint (VMM) will want
41 multiple copies of the same file. Each file is booted as a virtual
42 machine and that virtual machine will make distinct changes to that file
43 as it runs, thus it needs it own copy of the file. However that file
44 does not need to be transfered across the network more than once.
45 Lantorrent will send the file to each endpoint once and instruct that
46 endpoint to write it to multiple files if needed.
47
48 -----------------------------
49 Enabling Lantorrent in Nimbus
50 -----------------------------
51
52 Lan torrent is part of the nimbus distribution as of Nimbus 2.XXX.
53 However, due to system administrative overhead it is not enabled by
54 default. To enable Lantorrent in nimbus there are a few configurations
55 changes that must be made.
56
eed27d22 » priteau
2010-10-07 Add a note about dependencies
57 1) install Lantorrent dependencies
58 - Lantorrent is run out of xinetd thus it must be installed on all VMMs.
59 - Lantorrent depends on the Python simplejson module, and must be installed
60 on both the service node and the VMMs:
61 - on Debian, package python-simplejson
62
63 2) edit $NIMBUS_HOME/nimbus-setup.conf
0b8aa412 » BuzzTroll
2010-09-21 merged virga1 and master
64 change lantorrent.enabled: False -> lantorrent.enabled: True
65
eed27d22 » priteau
2010-10-07 Add a note about dependencies
66 3) edit $NIMBUS_HOME/services/etc/nimbus/workspace-service/other/common.conf
0b8aa412 » BuzzTroll
2010-09-21 merged virga1 and master
67 change the value of propagate.extraargs:
68 propagate.extraargs=$NIMBUS_HOME/lantorrent/bin/lt-request
69
70 be sure to expand $NIMBUS_HOME to its full and actual path.
71
eed27d22 » priteau
2010-10-07 Add a note about dependencies
72 3.1) restart the service: $NIMBUS_HOME/bin/nimbusctl restart
9385e484 » priteau
2010-10-06 Tell users to restart the service after changing the configuration
73
0b8aa412 » BuzzTroll
2010-09-21 merged virga1 and master
74 4) install lantorrent on VMM
75 - recursively copy $NIMBUS_HOME/lantorrent to /opt/nimbus/lantorrent.
76 - run ./vmm-install.sh on each node
77 either run it as your workspace control user or specify the workspace
78 control user as the first and only argument to the script.
79
80 4.1) install lantorrent into xinetd
81 - the vmm-install,sh script creates the file lantorrent.inet. This
82 file is ready to be copied into /etc/xinetd.d/. Once this is done
83 restart xinetd (/etc/init.d/xinetd restart).
84
495ba905 » BuzzTroll
2010-10-06 lantorrent patches so it will work with older versions of python
85 5) change the propagation method.
86 - edit the file:
87 $NIMBUS_HOME/services/etc/nimbus/workspace-service/other/authz-callout-ACTIVE.xml
88
89 and change:
90 <property name="repoScheme" value="scp" />
91 to:
92 <property name="repoScheme" value="lantorrent" />
93
94 6) [optional] if the path to nimbus on the workspace control nodes (VMMs)
0b8aa412 » BuzzTroll
2010-09-21 merged virga1 and master
95 is not /opt/nimbus you will also need to edit a configuration file on
96 all backends.
97
98 In the file:
99 <workspace control path>/control/etc/workspace-control/propagation.conf
100
101 make sure the value of:
102
103 lantorrentexe: /opt/nimbus/bin/ltclient.sh
104
3eb657a3 » priteau
2010-10-10 Fix typo
105 points to the proper location of you ltclient.sh script. This should
0b8aa412 » BuzzTroll
2010-09-21 merged virga1 and master
106 be a simple matter of changing /opt/nimbus to the path where you chose
107 to install workspace control.
108
109 ----------------
110 Protocol details
111 ----------------
112
113 requests
114 --------
115
116 transfer
117 --------
118
119 A transfer from one peer to another is performed for forming a TCP
120 connection. A json header is first sent down the socket describing the
121 transfer. It describes the files where data must be written, and the
122 other endpoints where the data must be sent. As data comes in the
123 server will store it to one or more files and forward it on to another
124 endpoint. All of this information is contained in the header. The
125 header has the following format:
126
127 {
128 host
129 port
130 length
131 requests = [ {filename, id, rename}, ]
132 destinations =
133 [ {
134 host
135 port
136 requests = [ { filename, id, rename } ]
137 block_size
138 }, ]
139 }
140
141 When the server forms a connection with the next endpoint it will form
142 another header by removing the first entry of the destination list and
143 using the values contained in it as root level values in the header
144 described above. If the destination list is empty then the server is the
145 last in the chain and no forwarding is needed.
146
147 After the header is sent the binary payload (the contents of the file)
148 is sent down the socket.
149
150 The 'requests' value of the json header contains destination
151 information. Each filename in the list is a file where data should be
152 written. The value of id is the id of the request for that file. The
153 value of rename is a boolean which determines if the file should be
154 first written to a temporary location and then moved to the final
155 location once all of the data is received. This feature is helpful in
156 monitoring file completion.
157
158 security
159 --------
160
161 Each connection to a lantorrent peer comes with a json header. Directly
162 following that header is a line of text in the following format:
163 EOH : <signature>
164
165 The value of signature is the hmac signature of everything that came
166 before the literal EOH. The password for that signature is found in the
167 lt.ini file. Every endpoint and the master repo must have the same
168 value for that password.
Something went wrong with that request. Please try again.