-
Notifications
You must be signed in to change notification settings - Fork 0
/
Portable_datacenter.mw
170 lines (132 loc) · 9.69 KB
/
Portable_datacenter.mw
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
= Target =
Have fix url or access, included with change of internet provider, or have multiple internet connexion. This include for other service than http, like: mail (pop, smtp), irc, xmpp, ... Have access to site web via the normal web thanks to inproxy (XXXXXX.b32.i2p.us), and less sensible to dns or routing security leak.
Have decentralised service, with minimum latency, when you can't/wan't dynamic dns and ipv6. Or multiple same service on same port, on you network. You can route 2 server on port 80 without change on internet port or have reverse proxy (haproxy and i2p).
= Topologie and installation =
[[File:personnal-network-topologie.png|left|thumb]]
[[File:nagios-statusmap.png|thumb]]
[[File:nagios-map-4.1.png|left|thumb]]
[[File:photo-portable-datacenter.jpg|thumb]]
[[File:nagios-bi-datacenter.jpg|left|thumb]]
On toc and utilite:
<code><pre>*nat
:PREROUTING ACCEPT [966:55985]
:INPUT ACCEPT [32:2024]
:OUTPUT ACCEPT [19:1212]
:POSTROUTING ACCEPT [0:0]
-A POSTROUTING -o ethToInternet -j MASQUERADE
COMMIT
*filter
:INPUT ACCEPT [112:7798]
:FORWARD DROP [1:60]
:OUTPUT ACCEPT [0:0]
-A FORWARD -m state --state RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -s 192.168.142.0/24 ! -d 192.168.142.0/24 -i ethLocalNetwork -j ACCEPT
# for utilite only
-A FORWARD -s 192.168.0.0/24 -d 172.16.0.0/16 -i ethToInternet -j ACCEPT
COMMIT
</pre></code>
And:
<code><pre>sysctl net.ipv4.ip_forward=1</pre></code>
On toc:
<code><pre>route add -net 172.16.0.0 netmask 255.255.0.0 gw 192.168.0.2 ethToInternet</pre></code>
On '''utilite''', '''deny the access to 192.168.X.X''', only forward to internet. But from 192.168.142.X will access to 192.168.0.X, 172.16.X.X and internet.
== Kernel ==
=== Vanilla kernel ===
The reason to have an vanilla kernel:
* Have the last fix for the kernel, most of ARM vendor ship an kernel and only update their part, mean forget all the security fixes since it have snapshoted their version
* Have access to patch as vserver, Grsec, ... the repatched ARM kernel is mostly not capable of this and their new driver not ship the corresponding part. In case of Grsec, some drivers have memory problem reveled with grsec and then not boot
* Kernel 4.1+ is highly recommander for LXC
=== Modern kernel ===
The modern kernel:
* Support better the ARM arch, for example by converting more assembly code to C code to be more same and have same behavoiur on all platform
* Better ARMv8
* More lock less code to greatly improve the performance on hardware without barrier/lock hardware (mean: ARMv6, ARMv7, ARMv8, support is included into: ARMv8.1+)
* More code optimisation to have less system time
= Income route =
To route the income connexion, for http I use haproxy for normal ip trafic. But I route via i2p too for secure and more general routing (http, MMORPG server, ...).
= Backup =
The backup is snapshoted by btrfs, rsync to be exported, and keep as snapshot to have backup diff as high efficienty and compressed backup
= Hot switching =
Hot switching is allowed by lxc 1.1 to migrate transparently the vm from one hardware to other.
LXC 2 give:
* More consistent user experience between the various LXC tools.
* Improved checkpoint/restore support.
* Complete rework of our CGroup handling code, including support for the CGroup namespace.
* A massive amount of bugfixes.
That allow physical mobility:
[[File:Datacenter-truck.jpg]]
[[File:Decentralised-datacenter.jpg|800px]]
[[File:Bolivia-load.jpg]]
* Place geographically where you have the best bandwith for datacenter and customer
* Easy close datacenter node
* Equilibrate the load on the hardware
* Nearest database to their site
= Routing and bandwidth =
== Advanced router ==
Standard level 2 switch is used, not administrable, then how do you protect? By the virtualisation layer, it force the ip and mac address, route de packet, do some firewall filter.
== Bandwidth ==
[[File:network-link-bandwidth.png]]
The first case is over sized network, the second case is similar network.
=== The over sized network ===
It's transparent seam less bandwidth. The final server customer don't need care of the server bandwidth because the top level network is over sized.
The customer don't think before buy. Mean if the need it's just have load balancing, it take 2 server anywhere, producing over consuming of top level network as the backbone. Same for server for http + php and another to database.
'''Advantage:''' No over loaded top level network. Effective for high availability strategies.
'''Disavantage:''' More cost for the customer (more expensive hardware and over usage of bandwidth). Tend to random place of server, mean higher latency.
=== The similar network ===
You have the plain bandwidth in local, and slow on server into another rack. The final customer need think where place it.
The customer need studie few bit the server topologie in case of multiple dependent server.
'''Advantage:''' Lower price, more accessible technologies
'''Disavantage:''' Lower bandwidth between server of different place. Need compression and take care for high availability strategies.
= Links =
* [[Latency and RTT in Bolivia]]
* [http://catchchallenger.first-world.info/other/proposition-internet-bolivia.odp Proposition internet bolivia]
= More advanced need =
More advanced need for complex structure.
[[File:actual-need-network-route.png|thumb]]
[[File:final-network-route.png|thumb]]
== Temperature cooling ==
This is the result of overall cluster temperature of each node after 2h at idle (minor burning time) at 14C ambiant temperature without cooling.
[[File:datacenter-temperature-without-cooling.png]]
Extracted form internet:
[[File:odroid-temperature.jpg]]
== Power consumption ==
[[File:professional-pannel-solar-rpi.jpg|left|thumb]]
The power consumption on 5V line is here with kernel 3.14.77+ and connected with 1000Mbps is:
* With governor performance at idle: 0.32A
* With governor performance at 100% of CPU: 0.67A
* With governor ondemand at idle: 0.29A
* With governor ondemand at 100% of CPU: 0.65A
* With governor powersave at idle: 0.29A
* With governor powersave at 100% of CPU: 0.30A
* With governor powersave at 100% of network: 0.315A
= Datacenter in bolivia =
More info on: [[Datacenter in bolivia]]
I have done the electrical power bus, it's 2*23 Odroid C2, 1000Mbps. I used 46 MicroSD of 64GB.
Power used (mesured at the 220V in Seasonic serie G power supply): '''56W''', disconnected power supply: 7W. 49W at 100Mbps. Peak: 111W
When all is working and used, I thinks buy 1000 Odroid C2 or Odroid XU5 (if is released). I really need vanilla linux kernel support (for headless server: Ethernet, MMC, ...), this allow usage of '''BTRFS''' to pass from 20MB/s read and 4MB/s write to 100/MBs read and 20MB/s write for my common case, mean '''average 5x boost''' without speak about space improvements.
* Other hardware used: TL-SG1024D (24 1000Mbps port switch not administrable), Industrial power supply Ebest CFS-200-5: OUTPUT: 5V 40A. Custom network wire to have the correct length and cat 5e due to sort distance.
* OS: '''Gentoo ARMv8''', Official Ubuntu 16.04 kernel (3.14LTS), hypervisor: '''Lxc''' + LXCFS (need more isolation because remain see the ressources usage of other VM), not vserver or openvz because ARM provide only vanilla kernel
* Hardware by node: [https://en.wikipedia.org/wiki/Amlogic#Media_player_SoCs_.28S9_family.29 Amlogic S905] ('''4*Cortex A53''' at 1.5GHz with 512KB of L2), memory: [http://www.samsung.com/semiconductor/global/file/product/DS_K4B4G1646D-BC_I_P_Rev102.pdf Samsung K4B4G1646D]: '''2GByte DDR3''' 32bit RAM, Ethernet: [http://www.datasheetspdf.com/datasheet/RTL8211F-CG.html Realtek RTL8211F] 1000Mbps
* From /dev/cpuinfo (Note: '''NO AES/SHA1/SHA2 for crypto acceleration'''):
** Features : fp asimd crc32
** CPU implementer : 0x41
** CPU architecture: 8
** CPU variant : 0x0
** CPU part : 0xd03
** CPU revision : 4
* Total cluster horse power: 184 64Bits cpu with SIMD, 138GB/s of main memory bandwidth for non cached data access as big data, estimated L1 bandwidth: 2TB/s, 92Gbps external bandwidth, 92GB of DDR3, 2944GB for storage.
'''Take care to well grounding all the hardware!''' [https://en.wikipedia.org/wiki/Single-phase_electric_power#Grounding Electric power grounding]
<blockquote>Typically a third conductor, called ground (or "safety ground") (U.S.) or protective earth (UK, Europe, IEC), is used as a protection against electric shock, and ordinarily only carries significant current when there is a circuit fault. Several different earthing systems are in use.</blockquote>
I have used odroid c2 power connector of rev0.2 20160226, but too hard to sold. Hole to small to stack 15 odroid. Then I have used the traditional power connector, all is connected on big wire with correct copper diameter to support 20A. (See the photo above)
Video: https://www.youtube.com/watch?v=DZXUCmBf_bg
= Photo =
[[File:bi-datacenter.jpg|thumb|400px|Bi portable datacenter, Red and Blue]]
[[File:running-cluster.jpg|thumb|400px|Running ARM cluster at 1000Mbps/1Gbps|left]]
[[File:running-cluster2.jpg|thumb|400px|Cloud of Odroid C2 ARMv8 Cortex A53]]
[[File:custom-power.jpg|thumb|400px|left|Power cord for High density cluster|left]]
[[File:odroid-c2-power-connector-rev0.2-20160226.jpg|thumb|400px|Custom power via Odroid c2 power connector rev0.2 20160226]]
[[File:running-bi-datacenter.jpg|thumb|400px|left|Running dual datacenter cluster|left]]
[[File:solar2common.jpg|thumb|400px|Solar conception for datacenter with common thing in bolivia]]
[[File:solar2efficiente.jpg|thumb|400px|left|Solar conception for datacenter with high efficiency]]
[[File:reverseproxy-ipv4toipv6.png|thumb|400px|Reverse proxy ipv4 to ipv6]]
[[File:simple-tunnel.jpg|thumb|400px|left|How cable the ovh ip to you home server into your local datacenter. After you attach the virtual machine in same way on too brige]]