/
PiCluster Build Notes.rtf
153 lines (125 loc) · 5.59 KB
/
PiCluster Build Notes.rtf
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
## Goofy Raspberry PI cluster stuff
## Uses a gl-inet router, 4 PIs, a TP-Link switch (all can be powered over usb) and an Anker 6 port USB power block
## I use a GeauxRobot 5 layer dogbone PI case (4 PIs + the switch)
## I use a TE100-55/a switch. When it is pulled out of its case, it fits nicely in one of the slots in the case
##
## So, my stack (from bottom up) is Anker power block (heavy so it is on the bottom for stability)
## Then a PI3 with a cluster hat and 4 PI zeros
## Another PI3
## The switch
## Then more 2 PI3s
## Then router mounted to the top (not inside the dogbone case)
## I made patch cables that are small and tight to manage the cables
## I also use 1' USB cables to manage cables
## This give a compact, portable cluster
## First download all the (clusterhat) things
## For simplicity grab the controller and PI zero isos from https://clusterhat.com/setup-software
## The rest of the PI3s run a standard raspian build
## make the appropirate raspbian disk images for each PI
## on mac (from terminal) do:
diskutil list
sudo diskutil unmount /dev/disk2s1 ## the disk ID of my SD card
sudo dd bs=1m if=2016-11-25-raspbian-jessie-lite.img of=/dev/disk2
## wash, rinse, repeat
## I use a gl-inet GL-MT300A device as the router/DHCP server/WISP/NTP server.
## It has an SD card slot that can be shared via NFS. I loaded a 64GB.
I also added static addresses for all the PIs in the gl-inet. It still does DHCP for any other devices
Vine IP: 192.168.31.40 ## (router name) Get it: 3.14...
Controller IP: 192.168.31.41 ## The PI with the clusterhat installed
Pi2 IP: 192.168.31.42
PI3 IP: 192.168.31.43
PI4 IP: 192.168.31.44
PZ1 IP: 192.168.31.141
PZ2 IP: 192.168.31.142
PZ3 IP: 192.168.31.143
PZ1 IP: 192.168.31.141
## On each PI do
raspi-config ## to expand the filesystem to the full SD card
passwd ## Change the PI user password
useradd ## Create some new user
usermod -aG sudo newuser ##Add your new user to the sudo group
## Edit /etc/hostname on each PI to name it
sudo vi /etc/hostname
## Edit /etc/hosts adding each PIs IP addresses
## sudo vi /etc/hosts
192.168.31.40 vine vine.local vine.net ## router name
192.168.31.41 PI1 PI1.local PI1.net
192.168.31.42 PI2 PI2.local PI2.net
### etc
## or add the host to the GL-inet router
## The gl-inet router serves as an NTP server, edit /etc/ntp.conf on each PI to use the router
sudo vi /etc/ntp.conf
server 192.168.31.40
## generate SSH keys and copy to each system
## edit, after monkeying around, I ended up just using the SSH key from the controller. I can shell into that and bounce to whatever
ssh-keygen -t rsa
## copy .pub to .ssh/authorized_keys on each PI
cat ~/.ssh/id_rsa.pub | ssh newuser@192.168.31.41 'cat >> .ssh/authorized_keys'
## or
ssh-copy-id user@hostname.example.com
## Setup NFS server on head router
## edit /etc/exports to include each PI (SD card named MicroSD)
/mnt/MicroSD PI1(rw,sync)
/mnt/MicroSD PI2(rw,sync)
### etc
sudo reboot
## Setup NFS on PIs
sudo apt-get install nfs-common
sudo mkdir /mnt/nfs
sudo chown -R newuser:newuser /mnt/nfs
sudo apt-get install autofs
## edit the /etc/auto.master file adding
/mnt/nfs /etc/auto.nfs ## at the end of the file.
## create the /etc/auto.nfs file, adding,
vine vine:/mnt/MicroSD
## restart the autofs service
sudo /etc/init.d/autofs restart.
## set rpcbind and nfs-common to start on boot
sudo update-rc.d rpcbind enable && sudo update-rc.d nfs-common enable
## reboot PIs
## after reboot you should be able to
cd /mnt/nfs/vine ## and see your SD card.
## Next (probably should have done it earlier) I installed Ansible on the controller
## I have an basic /etc/ansible/hosts file that looks like this
[PIS]
PI2
PI3
PI4
[PIZ]
PZ1
PZ2
PZ3
PZ4
## I'll add additional groups later (once I figure out what this thing is for)
## A couple quick ansible playbooks included in repository to add packages, shutdown the whole cluster, etc
## Use ansible to do a 'apt-get update && apt-get upgrade'
## Add any packages you want using
ansible-playbook add_package.yml ## specifying package in the yml
## Setup a method to manage dotfiles across all the PIs
## blatantly stolen from https://developer.atlassian.com/blog/2016/02/best-way-to-store-dotfiles-git-bare-repo/
## Intall git using
ansible-playbook add_package.yml ## specifying git in the yml
## Now with Switches
## Soldered right angle headers to the "run" points. Added female connectors, wire, and switches
## I drilled 4 holes (for each PI) on the top plate of the rack
## Soldered two wires to two of the poles of a 4 pole momenetary switch.
## Popped the switches into the holes and voila
## So right now I have a "stack" of PIs, but not really a "cluster".
## I still want to keep some of the PIs as distinct units, but I want to play with MPI
## So I decided to setup the Pi Zeros as a "cluster". Yes, I know they are underpowered and will likely be stupidly slow.
## This whole thing stack could be outmatched by a cheapo refurb computer. That's not the point of this frankenstack.
## I opted to go with MPICH on the PI zeros.
## Lots of this is based on: https://www.youtube.com/watch?v=vflPGqcXJkY
## Install packages (I did the first PI zero individually to help me refine the process, the other three through ansible)
sudo apt-get install mpich
sudo apt-get install python-dev
## Install mpi4py
mkdir mpi4py
cd mpi4py
wget https://bitbucket.org/mpi4py/mpi4py/downloads/mpi4py-2.0.0.tar.gz
tar -zxf mpi4py-2.0.0.tar.gz
cd mpi4py-2.0.0
sudo python setup.py build
sudo python setup.py install
## Export python path
export PYTHONPATH=/home/pi/mpi4py